The CORDEX community prescribes two European integration grids which differ only in resolution. The low-resolution EUR-44 domain's grid points are 0.44${}^{\circ }$ apart on a rotated lat–long grid limited to Europe (see inner orange box in Fig. , 106 $\times$ 103 grid boxes). For the high-resolution EUR-11 experiment, each EUR-44 grid box is divided into 16 0.11${}^{\circ }$-wide grid boxes. In K14, a total of 17 experiments were analysed by 9 different research groups. Eight groups performed both the EUR-11 and EUR-44 simulations, one group only EUR-11, and three groups used the same model (WRF) but with different physics parameterisations. All models are forced directly by ERA-Interim except for the experiment performed by CNRM. This group set up the global model ARPEGE (version 5.1) to be strongly nudged towards ERA-Interim outside of the CORDEX domain, but allowed the model to evolve freely inside of it. Further details on all models can be found in Table 1 of K14.

The main conclusions of K14 were that the higher resolution simulations did not perform significantly better and the models in the ensemble generally had a cold and wet bias, except for summers in southern Europe which are commonly warm and dry biased.

The ALARO-0 model used for this study is the identical configuration of the ALADIN system described in detail and validated by . Essentially, ALARO-0 uses the dynamical core of ALADIN, but with different physics routines (e.g. for radiation, microphysics and convection, cloudiness, turbulence), which are designed to tackle the issues that arise when using resolutions of 1–15 km, which is known as the grey zone for convection. Here, we only describe the EURO-CORDEX specific setup of the model, which is the coupling to the boundary conditions and the definition of the integration grids.

Similar to all other models in K14 (except for the global CNRM model), ALARO-0 is coupled to ERA-Interim by the classical Davies procedure . The relaxation zone consists of eight grid points irrespective of resolution, and new boundary conditions are provided every 6 hours. No further nudging or relaxation towards the boundary conditions was done inside of the domain. Some fields in ALARO-0 are constant during runtime, most notably sea surface temperatures (SSTs). Simulations are, however, interrupted and restarted monthly to allow for SSTs to be updated. Other fields that have monthly updates, but are constant during any given month are surface roughness length, surface emissivity, surface albedo and vegetation parameters. All other variables were computed continuously from 1 January 1979 to 31 December 2010 and thus, in contrast to , no daily restarts were done.

It would be preferable to use the exact rotated lat–long grids defined by the CORDEX community for the simulations. However, ALARO-0 does not support this projection but instead uses a conformal Lambert projection. Following the CORDEX guidelines, two new grids with a 12.5 and 50 km resolution were defined for the ALARO-0 simulations. Figure  shows the bounding boxes of the low-resolution (full green lines) and high-resolution (dashed green lines) ALARO-0 Lambert domains. The outer boxes show the complete domain, while the inner boxes exclude the relaxation zone. The grids were chosen such that the common EURO-CORDEX analysis domain (inner orange box in Fig. ) is completely included in the non-coupling zone. The low-resolution Lambert domain consists of 139-by-139 grid points, while the high-resolution domain consists of 501-by-501 grid points (both including eight coupling grid points at every boundary). In both simulations, the number of vertical levels was 46. Following K14, we will refer to the results with the acronym of the institute performing the simulations, yielding RMIB-UGent-11 and RMIB-UGent-44, for the high- and low-resolution simulations, respectively. These model data will be uploaded to the Earth System Grid Federation (ESGF, website: http://esgf.llnl.gov/) data nodes.

As an observational reference set, the E-OBS data set version 7 was used . The E-OBS data set has a 0.22${}^{\circ }$ rotated lat–long version (outer orange box in Fig. ) which encompasses the complete EURO-CORDEX domain. In the overlapping area, each E-OBS grid box contains four grid boxes of the EUR-11 domain and by consequence each EUR-44 box contains four E-OBS boxes.

In order to effectively compare model and observations, both need to share a common grid. The same approach as in K14 was taken to interpolate all data to a common grid. For the high-resolution simulations, first the values of the closest grid point were taken to go from the native Lambert ALARO-0 grid to the EUR-11 grid for both precipitation and temperature. For the latter, an additional height difference correction between the ALARO-0 and closest EUR-11 grid point was performed using the standard climatological lapse rate of 0.0064 K m${}^{-\mathrm{1}}$. Second, on this grid, for both precipitation and temperature, two-by-two grid box averages were calculated to obtain an identical grid to the E-OBS data set.

For the low-resolution simulations, again a closest grid point mapping from the native grid to the EUR-44 grid and temperature-height correction was performed. Then, the E-OBS data set was averaged over two-by-two grid boxes that are in every EUR-44 grid box and used as reference.

In K14, model performance is quantified for several metrics in different regions and seasons based on seasonal mean values of near-surface air temperature (or simply temperature from now on) and precipitation. All considered regions and their acronyms are shown in Fig.  and details regarding the definition of the different metrics can be found in K14, more specifically in Appendix A. Here, we only consider mean bias (BIAS), 95th percentile of the absolute grid point differences (95 %-P), ratio of spatial variability (RSV), pattern correlation (PACO), ratio of interannual variability (RIAV) and temporal correlation of interannual variability (TCOIAV). The climatological rank correlation (CRCO) and ratio of yearly amplitudes (ROYA) were not considered here, since these metrics showed very similar performance for all other models. Reanalysis forced simulations are by construction correlated with the observed weather at the seasonal timescale. For this reason, low correlation in time, even for short time periods, can be interpreted as an RCM deficiency for these simulations. This is not true for GCM-driven simulations, where only the correct number of occurrences in a certain time period (typically 30 years) are supposed to be represented and correlations at the shorter-than-decadal scale are meaningless due to strong interannual variability. Therefore, we expect TCOIAV to be positive for the simulations in this study, i.e. relatively cold/warm seasons in the simulations should coincide with relatively cold/warm observed seasons, while for GCM-driven simulations TCOIAV is expected to be zero. By contrast, all other scores should be similar for reanalysis-driven and GCM-driven RCM simulations if the GCM boundary conditions sufficiently represent the observed climatology. Due to realistic boundary conditions from reanalyses, the typical 30-year verification period for GCM-driven simulations can be shortened to 20 years, as in K14 where all scores are calculated based on the period 1989–2008. However, as the authors of K14 state, this implies that the “short evaluation period, leading to a sample size of only 20 seasonal/annual means, also hampers a sound analysis of statistical robustness”. The 32-year long integration period of ALARO-0 allows us to quantify how the scores change for different 20-year analysis periods and as such to test their robustness.

A jackknife procedure was applied for this purpose; let $\mathcal{I}$ $=$ $\left\{\right.$1979, …, 2010$\left..\right\}$ be the set of 32 years for which the ALARO-0 simulations were performed and $I$ a random subset of length 20 of $\mathcal{I}$. We write the score for the metric $s$ for a certain subregion $j$ and season $k$ based on the set of years $I$ as $s_{jk}\left(I\right)$ with $j$ $\in$ $\left\{\right.$BI, IP, FR, ME, SC, AL, MD, EA$\left..\right\}$, $k$ $\in$ $\left\{\right.$DJF: winter, MAM: spring, JJA: summer, SON: autumn, YEAR: year$\left..\right\}$. For example, in K14, values for $s_{jk}$ are calculated based on $I_{\text{K14}}$ $=$ $\left\{\right.$1989, …, 2008$\left..\right\}$. To study the robustness of $s_{jk}$ we study the distribution of $s_{jk}\left(I\right)$ for all possible $I$. The number of possible 20-year subsets from 32 years without repetition and ordering is given by the binomial coefficient: $32\mathrm{!}/\left(20\mathrm{!}\right(32-20\left)\mathrm{!}\right)$ $=$ 225 792 840. It is, however, not feasible to perform the calculations for all possible combinations and therefore only 1000 random sequences were chosen. The width of the 95 % confidence interval, limited by the 25th and 975th value of the ordered series of $s_{jk}$, then quantifies the robustness of the score.

Figure shows the spatial distribution of the daily mean temperature RMIB-UGent-11 BIAS in winter (DJF, left) and summer (JJA, right) for the years in $I_{\text{K14}}$. Compared to Fig. 2 from K14, the spatial bias of RMIB-UGent-11 in winter looks very similar to CNRM-11. Both models show a general cold bias in southern Europe, a warm bias in north-eastern Europe and a large east–west bias gradient linked to orography in Scandinavia. Compared to CNRM-11, the cold biases in mountainous regions are smaller for RMIB-UGent-11. In summer, again CNRM-11 and RMIB-UGent-11 share some biases although the difference is larger than in winter, and again the orographic forcing of the bias of CNRM-11 is more pronounced. Generally we find a cold bias, except in southern Europe where a warm bias is present.

Figure shows all metrics in separate columns for all different domains and seasons for seasonal and yearly mean temperature. The scale is shown at the bottom of each column, the full grey line shows the “optimal” score of the metric (0 K for BIAS and 95 %-P, 1 for all others). The grey circles show the scores for the high-resolution K14 ensemble (nine models). For each season and region, two transparent red bands are superimposed, which show the jackknife 95 % confidence interval for the high-resolution (top band) and low-resolution (bottom band) simulations with ALARO-0. The vertical red dashes show the value of $s_{jk}\left(I_{\text{K14}}\right)$, again for the high-resolution (top) and low-resolution (bottom) simulation. When the background colour is white, the RMIB-UGent-11 value of $s_{jk}\left(I_{\text{K14}}\right)$ lies within the K14 high-resolution ensemble spread. If the background colour is yellow, this value lies outside and is “worse” than the other members of the K14 ensemble. “Worse” means that the absolute distance from the RMIB-UGent-11 value based on $I_{\text{K14}}$ (top red dash) to the optimal value (grey line) is larger than that of any other K14 ensemble member. For example, the bias for the Iberian Peninsula in winter (in short written as BIAS-IP-DJF) is more negative than any other model, and it is in absolute value the furthest from the optimal 0 K. If instead the background colour is green, this indicates again the value is outside of the K14 ensemble but not the furthest from the optimal value. This implies that either RMIB-UGent-11 outperforms all other models (e.g. RSV-AL-DJF) or is not the worst model as defined above (e.g. RSV-EA-DJF is outside of the K14 ensemble, but not as bad as models at the other end of the ensemble).

Overall, Fig.  shows that (i) RMIB-UGent-11 mostly falls within the K14 ensemble (white background colour), (ii) the jackknife confidence intervals are always much smaller than the total spread of the K14 ensemble, except for RIAV and TCOIAV where the intervals often cover half of the ensemble spread, (iii) the difference between the RMIB-UGent-11 (top red dash) and RMIB-UGent-44 (bottom red dash) scores is very small considering the total range covered by the ensemble and the calculated jackknife confidence intervals.

A more detailed analysis shows that for BIAS, RMIB-UGent is almost always on the “cold side” of the K14 ensemble and even outside of its range on a fairly large amount of occasions. Especially for IP-DJF and SC-MAM, the cold bias is considerable. Also, RMIB-UGent-44 is slightly ($\sim$ 0.2 K) colder than RMIB-UGent-11, which may be due to regridding and the resolution difference. For 95 %-P, RMIB-UGent-11 is the worst model on four occasions among which most notably again are IP-DJF and SC-MAM.

For spatial correlation (PACO) and variability (RSV) RMIB-UGent-11 performs better. Although in K14 these two metrics are plotted on a Taylor diagram, we choose to show them here separately in one figure for clarity and conciseness. RSV for RMIB-UGent is almost always larger than 1, even where other models show less variability (e.g. ME). In the Alpine region (AL), RMIB-UGent seems to be able to grasp RSV well, but not at the right locations, as shown by the low PACO, especially in DJF. The jackknife confidence intervals are very small here, which indicates that both RSV and PACO produce very robust scores.

For RIAV and TCOIAV, RMIB-UGent again shows acceptable scores, some being outside of the K14 ensemble in a limited amount of cases. More notably, the jackknife confidence intervals are relatively large for these scores and this questions the robustness of these metrics. For example, for FR-MAM the TCOIAV based on $I_{\text{K14}}$ is 0.6, but the jackknife confidence interval extends from 0.6 to 0.8, covering all but two other models. For RIAV a similar situation for AL-JJA can be seen.

Figure  shows the spatial distribution of the relative seasonal precipitation BIAS (in %, (model $-$ observed)$/$observed) for the winter and summer season for the years in $I_{\text{K14}}$. Comparison to Fig. 3 of K14 shows that in winter, like all other models, RMIB-UGent-11 generally overestimates precipitation amounts, except in northern Africa. In contrast to temperature, RMIB-UGent-11 clearly differs from CNRM-11, with the latter showing large dry biases. In summer, RMIB-UGent-11 overestimates precipitation amounts, especially in the Mediterranean. Again, no clear resemblance to CNRM-11 is found.

Figure is constructed in the same way as Fig.  and shows all precipitation scores for all different metrics, regions and seasons. Similar to the temperature scores, the results for precipitation reveal that the majority of scores lie within the K14 ensemble, no difference between RMIB-UGent-11 and RMIB-UGent-44 is found and the jackknife confidence intervals are much smaller than the total ensemble range except for RIAV and TCOIAV. However, there is a clear absence of yellow scores and an increased presence of green scores, indicating that RMIB-UGent precipitation scores are generally better than the temperature scores.

RMIB-UGent has a wet BIAS for almost all regions and seasons. Remarkably, the best BIAS scores are obtained for SC-MAM and AL-DJF, where large temperature biases were found. Additionally, the corresponding 95 %-P scores are also on the low side which shows that the good performance is not due to compensating biases.

For RSV, RMIB-UGent performs relatively well and for PACO it excels, with 10 out of 80 region–season combinations performing better than the complete K14 ensemble. Only for AL-MAM is its performance not satisfactory, but remark that the actual score is an extreme outlier considering the jackknife confidence interval.

For RIAV, RMIB-UGent again performs consistently well, especially compared to the K14 ensemble which sometimes shows a large overestimation of interannual variability, i.e. very large values of RIAV. On the other hand, TCOIAV is mostly on the low side of the K14 ensemble, which shows that although RMIB-UGent gets the variability right, the actual temporal correlation is not well grasped. As for temperature, the large jackknife confidence intervals question the robustness of the scores.