2.1. 

Model configuration

In this section we introduce the perfect and forecast models. We assume that the phase-space for the large-scale dynamics is a subspace of the phase-space for the full high dimensional system. The complement of the subspace for the large-scales will correspond to the phase-space for the small-scale dynamics. The notation for the models will be in a partitioned form that separates the large and small scales. In particular, we denote the true, complete state at time t_k as ${(\begin{array}{cc}{\mathit{x}}^{l,t}& {\mathit{x}}^{s,t}\end{array})}_k^T\in {\mathbb{R}}^{N_t}$ such that ${\mathit{x}}^{l,t}\in {\mathrm{ℝ}}^{N_l}, {\mathit{x}}^{s,t}\in {\mathrm{ℝ}}^{N_s}$ and $N_t=N_l+N_s.$ Here, and throughout this paper, any component with a t-superscript indicates that it is a true variable. The l- and s-superscripts correspond to the large- and small-scale processes within the complete system dynamics. (We have deviated from the resolved/unresolved nomenclature of Janjic and Cohn 2006 for clarity, since the different filters used in our experiments resolve different scales).

An ideal linear model for the true state of a finite dimensional process can be expressed through the dynamical system

such that the matrix blocks ${\mathbf{M}}^{l,t}\in {\mathrm{ℝ}}^{N_l\times N_l}, {\mathbf{M}}^{ls,t}\in {\mathrm{ℝ}}^{N_l\times N_s}, {\mathbf{M}}^{s,t}\in {\mathrm{ℝ}}^{N_s\times N_s}$ and ${\mathbf{M}}^{sl,t}\in {\mathbb{R}}^{N_s\times N_l}.$ From a numerical modelling perspective, this partitioned description of the dynamics would be suited to a pseudospectral discretization (e.g. Fourier modes).

In numerical weather prediction (NWP), the true models that govern the evolution of the atmosphere are unknown and have to be approximated. For our approximation of the true dynamical system (2.1), we assume that any subgrid-scale parameterizations used to approximate the contribution from the small-scale processes to the large-scale state are contained within the large-scale model (Janjić and Cohn, 2006; Janjić et al., 2018). Hence, the model block ${\mathbf{M}}^{ls}={\mathbf{0}}_{N_l\times N_s}$ and our approximate dynamical model describing the complete system satisfies

In (2.2) each model block has the same dimensions as its true model counterpart. The large- and small-scale model errors are given by ${\mathit{\eta }}^l\in {\mathrm{ℝ}}^{N_l}$ and ${\mathit{\eta }}^s\in {\mathrm{ℝ}}^{N_s}$ respectively. Model errors are assumed to be random and unbiased with covariance given by

Here, using $〈·〉$ to indicate the mathematical expectation over the corresponding error distribution, the matrices ${\tilde{\mathbf{Q}}}^{ll}\equiv 〈{\mathit{\eta }}^l{\left({\mathit{\eta }}^l\right)}^T〉\in {\mathrm{ℝ}}^{N_l\times N_l}, {\tilde{\mathbf{Q}}}^{ss}\equiv 〈{\mathit{\eta }}^s{\left({\mathit{\eta }}^s\right)}^T〉\in {\mathrm{ℝ}}^{N_s\times N_s}$ and ${\tilde{\mathbf{Q}}}^{ls}\equiv 〈{\mathit{\eta }}^l{\left({\mathit{\eta }}^s\right)}^T〉\in {\mathrm{ℝ}}^{N_l\times N_s}$ (with ${\tilde{\mathbf{Q}}}^{ls}={({\tilde{\mathbf{Q}}}^{sl})}^T$) are the true model error covariances of the large-scale, the small-scale and cross-covariances between the large- and small-scale, respectively. We note that for the purposes of this work, the model error distribution is assumed to be stationary, so that $\tilde{\mathbf{Q}}$ is not a function of time.

Analogously, the complete forecast state ${(\begin{array}{cc}{\mathit{x}}^{l,f}& {\mathit{x}}^{s,f}\end{array})}^T\in {\mathbb{R}}^{N_t}$ satisfies

The forecast errors can then be defined as

where ${\mathbf{e}}^{l,f}\in {\mathbb{R}}^{N_l}$ and ${\mathbf{e}}^{s,f}\in {\mathbb{R}}^{N_s}$ are the large- and small-scale forecast errors respectively. The true forecast error covariance is denoted

Here, ${\tilde{\mathbf{P}}}_k^{ll,f}\equiv 〈{\mathbf{e}}_k^{l,f}{\left({\mathbf{e}}_k^{l,f}\right)}^T〉\in {\mathrm{ℝ}}^{N_l\times N_l}, {\tilde{\mathbf{P}}}_k^{ss,f}\equiv 〈{\mathbf{e}}_k^{s,f}{\left({\mathbf{e}}_k^{s,f}\right)}^T〉\in {\mathrm{ℝ}}^{N_s\times N_s}$ and ${\tilde{\mathbf{P}}}_k^{ls,f}\equiv 〈{\mathbf{e}}_k^{l,f}{({\mathbf{e}}_k^{s,f})}^T〉\in {\mathbb{R}}^{N_l\times N_s}$ (with ${\tilde{\mathbf{P}}}^{ls,f}={({\tilde{\mathbf{P}}}^{sl,f})}^T$) are the true forecast error covariances of the large-scale, the small-scale and cross-covariances between the large- and small-scale, respectively.

This formulation of the complete finite-dimensional dynamics allows us to consider several filters with different approaches to the treatment of large- and small-scales. Moreover, we can consider the interactions between scales and the effect they have on the modelling of observations.

2.2. 

Observations and their uncertainties

In this section we express the equations relating the observations, ${\mathit{y}}_k\in {\mathbb{R}}^p,$ to the model state in a partitioned form and describe their uncertainties. For the rest of this section, we assume that the model state and observations are valid at the same time, and drop the time subscript, k. At time t_k, the observations are related to the true model state as

where $\mathit{ϵ}\in {\mathrm{ℝ}}^p$ is the instrument error, assumed to be random and unbiased with covariance ${\tilde{\mathbf{R}}}^I=〈\mathit{ϵ}{\mathit{ϵ}}^T〉\in {\mathrm{ℝ}}^{p\times p}$ and ${\mathbf{H}}^{l,t}\in {\mathbb{R}}^{p\times N_l}$ and ${\mathbf{H}}^{s,t}\in {\mathbb{R}}^{p\times N_s}$ are the true linear observation operators which map the large- and small-scale states into observation space respectively. The observation operator $(\begin{array}{cc}{\mathbf{H}}^{l,t}& {\mathbf{H}}^{s,t}\end{array})$ is the (linear) finite-dimensional counterpart to the continuum observation operator of Janjić and Cohn (2006). We will not consider nonlinear observation operators in the remainder of this paper.

Throughout this paper, we assume that there is no pre-processing error. Hence, we will be concerned with the two cases described in sections 2.2.1 (all scales analysed) and 2.2.2 (large scales analysed) below. Case 1 shows the form of the representation error for filters that resolve all scales and is pertinent to the theoretical optimal Kalman filter discussed in section 3.2. Case 2 shows the form of the representation error for filters typically used in operational practice and is pertinent to the reduced-state Kalman filter and the Schmidt-Kalman filter discussed in sections 3.3 and 3.4 respectively.

2.3. 

Bias due to unresolved scales

Analysing only the large-scales will result in an error due to unresolved scales (section 2.2.2) that is sequentially correlated in time and correlated with the resolved state, leading to a potential bias (Janjić and Cohn, 2006). Assuming that the large-scale observation operator is unbiased, taking the expectation of the error due to unresolved scales, (2.11), (and reintroducing the time subscript k) results in

where $〈·〉$ denotes the mathematical expectation over the distribution of representation errors at time k. Using dynamical system (2.1), repeated substitution for the equation governing ${\mathit{x}}^{s,t}$ into the expected error due to unresolved scales yields

Here the underlined terms represent the contribution from the large scales. For many non-trivial models, these terms will not be identically zero, and potentially introduce a bias even if the initial value for the small-scale state is zero, ${\mathit{x}}_0^{s,t}=\mathbf{0}.$ For example, Janjić and Cohn (2006) solved a model of non-divergent linear advection on a sphere using a truncated expansion in spherical harmonics. Introducing a shear flow results in a dynamical system where the unresolved small-scales do not directly influence the resolved large-scales, but the large-scales influence the small-scales. This yields an error and bias due to unresolved scales. Janjić and Cohn (2006) were able to mitigate the bias using specific initial conditions. However, this experimental freedom would not be available in less-idealized situations. Therefore, when accounting for the unresolved scales in data assimilation we must also determine and treat any bias arising. In the new results in sections 5 - 6 below, we carefully construct our model to avoid bias due to unresolved scales. However, we revisit this problem in sections 7 - 9 where we use filters with bias-correction schemes.

3.1. 

A linear filter

A linear filter algorithm can be divided into analysis update and model prediction steps. The general form of the analysis update at time t_k, is given by

where ${\mathit{x}}_k^a$ is the analysis (state estimate), ${\mathit{x}}_k^f$ is the forecast state, ${\mathbf{K}}_k$ is the gain matrix and ${\mathbf{d}}_k^{o,f}={\mathit{y}}_k-{\mathbf{H}}_k{\mathit{x}}_k^f$ is the innovation, defined as the observation-minus-forecast departure. In this general setting we have not defined the dimensions of the vectors and matrices in (3.1), as this will depend on the specific choice of filter. For example, the state x in equation (3.1) could be either the complete state ${(\begin{array}{cc}{\mathit{x}}^l& {\mathit{x}}^s\end{array})}^T$ or just the large-scale state ${\mathit{x}}^l.$ There are various approaches to determine the gain matrix which will be discussed in sections 3.2, 3.3 and 3.4.

The perceived analysis error covariance update calcluated by the filter at time t_k is given by

where I is the identity matrix, ${\mathbf{H}}_k$ is the observation operator and ${\mathbf{P}}_k^f$ is the perceived forecast error covariance. Equation (3.2) is known as the short form of the analysis error covariance update. This equation only provides the correct estimate of the analysis error covariance if the background and observation error statistics used in the filter reflect the true error statistics. The use of a suboptimal gain and the short form update equation (3.2) will result in the filter producing incorrect error statistics. The true error statistics will be derived in section 4.

For the model prediction step, the forecast state at time $t_{k+1}$ is evolved from the analysis at the previous time-step and is given by

where M is a linear model. A model error term is not included in the forecast state update as linear filters estimate the mean state and we have assumed that the model error is unbiased. However, the error in the model M is accounted for in the forecast error covariance update given by which will be discussed further in section 4. We note that (3.4) will only produce correct error statistics when ${\mathbf{P}}_k^a$ and Q are equal to their true statistics counterparts.

Equations (3.1)–(3.4) form the core components of the linear filter algorithm. In the following sections we discuss three linear filters, each based on the Kalman filter (Kalman, 1960), that we will use in this paper. Table 1 summarizes the key vectors and matrices used in these three Kalman filters.

3.2. 

The optimal kalman filter (OKF)

For the optimal Kalman filter (OKF), we assume that we are able to model the processes for all scales and know the correct error statistics for the initial state, observations and model. Therefore, the perceived error statistics for the OKF will be equivalent to the true error statistics. The OKF simultaneously updates the large- and small-scale states, ${\mathit{x}}^l\in {\mathbb{R}}^{N_l}$ and ${\mathit{x}}^s\in {\mathbb{R}}^{N_s},$ so that the analysis update takes the form

cf. (3.1). The gain matrix for the OKF is partitioned into large- and small-scale components ${\mathbf{K}}^l\in {\mathbb{R}}^{N_l\times p}$ and ${\mathbf{K}}^s\in {\mathbb{R}}^{N_s\times p}$ respectively, and is given in Table 1. This is the optimal Kalman gain which minimises the trace of the analysis error covariance (e.g. Nichols, 2010). The analysis error covariance update is calculated using (3.2) with state error covariances with the same block structure as the forecast error covariance (2.6).

As the OKF filters all scales, the total observation error is described by Case 1 (all scales analysed, section 2.2.1). Hence, the observation error covariance for the OKF is $\mathbf{R}={\mathbf{R}}^I+{\mathbf{R}}^G.$

For the OKF forecast step we use the matrix

as our forecast model in (3.3) and the partitioned model error covariance given in Table 1 in the forecast error covariance prediction (3.4).

In summary, the analysis and forecast updates for the OKF state and covariance are a partitioned form of (3.1) - (3.4). By treating all scales in the assimilation the OKF has no error due to unresolved scales in the associated observation equation. However, due to computational constraints and inadequate knowledge of small-scale processes it is not possible to apply this technique in practice. Hence, methods that approximate the influence of small-scale processes must be employed instead.

3.3. 

The reduced-state kalman filter (RKF)

The suboptimal Kalman filter which estimates only the large-scale state and completely neglects the modelling of small-scale processes will be referred to as the reduced-state Kalman filter (RKF).

The analysis and forecast update equations for the RKF are simply the linear filter equations (3.1)-(3.4) where, as described in Table 1, we use the large-scale state, error covariances and observation operator. Thus, the forecast innovation is

where the second inequality can be established by adding and subtracting the term ${\mathbf{H}}^l{\mathit{x}}^{l,t}.$ Assuming each error has zero-mean, taking the expectation of the outer product of (3.7) yields the true innovation covariance (i.e. all contributing error covariances are true error statistics). However, the innovation covariance used by the RKF is given by where the large-scale forecast error covariance, instrument error covariance and representation error covariance are perceived error statistics. The influence of any small-scale processes is now accounted for through the representation error covariance ${\mathbf{R}}^H$ which needs to be approximated.

Reduced-state methods form an attractive approach in situations where computational expense is an important consideration. However, it is necessary to approximate the representation error covariance, ${\mathbf{R}}^H.$ Hence, the Kalman gain for the RKF will not minimise the analysis error covariance and the filter will be suboptimal.

3.4. 

The Schmidt-Kalman filter (SKF)

The Schmidt-Kalman Filter (SKF) estimates only the large-scale state, but the statistics of any unmodelled processes are used to determine the Kalman gain for the filtered state (Schmidt, 1966; Janjić and Cohn, 2006). A summary of the relevant equations is included in Table 1.

As only the large-scale state is estimated the forecast innovation is computed using the large-scale state, ${\mathit{x}}^{l,f},$ and observation operator, ${\mathbf{H}}^l.$ To determine the innovation covariance we start with the innovation (3.7) and add and subtract the term $(\begin{array}{cc}{\mathbf{H}}^l& {\mathbf{H}}^s\end{array}){(\begin{array}{cc}{\mathit{x}}^{l,t}& {\mathit{x}}^{s,t}\end{array})}^T.$ This allows us to write the innovation in the form

The innovation is now written in terms of the observation errors corresponding to case 1 (where all scales are analysed, see section 2.2.1), the large-scale forecast error mapped into observation space, ${\mathbf{H}}^l{\mathbf{e}}^{l,f},$ and the term ${\mathbf{H}}^s{\mathit{x}}^{s,t},$ the true small-scale state mapped into observation space. Assuming each error and ${\mathit{x}}^{s,t}$ has zero mean, taking the expectation of the outer product of (3.9) gives the true innovation covariance. The innovation covariance used by the SKF is given by

Here, we have abused our notation, to write ${\mathbf{P}}^{ls,f}$ as the perceived approximation of $〈-{\mathbf{e}}^{l,f}{({\mathit{x}}^{s,t})}^T〉,$ such that ${\mathbf{P}}^{sl,f}={({\mathbf{P}}^{ls,f})}^T.$ Using this notation for the cross-covariances of the SKF is common amongst other literature on the filter (e.g. Janjić and Cohn, 2006; Janjić et al., 2018). Following Janjić and Cohn (2006), we employ a prescribed error covariance ${\mathbf{C}}^s$ as a time-independent approximation of $〈{\mathit{x}}^{s,t}{({\mathit{x}}^{s,t})}^T〉.$ We note that as the small-scale error covariance is prescribed, the innovation covariance is an inexact approximation. The innovation covariance for the SKF is theoretically the same as the innovation covariance for the RKF (3.8) but expressed in a different form that includes contributions from the small scale processes.

The analysis state update for the SKF is given by

where ${\mathbf{K}}_k^l=({\mathbf{P}}_k^{ll,f}{({\mathbf{H}}^l)}^T+{\mathbf{P}}_k^{ls,f}{({\mathbf{H}}^s)}^T){\mathbf{D}}_k^{-1}$ is the Schmidt-Kalman large-scale gain. To obtain an analysis error covariance update equation we augment ${\mathbf{K}}^l$ with ${\mathbf{K}}^s={\mathbf{0}}_{N_s\times p}$ and substitute into equation (3.2). This is justified as the unfiltered state is assumed to have a small magnitude. Large uncertainty in the small-scale state or a small magnitude observation operator for this state would also justify this assumption (Simon, 2006). As the short-form analysis error covariance update for the SKF is not symmetric, we calculate ${\mathbf{P}}^{ll,a}$ and ${\mathbf{P}}^{ls,a}$ through the short-form update only and set ${\mathbf{P}}_k^{sl,a}={({\mathbf{P}}_k^{ls,a})}^T.$ Thus, the SKF analysis error covariance update equations are

We note that the term $-{\mathbf{K}}_k^l{\mathbf{H}}_k^s$ will usually be non-zero for the SKF. This term couples the large-scale uncertainty to the small-scale variability. If this term were zero, the large-scale state and uncertainty estimates produced by the RKF and SKF may still differ because of the differing innovation covariances between the filters.

The SKF treatment of the forecast step has a similar philosophy to the analysis step. The state prediction (3.3) is obtained through evolving the large-scale state ${\mathit{x}}^l$ with the large-scale forecast model ${\mathbf{M}}^l:$

The large-scale and cross-covariance blocks of the forecast error covariance are calculated using the complete forecast model and model error covariance in (3.4). The SKF forecast error covariance update equations are

The prescribed small-scale error covariance ${\mathbf{C}}^s$ is assumed constant in time and is not updated.

The appeal of the SKF is in its ability to compensate for small-scales without estimation of the small-scale state. Practical implementation of the SKF would require the filter to be adapted to nonlinear models. However, even for linear systems, the models evolving the small-scale processes would be unknown and their influence on the error covariances would need to be quantified. Additionally, the propagation of the state cross-covariances poses a considerable computational cost.

3.5. 

Discussion

The OKF, SKF and RKF represent three different approaches for dealing with observation uncertainty due to unresolved scales (see Table 1). The OKF analyses all scales, thus avoiding the error due to unresolved scales altogether, while the RKF completely disregards the small-scale processes and accounts for the error due to unresolved scales through the representation error covariance matrix. The SKF, however, takes a compromise approach where only the large-scale state is estimated, but the uncertainty in all-scales is accounted for in the estimation. Additionally, the SKF accounts for the flow-dependence of the correlations between the large-scale errors and small-scale processes (albeit approximately) through the cross-covariances in the analysis and forecast error covariances given in equations (3.13), (3.14), (3.17) and (3.18). Applications where it is a poor approximation to neglect these cross-covariances will benefit the most from using the SKF (as opposed to the RKF where these cross-covariances are neglected).

4.1. 

Case 1: true analysis error covariance when all scales are filtered

To obtain the true analysis error equation we assume that we have complete knowledge of all scales as with the OKF. As in section 2.2.1 the observation error will consist of instrument error, $\mathit{ϵ},$ and the observation operator error for large- and small-scales, ${\mathit{\gamma }}^l+{\mathit{\gamma }}^s.$ Under these assumptions the true analysis error equation will be a partitioned form of equation (4.1) and the true analysis error covariance will be a partitioned form of (4.2) with the components given in column 1 of Table 2.

4.2. 

Case 2: true analysis error covariance when only large-scales are filtered

The true analysis error equation for case 2 applies to filters that estimate the large-scale state only like the RKF and SKF. Using equation (4.1) and the state gain matrices and observation operators, the large-scale analysis error for the RKF and SKF is given by

We note that the observation errors correspond to case 2 described in 2.2.2 as both the RKF and SKF filter the large-scale state only. Hence, the effect of the small-scale processes on ${\mathbf{e}}_k^{l,a}$ in equation (4.4) is determined through the error due to unresolved scales ${\mathbf{H}}^{s,t}{\mathit{x}}_k^{s,t}.$ We observe that the true large-scale error covariance may thus be written in terms of the representation error covariance as

However, the true error statistics contributing to the true analysis error covariance are unknown in practice making the use of (4.5) to evaluate filter performance unfeasible. For theoretical experiments where most error statistics can be prescribed, determining ${\tilde{\mathbf{R}}}_k^H$ requires a Monte Carlo approach due to its dependence on ${\mathit{x}}^{s,t}.$ Alternatively, a different form of the analysis error equation may be more practical.

Assuming we know the true behaviour for the small-scales we can express the true analysis error equation for the RKF and SKF as

where ${\mathbf{e}}_k^{l,f}=\mathbf{M}{\mathbf{e}}_{k-1}^{l,a}+{\mathit{\eta }}_k^l$ and ${\mathbf{e}}_k^{s,f}={\mathbf{M}}^{sl}{\mathbf{e}}_{k-1}^{l,a}+{\mathbf{M}}^s{\mathbf{e}}_{k-1}^{s,a}+{\mathit{\eta }}_k^s.$ We note that as the small-scale state isn’t estimated the small-scale gain is a zero matrix of dimension $N_s\times p.$ We also note that, while the large- and small-scale errors ostensibly appear uncoupled in equation (4.6), they are in fact coupled as ${\mathbf{e}}_k^{s,a}$ and ${\mathbf{e}}_k^{s,f}$ each depend on ${\mathit{x}}_k^{s,t}.$ Adding and subtracting the term ${\mathbf{K}}_k^l{\mathbf{H}}_k^s{\mathbf{e}}_k^{s,f}$ from the large-scale component of the analysis error, (4.6) may be written as

Using the definitions of the small-scale observation operator error (2.9), the error due to unresolved scales (2.11) and the small-scale forecast error (3.9) we find that

Thus the right-hand-side of (4.7) can be evaluated without knowledge of the error due to unresolved scales specifically. Instead, this can be written in terms of the observation operator error and a small-scale forecast:

The partitioned case 2 error equation (4.9) can be used to obtain the true analysis error covariance for the SKF and RKF without knowing the full representation error covariance ${\tilde{\mathbf{R}}}^H.$ However, when using this form of the analysis error equation to obtain the true error statistics the correlations between ${\mathit{x}}^{s,f}$ and ${\mathbf{e}}^{s,f}$ may be non-negligible. We note that, while ${\mathit{x}}^{s,f}$ and ${\mathit{x}}^{s,t}$ will also be unknown in practice, they could be approximated offline with high-resolution models.

5.1. 

Gaussian random walk model

We now consider the methodology for numerical experiments where we apply the three filters to the simple model system

such that ${\eta }_k^l\sim (0, {\mathrm{Q}}^l), {\eta }_k^s\sim (0, {\mathrm{Q}}^s),$ and ${ϵ}_k\sim (0,R^I)$ (Brown and Hwang, 2012). This system uses one variable for the large-scale state, x^l, and one variable for the small-scale state, x^s. The large-scale state x^l and small-scale state x^s are random walk variables driven by the errors η^l and η^s whose structures are determined by the variances ${\mathrm{Q}}^l$ and ${\mathrm{Q}}^s$ respectively. There are no cross-covariances in the model error statistics. The model component ${\mathrm{M}}^{sl}$ is the contribution from the large-scale processes to the small-scale state. The observations will be taken to be the sum of the large- and small-scale states plus instrument error.

The random walk model (5.1) will first be used for a “nature run” from which observations can be created. The filters described in section 3 will then be used to assimilate these observations and the true large-scale analysis error variance calculated at the end of the assimilation window. As the RKF and SKF are suboptimal, they propagate inexact error variances. Therefore, the true error variances for the RKF and SKF are calculated using (4.9) to provide a comparison between their performances.

Through our experimental design we are able to easily control the magnitude of the observation error due to unresolved scales by adjusting ${\mathrm{Q}}^s.$ The relationship between ${\mathrm{Q}}^s$ and the error due to unresolved scales is described in section 5.3. This framework also allows for the determination of the optimal ${\mathrm{C}}^s$ as well as the sensitivity of the SKF to this modelled variance.

5.2. 

Initial conditions and filter parameters

For our experiments, we choose the initial conditions for the true state (nature run) to be $x_0^l=10$ and $x_0^s=0$ so that the true resolved state is an order of magnitude larger than the unresolved state. Setting the small-scale true state to zero also ensures that the representation errors are initially unbiased.

We set the initial conditions for the forecast state and forecast error covariance to be

where ${\alpha }^l\sim \mathcal{N}(0, {\mathrm{P}}_0^{ll,f})$ and ${\alpha }^s\sim \mathcal{N}(0, {\mathrm{P}}_0^{ss,f})$ are perturbations from the true states. We have assumed that the initial large- and small-scale forecast errors are uncorrelated.

For our first set of experiments we set the model component ${\mathrm{M}}^{sl}=0$ so that the representation errors remain unbiased throughout the assimilation. The large-scale model error variance, ${\mathrm{Q}}^l=1,$ is used throughout our experiments while ${\mathrm{Q}}^s$ will vary for different experiments.

Observations are assimilated every time-step. The true observation operator is $\mathbf{H}=(\begin{array}{cc}1& 1\end{array})$ which is used by all three filters; this ensures that there is no observation operator error. Unless otherwise specified, the instrument error variance is set to ${\mathrm{R}}^I=0.1,$ and ${\tilde{\mathrm{R}}}_k^I={\mathrm{R}}^I$ so that each filter correctly accounts for the instrument error. For the RKF, we set ${\mathrm{R}}^H=0$ so that the filter completely ignores the small-scale processes. The prescribed small-scale error variance ${\mathrm{C}}^s$ is varied throughout our experiments.

To calculate the true analysis error variance with (4.9), we neglect the variance of $x^{s,f}$ and the correlations between $x^{s,f}$ and ${\mathrm{e}}^{s,f}$ as the solution for $x^{s,f}$ is exponentially decaying with time. This method of calculating the true analysis error covariance has been validated against a Monte Carlo approach.

5.3. 

Determining the small-scale variability over the assimilation window

For the SKF and RKF, which do not update the small-scale state, the true small-scale analysis and forecast error statistics are equal at the same time-step. Setting ${\mathrm{M}}^{sl}=0,$ the true small-scale error covariance, denoted ${\tilde{\mathrm{P}}}^{ss},$ is evolved through the difference equation

For the Gaussian random walk model, we may use a scalar version of this equation, given by

where ${\tilde{\mathrm{P}}}_0^{ss}={\tilde{\mathrm{P}}}_0^{ss,f}$ and ${\mathrm{M}}^s=\hspace{0.17em}\text{exp}\hspace{0.17em}(-1/2).$ Noting that the summation term in (5.4) is a geometric series we can express ${\tilde{\mathrm{P}}}_k^{ss}$ as

For large k, the first term in equation (5.5) decays to zero while the second term tends to the limit ${\tilde{\mathrm{Q}}}^s/(1-e^{-1}).$ Hence, after a burn-in period the error due to unresolved scales for the SKF and RKF is primarily determined by the size of ${\tilde{\mathrm{Q}}}^s$ and increases each time-step.

6.1. 

Determining the optimal C^s

Before using the SKF, we first need to approximate the matrix ${\mathrm{C}}^s$ (see Table 1). To find the optimal value of ${\mathrm{C}}^s$ over the whole assimilation window we carry out numerical experiments for a range of values of ${\mathrm{R}}^I$ and ${\mathrm{Q}}^s.$ Both of these parameters will affect the magnitude of the true large-scale analysis error variance. For each $({\mathrm{R}}^I, {\mathrm{Q}}^s)$ parameter pair, we test a number of values of ${\mathrm{C}}^s$ to determine the value of ${\mathrm{C}}^s$ which gives the smallest true large-scale analysis error variance at the final assimilated observation. As we are calculating the variances only, the calculation is deterministic and the choice of noise realisation is irrelevant. For this experiment, we assimilate 15 observations. We start with ${\mathrm{C}}^s=0$ and increase ${\mathrm{C}}^s$ in steps of $\Delta {\mathrm{C}}^s=0.001$ until ${\mathrm{C}}^s=1.$

The optimal values of ${\mathrm{C}}^s$ that produce the minimum large-scale analysis error variance for the SKF at the final time-step are shown in Fig. 1. The optimal value of ${\mathrm{C}}^s$ increases as both ${\mathrm{R}}^I$ and ${\mathrm{Q}}^s$ increase. In particular, the optimal value of ${\mathrm{C}}^s$ is most sensitive to any increase in ${\mathrm{Q}}^s$ as the error due to unresolved scales is primarily determined by this error variance in our model. While not as sensitive, we find that large ${\mathrm{R}}^I$ also affects the optimal value of ${\mathrm{C}}^s.$ This is because the optimal value of ${\mathrm{C}}^s$ is a function of ${\mathrm{R}}^I$ and ${\mathrm{Q}}^l$ after the initial time. We also find that for small ${\mathrm{R}}^I$ and ${\mathrm{Q}}^s$ the optimal value of ${\mathrm{C}}^s$ over the whole assimilation window is similar to ${\tilde{\mathrm{P}}}^{ss}$ given by equation (5.5) for the final time-step. For large ${\mathrm{R}}^I$ and ${\mathrm{Q}}^s,$ the optimal value of ${\mathrm{C}}^s$ is approximately 1.4 times larger than ${\tilde{\mathrm{P}}}^{ss}$ evaluated at the final time-step.

In operational settings we would not be able to optimise ${\mathrm{C}}^s$ in this way. However, it may be possible to approximate part of the representation error covariance using high resolution observations (Oke and Sakov, 2008) or model data (Waller et al., 2014) and use these approximate representation error values to guide the choice of ${\mathrm{C}}^s.$ To mimic this situation in our experiments, we create an ensemble of 50,000 realizations of x^s for the length of the assimilation windows using the random walk model (5.1) and calculate the variance, averaged over the whole ensemble and time. The variance of this ensemble will be denoted $\mathrm{S}$. The variance, $\mathrm{S}$, represents an approximation to the total small-scale variability over the assimilation window. We now compare the values of ${\mathrm{C}}^s$ computed in Fig. 1 with the values of S.

Figure 2(a) shows the optimal ${\mathrm{C}}^s$ values when ${\mathrm{R}}^I=0.1$ (dashed line) and ${\mathrm{R}}^I=0.5$ (dotted line) for different values of ${\mathrm{Q}}^s.$ The grey region shows all points between $\mathrm{S}$ and 2S. As ${\mathrm{Q}}^s$ is increased the variance $\mathrm{S}$ also increases. Both optimal ${\mathrm{C}}^s$ lines lie within the shaded region for nearly all ${\mathrm{Q}}^s.$ We note that when there is little small-scale variability (i.e. ${\mathrm{Q}}^s\approx 0$) the optimal ${\mathrm{C}}^s$ values are less than $\mathrm{S}$ but both are close to zero. Figure 2(b) shows the effect of changing ${\mathrm{C}}^s$ on the SKF true large-scale analysis error variance (solid line) when ${\mathrm{R}}^I=0.1$ and ${\mathrm{Q}}^s=0.35$ in comparison to the true large scale analysis error variances for the RKF and OKF. Thus, for these experiments, a reasonable rule of thumb to avoid areas where the SKF under- or overcompensates for the error due to unresolved scales, is to choose

6.2. 

Comparison of the SKF with the RKF and OKF

Using the optimal values of ${\mathrm{C}}^s$ calculated in Fig. 1, we now carry out experiments comparing the SKF and RKF for a range of values of ${\mathrm{R}}^I$ and ${\mathrm{Q}}^s$ relative to the OKF. The results are illustrated in terms of relative error percentage for the RKF in Fig. 3a and the SKF in Fig. 3b. The relative error percentage is defined as

where $|·|$ indicates the absolute value and each term is evaluated at the end of the assimilation window. In these experiments, the SKF always has a true analysis error variance smaller than or equal to the RKF. When there is no error due to unresolved scales (i.e. ${\mathrm{Q}}^s=0$) we have that ${\mathrm{C}}^s=0$ is the optimal value for ${\mathrm{C}}^s$ and the SKF would reduce to the RKF. The largest relative error percentages for both the RKF and SKF occur when there is large uncertainty due to unresolved scales (large ${\mathrm{Q}}^s$) and small ${\mathrm{R}}^I$ and the smallest differences are when ${\mathrm{Q}}^s$ is small. We also find that larger values of ${\mathrm{R}}^I$ limit the difference in performance between the RKF and SKF with the OKF. Therefore, the benefits of using the SKF are most apparent when there is considerable error due to unresolved scales and small instrument error. Comparing Fig. 3a to Fig. 3b we see that, for any fixed value of ${\mathrm{R}}^I,$ as the uncertainty due to unresolved scales is increased the improvement of the SKF over the RKF will also increase.

To examine the performance perceived by the filter we compare it to the true performance of the filter at the final time-step. Before discussing the results, we note that the SKF perceived analysis error variance will not be a smooth field for the $({\mathrm{R}}^I, {\mathrm{Q}}^s)$ parameter pairs considered. This is because in section 6.1 the optimal value of ${\mathrm{C}}^s$ was calculated to a limited precision of 0.001.

In Fig. 4 we plot the difference between the perceived and true analysis error variance at the final time-step. We note that the magnitude of the difference between the perceived and true error variances is smallest for large ${\mathrm{R}}^I$ and ${\mathrm{Q}}^s.$ Here, the SKF (RKF) perceived error variance is approximately 1.25 (0.5) times the size of the true error variance. The SKF perceived-minus-truth difference shown in panel (a) is always positive for non-negligible representation uncertainty. This shows the SKF is a conservative filtering strategy when compensating for observations exhibiting error due to unresolved scales. As both ${\mathrm{R}}^I$ and ${\mathrm{Q}}^s$ are increased the SKF perceived-minus-truth difference increases. This is due to two reasons. The first reason is because the perceived analysis error variance, ${\mathrm{P}}^{ll,a},$ increases with larger ${\mathrm{C}}^s$ as it is calculated using the short form update (3.2) and the optimal ${\mathrm{C}}^s$ will be larger for higher values of ${\mathrm{R}}^I$ and ${\mathrm{Q}}^s.$ The second reason is because, for non-negligible representation uncertainty, the true analysis error variance, ${\tilde{\mathrm{P}}}^{ll,a},$ will decrease as ${\mathrm{C}}^s$ approaches its optimal value. An illustrative case is provided by Fig. 2b for high representation uncertainty and low instrument uncertainty. Figure 4(b) shows the RKF perceived-minus-truth difference. This is always negative for non-negligible representation uncertainty. This shows the RKF is an overconfident filtering strategy for observations exhibiting error due to unresolved scales. The RKF is most overconfident in regimes of low instrument uncertainty and high representation uncertainty.

7.1. 

The Schmidt-Kalman filter with observational bias correction

Bias correction through state augmentation is a common method used in operational centres but use of a bias correction scheme with the SKF, which will be denoted SKFbc, is novel.

We assume that we have knowledge of the processes for all scales. We further assume that we have a model and prior estimate for the bias. The filtered state vector for the SKFbc is given by equation (7.1) and only includes the large-scale state and the bias term. The small-scale state is split into a biased and unbiased component, i.e.

such that $〈{\mathit{x}}^s〉={\mathit{x}}^{\beta }.$ The unbiased small-scale processes, ${\mathit{x}}^{\delta },$ will be accounted for through their statistics. The full observation operator for the SKFbc is given by where ${\mathbf{H}}^{\beta }\in {\mathbb{R}}^{p\times N_s}$ and ${\mathbf{H}}^{\delta }\in {\mathbb{R}}^{p\times N_s}$ are the linear observation operators which map the bias and unbiased small-scale states into observation space respectively. However, as with the SKF, the analysis update equation (3.1) uses a forecast innovation that takes no account of the unbiased small-scales,

This innovation is unbiased and hence the large-scale analysis errors are also unbiased. The Kalman gain for the SKFbc consists of a large-scale gain ${\mathbf{K}}^l\in {\mathbb{R}}^{N_l\times p}$ and a bias estimate gain ${\mathbf{K}}^{\beta }\in {\mathbb{R}}^{N_s\times p}$ given by

where ${\mathbf{P}}^{l\beta }\in {\mathbb{R}}^{N_l\times N_s}$ is the perceived cross-covariance between the large-scale errors and bias estimate errors, ${\mathbf{P}}^{l\delta }\in {\mathbb{R}}^{N_l\times N_s}$ is the perceived cross-covariance between the large-scale errors and unbiased small-scale errors, ${\mathbf{P}}^{\beta \beta }\in {\mathbb{R}}^{N_s\times N_s}$ is the perceived covariance of the bias estimate errors and ${\mathbf{P}}^{\beta \delta }\in {\mathbb{R}}^{N_s\times N_s}$ is the perceived cross-covariance between the bias estimate errors and unbiased small-scale errors. The perceived augmented innovation covariance D is given in Table 3. This increases the uncertainty the filter attributes to the forecast innovation compared with the standard SKF. The additional uncertainty is a result of the errors accrued in the estimation of the bias. The term ${\mathbf{H}}^s{\mathbf{C}}^{\delta }{({\mathbf{H}}^s)}^T$ in the SKFbc innovation error covariance corresponds to the variability of the unbiased small-scale processes.

The SKFbc equations are obtained from augmenting the large-scale terms with bias terms and defining the cross-covariance terms appropriately. The analysis state update for the SKFbc is then

To obtain the analysis error covariance update we augment the gain (7.6) with ${\mathbf{K}}^{\delta }={\mathbf{0}}_{N_s\times p}$ and substitute into the short-form analysis error covariance update (3.2). To mirror the SKF analysis error covariance update equations (3.12)-(3.14), we express the SKFbc analysis error covariance update equations as

Since in the context of the SKFbc the complete model evolving all scales is assumed to be known, it is appropriate to update the bias term using this model (7.2). Thus, the forecast state update is given by

For the forecast error covariance we need the model for the unbiased small-scale processes. To determine this model we use the definition (7.3) together with the bias evolution equation (7.2), to give

Note that the small-scale model error ${\mathit{\eta }}_{}^s$ is assumed to be unbiased. To mirror the SKF forecast error covariance update equations (3.16)-(3.18), we express the SKFbc forecast error covariance updates as

The prescribed unbiased small-scale error covariance ${\mathbf{C}}^{\delta }$ is assumed constant in time and is not updated.

The SKFbc allows us to correct biases due to unresolved scales and consider the effects of the unbiased small-scale processes on the large-scale state. A key advantage in this method is that the cross-correlations between the large-scale errors and small-scale errors are retained. However, the SKF is a computationally expensive procedure. This issue is exacerbated by the use of state augmentation for bias correction.