Assimilation of satellite sounder profiles in NWP

The aims are to maximise the amount of information conveyed to the model and to minimise the computational load during the assimilation process. The study was performed at ECMWF, in their operational Integrated Forecast System (IFS), and these preliminary experiments assimilated data over oceans. The forecast skills were evaluated against independent reference measurements. The performance of the forecasts in the free troposphere with L2 were already close to the operational baseline with L1. In the boundary layer, the effectiveness of the temperature forecasts having assimilated L1 and L2 were equivalent, and L2 assimilation yielded better humidity forecasts than radiances.

An accurate description of the state of the atmosphere is crucial to initialise NWP, for which satellite observations provide key information. In particular, hyperspectral IR sounders have become one of the most important contributors in the last two decades, with the AIRS, IASI and CrIS missions.

As outlined in [Eyre et al. 2019], geophysical parameters retrieved from satellite observations were assimilated in numerical models up to the 1980s. The NWP centres then changed their approach and started to assimilate radiances (L1) in the 1990s, instead of retrieved physical quantities (L2). One motivation at the time was to avoid degrading the model background with inaccurate information from the null space, eg with small-scale vertical structures or from atmospheric layers for which these passive sounders have little to no sensitivity. This was guaranteed by the utilisation of the radiative transfer model as an observation operator.

Despite the operational maturity and success of this approach, it still faces scientific and technical challenges, which limits the exploitation of the IR sounders to their full extent:

• Uncertainties due to the parameters not accurately modelled: trace gases, surface emissivity, levels above the model top, misclassified clouds etc.
• Radiative transfer in clouds still being an active area of research.
• The heavy computational load incurred by the radiative transfer calculations.

Consequently, only a limited subset of channels is assimilated, typically from a few 10s to about 200 channels, depending on the numerical models and the scenes. The selection is usually limited to cloud-free pixels or to those channels peaking above the clouds. Because of the uncertainties in the radiative transfer modelling, channels peaking in the lower troposphere tend to be excluded or down-weighted, in particular for humidity and over land, where even fewer channels are assimilated. The definition of a functional observation error matrix, required in the assimilation process, has been mostly empirical and is still subject to active studies, especially for spectrally correlated observations, like with hyperspectral sounders (Assimilation of reconstructed radiances).

The present study explored the operational potential for NWP — both from a scientific and a functional perspective — of assimilating all-sky L2 products from satellite sounders, with a scene-dependent error characterisation and observation operators. It builds on the promising results from preliminary studies at ECMWF and Météo-France, where retrieved profiles were assimilated as pseudo-sondes. Though not accounting at the time for the vertical error correlations and the vertical sensitivity functions, the L2 yielded positive impacts on forecasts, including with cloudy retrievals (Preliminary assimilation of L2).

The atmospheric temperature and humidity profiles used here are retrieved with a prototype evolution of the operational machine learning method (An introduction to the PWLR), from which it is also possible to compute associated averaging kernels and error estimate covariance. In this new approach, the L2 products are represented in a limited set of principal components (typically 10 to 25), which already accounts for the intrinsic vertical resolution of the sounder. The study was performed with retrievals using IASI measurements only — i.e without exploiting the companion microwave sounders onboard Metop, to serve as proxy for MTG-IRS products.

The aim was to convey more geophysical information to the NWP system than through radiances, because the retrievals exploit the full spectral range measured and because of the provision of sounding in cloudy pixels. Furthermore, as it does not necessitate a Radiative Transfer Model (RTM), the assimilation of such L2 products is expected to be computationally significantly less expensive than with L1 radiances. This technique could, hence, be particularly interesting in contexts where strict trade-offs exist between stricter timeliness requirements in NWP, cost of CPU resources, and the ever rising amount of satellite information in NWP. MTG-IRS, in particular, will produce 100 times more observations than current instruments like IASI.

Objectives

The study objectives were to:

• evaluate the practicalities of assimilating atmospheric profiles in PC scores, with scene-dependent observation operators;
• evaluate the precision of the sounding products (L2) and the pertinence of the associated uncertainty estimates;
• set-up, run and evaluate assimilation experiments of L2 products in the ECMWF IFS, compared to the assimilation of radiances;
• provide feedback and guidelines to the users and product developers, in view of their potential utilisation in NWP.

Overview

• System preparation and assimilation experiments definition
• Passive monitoring: products precision, uncertainty estimates and observation operators
• First assimilation experiments: depleted environment, maritime, clear-sky
• Optimising the observation error
• Assimilating cloudy retrievals
• Increasing spatial density, with intelligent data thinning
• Full system, assimilating maritime all-sky retrievals
• Conclusions
• Outlook

System preparation and assimilation experiments definition

In this study, the elements of the cost function minimised during data assimilation

J(x) = (x-x_b)^TB^-1 (x-x_b) + (y — H[x])^TR^-1 (y — H[x])

are

• x is the geophysical state vector resulting from the assimilation — here T and q profiles.
• x_b is the background state vector, the forecast from the previous run
• B is the background error covariance matrix.
• y is the observation vector — here the L2 products T/q in PC.
• H[x] is the observation operator — here transforming the vertical profiles of T/q in PCs.
• R is the observation error covariance matrix.

The observation operator H writes

H[x] = (U^TSU)^-1/2U^TE^T(x-<x>)

where

• x is the geophysical state vector analysed during the assimilation — here T and q profiles.
• <x> is the average state vector.
• E are the truncated eigenvectors (leading directions) transforming the profiles in PCs.
• U is the left singular-vector of the retrieval averaging kernels.
• S is the retrieval uncertainty covariance matrix.

E, U and S are scene-dependent, defined for different observations classes. The classification is obtained by k-mean clustering applied to the observations (An introduction to the PWLR). By construction, the error covariance matrix of the observations is expected to be the Identity matrix.

The IFS framework has been successfully adapted to ingest the L2 products in PC space. The assimilation experiments are composed as follows:

Passive monitoring: products precision, uncertainty estimates and observation operators

Main outcomes of the passive monitoring of the IASI L2 against ECMWF forecasts, in profile (Figure1) and PC (Figure 2) space:

• Good quality retrievals in profile space.
• Reliable auxiliary information for quality control and data acceptance — cloudiness intensity (OmC, OBS-CALC estimates in window channels) and uncertainty estimates (QI).
• Best precision, within or below 1K all along in the vertical, is achieved with clearest pixels (|OmC| < 1K) and best quality (uncertainty QIT < 2K).
• The quality in the lower atmosphere decreases as the cloudiness and error estimates increase (consistent with IASI only providing information from above the clouds in strong overcast).
• The O-B variance for temperature in PC is overall close to 1 as per theoretical expectations for most situations. However, it exceeds 1 in the trailing directions and more surprisingly in the first leading direction in clear-sky.
• The O-B variance for humidity in PC is lower than 1 overall, suggesting that the error estimates on IASI L2 humidity might be slightly overestimated.

First assimilation experiments: depleted environment, maritime, clear-sky

The first assimilation experiments were carried out in IFS with:

• L2 temperature and humidity in PC — both observation errors set to Identity matrix;.
• L1c radiances as per operational baseline;.
• Clear-sky (different cloud mask, based on L1 and L2 information respectively);
• Maritime pixels.

Outcome:

✓ Positive impact of temperature and humidity forecasts with L2.
✓ Forecast performance with L2 lower but close to L1 in the free troposphere.
✓ L2 yields comparable positive impact as L1 on temperature in the lower troposphere.
✓ L2 yields stronger positive impact than L1 on humidity in the lower troposphere.

Optimising the observation error

As explained in the first paragraph, the observation error covariance matrix should be 1, by construction of the observation operator. However, Figure 3 hints that the error estimates provided along the L2 products may be overestimated for humidity. Though the statistics of the difference model v L2 also include forecasts errors in addition to L2’s errors, it may be that the L2 uncertainty estimates are underestimated in some directions for temperature.

A series of experiments was, therefore, attempted to try and optimise:

• information conveyed to the model, i.e. truncating or using the full 15 eigenvectors;
• observation error, i.e. decreasing or inflating the observation errors in step between 0.25 and 1.75.

Outcome:

This best forecast scores (see Figure 5) were obtained with:

• all 15 directions retained for temperature and humidity;
• error on temperature set to 1.5 in all PC directions;
• error on humidity set to 0.75 in all PC directions.

This forms the baseline for the rest of the study.

Assimilating cloudy retrievals

This series of experiments aimed at increasing the amount of information conveyed to the model by assimilating cloudy pixels. Relying on the error estimates provided together with the products, no further adjustments to the observation error matrix was made here. The experiments gradually included observations with stronger and stronger cloud signal as indicated by the OmC indicator (OBS-CALC estimates retrieval in window channels).

Outcome:

Further experiments could explore the benefits of inflating the observation error for the cloudiest scenes.

• Including some cloudy retrievals further improves the forecast’s performance.
• The uncertainty estimates in cloudy pixels appears reliable and functional.
• Assimilating the cloudiest retrievals starts degrading the forecasts.

Increasing spatial density, with intelligent data thinning

In the nominal operational baseline, only 1 pixel in a 80km x 80km box is currently assimilated. The objectives here are to increase the density of IASI pixels being assimilated, basing data screening on L2 quality indicators.

Ongoing

Full system, assimilating maritime all-sky retrievals

In these experiments, the assimilation of L1 and L2 will be repeated in a full environment (i.e all input data except IASI), using the optimised configurations resulting from the activities above. The impact of IASI in a full assimilation environment will be more limited (whether L1 or L2). The objective is to confirm the positive impact of L2 and the conclusions obtained in a depleted environment relative to assimilating L1 radiances.

To follow

Conclusions

✓ The assimilation of L2 profiles with legitimate mathematical representation (PC) and expression of scene-dependent observation operator (error-normalised averaging kernels) is a functional strategy in an operational NWP system.
✓ The assimilation of temperature and humidity profiles has a clear positive impact on both temperature and humidity forecasts.
✓ L2 positive impact is lower but close to L1 in the free troposphere and stratosphere for temperature and humidity.
✓ L2 impact is equivalent to L1 for temperature in the lower troposphere.
✓ L2 impact is superior to L1 for humidity in the lower troposphere.

Outlook

The assimilation of L2 has the potential to effectively convey the same or more information into an NWP model than through radiances, especially in (regional) systems where stringent trade-offs exist between timeliness requirements, computational costs and the need to manage increasing amount of satellite data.

Further activities building on these first series of experiments over ocean are ongoing or are planned in order to optimise the assimilation settings, extend learning and feed back into L2 developments:

• Tuning the observation error.
• Assimilating cloudy retrievals.
• Increasing the density of IASI L2 profiles conveyed to the system.
• Evaluate the impact in a full assimilation environment.
• Evaluate the impact over land.
• Evaluate the most effective technical solutions to transmit profiles and scene-dependent observation operators to the system.