Infectious disease forecasting

A forecast is a quantitative statement about an event, outcome, or trend that has not yet been observed, conditional on prior data that has been observed.

Adapted from: Lauer, S.A. et al. 2020. Infectious disease forecasting for public health. In Population Biology of Vector-borne Diseases edited by J.M. Drake, M.B. Bonsall, and M.R. Strand. Oxford University Press.

Typically, it is assumed that forecasts are of future events, but this definition allows for “retrospective” forecasts, including nowcasts.

Probabilistic forecasts are an important special case.

Both mechanistic and statistical models may be used for forecasting.

Statistical models typically outperform mechanistic models.

Discussion question: What do you expect to be the strengths and weaknesses of these two kinds of models?

Mechanistic

• pro: can predict the effect of intervention
• con: underperform compared with statistical models

Statistical

• pro: high performing, generalizable
• con: assume stationary data-generating process

A 2014 scoping review of influenza forecasting models found that 17/35 models used compartmental models and 18/35 studies used statistical models

Chretien J-P, George D, Shaman J, Chitale RA, McKenzie FE 2014. Influenza Forecasting in Human Populations: A Scoping Review. PLoS ONE 9(4):e94130. doi:10.1371/journal.pone.0094130

• System complexity
• Data sparsity
• Behavior change
• Forecasting feedback loop

Targets are the unknown (but verifiable) quantities that forecasts make quantitative statements about (i.e. an event, outcome, or trend).

Examples include:

• Disease incidence, hospitalizations, or deaths at specific (future) times
• Peak disease incidence, hospitalizations, or deaths
• The time of peak disease incidence, hospitalizations or deaths
• A binary indicator for whether incidence, hospitalizations, or deaths exceeds a specified threshold

• Time ranges from 1 to \( T \), typically in regular intervals (days, weeks, etc.), i.e. \( t \in {1, 2, 3, ... T} \)
• Observations (\( y \)) may be indexed by time, i.e., the time series \( {y_1, y_2, y_3, ... y_T} \) assumed to be draws from random variables \( Y_1, Y_2, Y_3,... Y_T \)
• Covariates may also be indexed by time, i.e. \( x_1, x_2, x_3, ... x_T \) and may be scalar or vector-valued
• Often we are interested in the predicted value of a target at a future relative time, indicated by a lag of \( k \), i.e. the value of the target at time \( t+k \) or the \( k \)-step-ahead prediction
• Targets may or may not be indexed by time and are represented by random variables \( Z \).

Data Percent weighted influenza-like illness from the US Outpatient Influenza-like Illness Surveillance Network (wILIL)

Prediction targets: Epidemic onset, Peak height, Peak week, Duration (how long wILI remains above the 2013-2014 national baseline of 2%)

Bayes' Theorem:

\[ P(A|B) = \frac{P(B|A)P(B)}{P(B)} \]

In Bayesian Statistics, \( P(A) \) is known as the prior.

Five procedures

1. Model past seasons' epidemic curves as smoothed versions plus noise. (“quadratic trend filtering”)

2. Construct prior probability for the current season's epidemic curve by considering “transformations” of past seasons' curves.

3. Estimate what the wILI values the in recent past will be after their final revisions, using non-final wILI and Google Flu Trends data.

4. Weight possibilities for the current season's epidemic curve using estimates of final revised wILI.

5. Calculate forecasting targets for each possibility; report results.

Data

• Center for Systems Science and Engineering (Johns Hopkins University) COVID-19 cases and deaths
• healthdata.gov COVID-19 Reported Patient Impact and Hospital Capacity by State
• Time spent in residential areas according to Google community mobility reports
• Vaccine uptake (December 2020 to May 2021)

Prediction targets:

• 1-28 day ahead hospital admissions
• 1-4 week ahead cases and deaths

Flow diagram of differential equations for expected values of state variables.

• Local linearization (van Kampen expansion) supplies a differential equation for the covariance in fluctuations

• Estimation using extended Kalman filter

Ensemble models (commonly used in weather forecasting) combine the forecasts of multiple models into a single forecast, e.g. by averaging the individual model predictions weighted by past performance

A forecasting system comprises an online algorithm to estimate \( f \), a target prediction function to transform \( f \) into the desired form for output (e.g. the probability that the target will be within a particular range, referred to as bins), and an estimate of performance.

George, D.B. et al. 2019. Technology to advance infectious disease forecasting for outbreak management. Nature Communications 10:3932 https://doi.org/10.1038/s41467-019-11901

Evaluation metrics should be defined and finite for all conceivable applications.

Forecasts should be evaluated using proper scoring rules.

Forecast accuracy should be evaluated using out-of-sample observations.

Log score: \( LogS = \frac{1}{T} \sum_{t=1}^T log \hat p (z_t) \)

Continuous ranked probability score: \( CRPS = \int \left( (F(z)) - \mathcal{H}(z-y) \right)^2 dz \) where \( F(z) \) is the forecasted cumulative distribution function and

\[ \mathcal{H}(z) = \begin{cases} 1,& \text{if } z > 1\\ 0, & z \leq 1 \end{cases} \]

is known as the Heaviside function. The CRPS generalizes the concept of the mean squared error of a point prediction to the case of the function-valued probabilistic forecast.

To evaluate performance over multiple observations, the scores may be averaged, e.g.

\( \frac{1}{T} \sum_{t=1}^T log \hat p (z_t) \).

1. Data split into model training and testing sequences
2. Tuning typically performed via cross-validation (but this can be tricky with time series data)

1. Infectious disease forecasting is an active area of research
2. Models optimized for forecasting may be different to those optimized for inference
3. The structure of the forecasting problem creates specific problems for model construction, estimation, and evaluation
4. Semiparametric models are a promising approach to navigating the bias-variance tradeoff

Brooks LC, Farrow DC, Hyun S, TibshiraniRJ, Rosenfeld R. 2015. Flexible modeling of epidemics with an empirical Bayes framework. PLoS Computational Biology 11:e1004382.

Gibson GC, Moran KR, Reich NG, Osthus D. 2021. Improving probabilistic infectious disease forecasting through coherence. PLoS Computational Biology 17:e1007623.

O'Dea EB, Drake JM. 2022. A semi-parametric, state-space compartmental model with time-dependent parameters for forecasting COVID-19 cases, hospitalizations and deaths. Journal of the Royal Society Interface 19:20210702.

Licensed under the Creative Commons attribution-noncommercial license, http://creativecommons.org/licenses/bync/3.0/. Please share and remix noncommercially, mentioning its origin.