Dataset Information

Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States.

ABSTRACT: Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multimodel ensemble forecast that combined predictions from dozens of groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naïve baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-wk horizon three to five times larger than when predicting at a 1-wk horizon. This project underscores the role that collaboration and active coordination between governmental public-health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks.

SUBMITTER: Cramer EY

PROVIDER: S-EPMC9169655 | biostudies-literature | 2022 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States.

Cramer Estee Y EY Ray Evan L EL Lopez Velma K VK Bracher Johannes J Brennen Andrea A Castro Rivadeneira Alvaro J AJ Gerding Aaron A Gneiting Tilmann T House Katie H KH Huang Yuxin Y Jayawardena Dasuni D Kanji Abdul H AH Khandelwal Ayush A Le Khoa K Mühlemann Anja A Niemi Jarad J Shah Apurv A Stark Ariane A Wang Yijin Y Wattanachit Nutcha N Zorn Martha W MW Gu Youyang Y Jain Sansiddh S Bannur Nayana N Deva Ayush A Kulkarni Mihir M Merugu Srujana S Raval Alpan A Shingi Siddhant S Tiwari Avtansh A White Jerome J Abernethy Neil F NF Woody Spencer S Dahan Maytal M Fox Spencer S Gaither Kelly K Lachmann Michael M Meyers Lauren Ancel LA Scott James G JG Tec Mauricio M Srivastava Ajitesh A George Glover E GE Cegan Jeffrey C JC Dettwiller Ian D ID England William P WP Farthing Matthew W MW Hunter Robert H RH Lafferty Brandon B Linkov Igor I Mayo Michael L ML Parno Matthew D MD Rowland Michael A MA Trump Benjamin D BD Zhang-James Yanli Y Chen Samuel S Faraone Stephen V SV Hess Jonathan J Morley Christopher P CP Salekin Asif A Wang Dongliang D Corsetti Sabrina M SM Baer Thomas M TM Eisenberg Marisa C MC Falb Karl K Huang Yitao Y Martin Emily T ET McCauley Ella E Myers Robert L RL Schwarz Tom T Sheldon Daniel D Gibson Graham Casey GC Yu Rose R Gao Liyao L Ma Yian Y Wu Dongxia D Yan Xifeng X Jin Xiaoyong X Wang Yu-Xiang YX Chen YangQuan Y Guo Lihong L Zhao Yanting Y Gu Quanquan Q Chen Jinghui J Wang Lingxiao L Xu Pan P Zhang Weitong W Zou Difan D Biegel Hannah H Lega Joceline J McConnell Steve S Nagraj V P VP Guertin Stephanie L SL Hulme-Lowe Christopher C Turner Stephen D SD Shi Yunfeng Y Ban Xuegang X Walraven Robert R Hong Qi-Jun QJ Kong Stanley S van de Walle Axel A Turtle James A JA Ben-Nun Michal M Riley Steven S Riley Pete P Koyluoglu Ugur U DesRoches David D Forli Pedro P Hamory Bruce B Kyriakides Christina C Leis Helen H Milliken John J Moloney Michael M Morgan James J Nirgudkar Ninad N Ozcan Gokce G Piwonka Noah N Ravi Matt M Schrader Chris C Shakhnovich Elizabeth E Siegel Daniel D Spatz Ryan R Stiefeling Chris C Wilkinson Barrie B Wong Alexander A Cavany Sean S España Guido G Moore Sean S Oidtman Rachel R Perkins Alex A Kraus David D Kraus Andrea A Gao Zhifeng Z Bian Jiang J Cao Wei W Lavista Ferres Juan J Li Chaozhuo C Liu Tie-Yan TY Xie Xing X Zhang Shun S Zheng Shun S Vespignani Alessandro A Chinazzi Matteo M Davis Jessica T JT Mu Kunpeng K Pastore Y Piontti Ana A Xiong Xinyue X Zheng Andrew A Baek Jackie J Farias Vivek V Georgescu Andreea A Levi Retsef R Sinha Deeksha D Wilde Joshua J Perakis Georgia G Bennouna Mohammed Amine MA Nze-Ndong David D Singhvi Divya D Spantidakis Ioannis I Thayaparan Leann L Tsiourvas Asterios A Sarker Arnab A Jadbabaie Ali A Shah Devavrat D Della Penna Nicolas N Celi Leo A LA Sundar Saketh S Wolfinger Russ R Osthus Dave D Castro Lauren L Fairchild Geoffrey G Michaud Isaac I Karlen Dean D Kinsey Matt M Mullany Luke C LC Rainwater-Lovett Kaitlin K Shin Lauren L Tallaksen Katharine K Wilson Shelby S Lee Elizabeth C EC Dent Juan J Grantz Kyra H KH Hill Alison L AL Kaminsky Joshua J Kaminsky Kathryn K Keegan Lindsay T LT Lauer Stephen A SA Lemaitre Joseph C JC Lessler Justin J Meredith Hannah R HR Perez-Saez Javier J Shah Sam S Smith Claire P CP Truelove Shaun A SA Wills Josh J Marshall Maximilian M Gardner Lauren L Nixon Kristen K Burant John C JC Wang Lily L Gao Lei L Gu Zhiling Z Kim Myungjin M Li Xinyi X Wang Guannan G Wang Yueying Y Yu Shan S Reiner Robert C RC Barber Ryan R Gakidou Emmanuela E Hay Simon I SI Lim Steve S Murray Chris C Pigott David D Gurung Heidi L HL Baccam Prasith P Stage Steven A SA Suchoski Bradley T BT Prakash B Aditya BA Adhikari Bijaya B Cui Jiaming J Rodríguez Alexander A Tabassum Anika A Xie Jiajia J Keskinocak Pinar P Asplund John J Baxter Arden A Oruc Buse Eylul BE Serban Nicoleta N Arik Sercan O SO Dusenberry Mike M Epshteyn Arkady A Kanal Elli E Le Long T LT Li Chun-Liang CL Pfister Tomas T Sava Dario D Sinha Rajarishi R Tsai Thomas T Yoder Nate N Yoon Jinsung J Zhang Leyou L Abbott Sam S Bosse Nikos I NI Funk Sebastian S Hellewell Joel J Meakin Sophie R SR Sherratt Katharine K Zhou Mingyuan M Kalantari Rahi R Yamana Teresa K TK Pei Sen S Shaman Jeffrey J Li Michael L ML Bertsimas Dimitris D Skali Lami Omar O Soni Saksham S Tazi Bouardi Hamza H Ayer Turgay T Adee Madeline M Chhatwal Jagpreet J Dalgic Ozden O OO Ladd Mary A MA Linas Benjamin P BP Mueller Peter P Xiao Jade J Wang Yuanjia Y Wang Qinxia Q Xie Shanghong S Zeng Donglin D Green Alden A Bien Jacob J Brooks Logan L Hu Addison J AJ Jahja Maria M McDonald Daniel D Narasimhan Balasubramanian B Politsch Collin C Rajanala Samyak S Rumack Aaron A Simon Noah N Tibshirani Ryan J RJ Tibshirani Rob R Ventura Valerie V Wasserman Larry L O'Dea Eamon B EB Drake John M JM Pagano Robert R Tran Quoc T QT Ho Lam Si Tung LST Huynh Huong H Walker Jo W JW Slayton Rachel B RB Johansson Michael A MA Biggerstaff Matthew M Reich Nicholas G NG

Proceedings of the National Academy of Sciences of the United States of America 20220408 15

Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hu ...[more]

PMID: 35394862

Similar Datasets

Project description:BackgroundA combined forecast from multiple models is typically more accurate than an individual forecast, but there are few examples of studies of combining in infectious disease forecasting. We investigated the accuracy of different ways of combining interval forecasts of weekly incident and cumulative coronavirus disease-2019 (COVID-19) mortality.MethodsWe considered weekly interval forecasts, for 1- to 4-week prediction horizons, with out-of-sample periods of approximately 18 months ending on 8 January 2022, for multiple locations in the United States, using data from the COVID-19 Forecast Hub. Our comparison involved simple and more complex combining methods, including methods that involve trimming outliers or performance-based weights. Prediction accuracy was evaluated using interval scores, weighted interval scores, skill scores, ranks, and reliability diagrams.ResultsThe weighted inverse score and median combining methods performed best for forecasts of incident deaths. Overall, the leading inverse score method was 12% better than the mean benchmark method in forecasting the 95% interval and, considering all interval forecasts, the median was 7% better than the mean. Overall, the median was the most accurate method for forecasts of cumulative deaths. Compared to the mean, the median's accuracy was 65% better in forecasting the 95% interval, and 43% better considering all interval forecasts. For all combining methods except the median, combining forecasts from only compartmental models produced better forecasts than combining forecasts from all models.ConclusionsCombining forecasts can improve the contribution of probabilistic forecasting to health policy decision making during epidemics. The relative performance of combining methods depends on the extent of outliers and the type of models in the combination. The median combination has the advantage of being robust to outlying forecasts. Our results support the Hub's use of the median and we recommend further investigation into the use of weighted methods.

Project description:Imposing stricter regulations for PM2.5 has the potential to mitigate damaging health and climate change effects. Recent evidence establishing a link between exposure to air pollution and COVID-19 outcomes is one of many arguments for the need to reduce the National Ambient Air Quality Standards (NAAQS) for PM2.5. However, many studies reporting a relationship between COVID-19 outcomes and PM2.5 have been criticized because they are based on ecological regression analyses, where area-level counts of COVID-19 outcomes are regressed on area-level exposure to air pollution and other covariates. It is well known that regression models solely based on area-level data are subject to ecological bias, i.e., they may provide a biased estimate of the association at the individual-level, due to within-area variability of the data. In this paper, we augment county-level COVID-19 mortality data with a nationally representative sample of individual-level covariate information from the American Community Survey along with high-resolution estimates of PM2.5 concentrations obtained from a validated model and aggregated to the census tract for the contiguous United States. We apply a Bayesian hierarchical modeling approach to combine county-, census tract-, and individual-level data to ultimately draw inference about individual-level associations between long-term exposure to PM2.5 and mortality for COVID-19. By analyzing data prior to the Emergency Use Authorization for the COVID-19 vaccines we found that an increase of 1 μg/m3 in long-term PM2.5 exposure, averaged over the 17-year period 2000-2016, is associated with a 3.3% (95% credible interval, 2.8 to 3.8%) increase in an individual's odds of COVID-19 mortality. Code to reproduce our study is publicly available at https://github.com/NSAPH/PM_COVID_ecoinference. The results confirm previous evidence of an association between long-term exposure to PM2.5 and COVID-19 mortality and strengthen the case for tighter regulations on harmful air pollution and greenhouse gas emissions.

Dataset Information

Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States.

Publications

Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets