Dataset Information

Statistical and Machine Learning forecasting methods: Concerns and ways forward.

ABSTRACT: Machine Learning (ML) methods have been proposed in the academic literature as alternatives to statistical ones for time series forecasting. Yet, scant evidence is available about their relative performance in terms of accuracy and computational requirements. The purpose of this paper is to evaluate such performance across multiple forecasting horizons using a large subset of 1045 monthly time series used in the M3 Competition. After comparing the post-sample accuracy of popular ML methods with that of eight traditional statistical ones, we found that the former are dominated across both accuracy measures used and for all forecasting horizons examined. Moreover, we observed that their computational requirements are considerably greater than those of statistical methods. The paper discusses the results, explains why the accuracy of ML models is below that of statistical ones and proposes some possible ways forward. The empirical results found in our research stress the need for objective and unbiased ways to test the performance of forecasting methods that can be achieved through sizable and open competitions allowing meaningful comparisons and definite conclusions.

SUBMITTER: Makridakis S

PROVIDER: S-EPMC5870978 | biostudies-literature | 2018

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Statistical and Machine Learning forecasting methods: Concerns and ways forward.

Makridakis Spyros S Spiliotis Evangelos E Assimakopoulos Vassilios V

PloS one 20180327 3

Machine Learning (ML) methods have been proposed in the academic literature as alternatives to statistical ones for time series forecasting. Yet, scant evidence is available about their relative performance in terms of accuracy and computational requirements. The purpose of this paper is to evaluate such performance across multiple forecasting horizons using a large subset of 1045 monthly time series used in the M3 Competition. After comparing the post-sample accuracy of popular ML methods with ...[more]

PMID: 29584784

Similar Datasets

Project description:ImportanceForecasting the volume of hospital discharges has important implications for resource allocation and represents an opportunity to improve patient safety at periods of elevated risk.ObjectiveTo determine the performance of a new time-series machine learning method for forecasting hospital discharge volume compared with simpler methods.DesignA retrospective cohort study of daily hospital discharge volumes at 2 large, New England academic medical centers between January 1, 2005, and December 31, 2014 (hospital 1), or January 1, 2005, and December 31, 2010 (hospital 2), comparing time-series forecasting methods for prediction was performed. Data analysis was conducted from February 28, 2017, to August 30, 2018. Group-level data for all discharges from inpatient units were included. In addition to conventional methods, a technique originally developed for allocating data center resources, and comparison strategies for incorporating prior data and frequency of model updates, was conducted to identify the model application that optimized forecast accuracy.Main outcomes and measuresModel calibration as measured by R2 and, secondarily, number of days with errors greater than 1 SD of daily volume.ResultsDuring the forecasted year, hospital 1 had 54 411 discharges (daily mean, 149) and hospital 2 had 47 456 discharges (daily mean, 130). The machine learning method was well calibrated at both sites (R2, 0.843 and 0.726, respectively) and made errors greater than 1 SD of daily volume on only 13 and 22 days, respectively, of the forecast year at the 2 sites. Last-value-carried-forward models performed somewhat less well (calibration R2, 0.781 and 0.596, respectively) with 13 and 46 errors of 1 SD or greater, respectively. More frequent retraining and training sets of longer than 1 year had minimal effects on the machine learning method's performance.Conclusions and relevanceVolume of hospital discharges can perhaps be reliably forecasted using simple carry-forward models as well as methods drawn from machine learning. The benefit of the latter does not appear to be dependent on extensive training data and may enable forecasts up to 1 year in advance with superior absolute accuracy to carry-forward models.

Dataset Information

Statistical and Machine Learning forecasting methods: Concerns and ways forward.

Publications

Statistical and Machine Learning forecasting methods: Concerns and ways forward.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets