Dataset Information

Exploring Changes to the Actionability of COVID-19 Dashboards Over the Course of 2020 in the Canadian Context: Descriptive Assessment and Expert Appraisal Study.

ABSTRACT:

Background

Public web-based COVID-19 dashboards are in use worldwide to communicate pandemic-related information. Actionability of dashboards, as a predictor of their potential use for data-driven decision-making, was assessed in a global study during the early stages of the pandemic. It revealed a widespread lack of features needed to support actionability. In view of the inherently dynamic nature of dashboards and their unprecedented speed of creation, the evolution of dashboards and changes to their actionability merit exploration.

Objective

We aimed to explore how COVID-19 dashboards evolved in the Canadian context during 2020 and whether the presence of actionability features changed over time.

Methods

We conducted a descriptive assessment of a pan-Canadian sample of COVID-19 dashboards (N=26), followed by an appraisal of changes to their actionability by a panel of expert scorers (N=8). Scorers assessed the dashboards at two points in time, July and November 2020, using an assessment tool informed by communication theory and health care performance intelligence. Applying the nominal group technique, scorers were grouped in panels of three, and evaluated the presence of the seven defined features of highly actionable dashboards at each time point.

Results

Improvements had been made to the dashboards over time. These predominantly involved data provision (specificity of geographic breakdowns, range of indicators reported, and explanations of data sources or calculations) and advancements enabled by the technologies employed (customization of time trends and interactive or visual chart elements). Further improvements in actionability were noted especially in features involving local-level data provision, time-trend reporting, and indicator management. No improvements were found in communicative elements (clarity of purpose and audience), while the use of storytelling techniques to narrate trends remained largely absent from the dashboards.

Conclusions

Improvements to COVID-19 dashboards in the Canadian context during 2020 were seen mostly in data availability and dashboard technology. Further improving the actionability of dashboards for public reporting will require attention to both technical and organizational aspects of dashboard development. Such efforts would include better skill-mixing across disciplines, continued investment in data standards, and clearer mandates for their developers to ensure accountability and the development of purpose-driven dashboards.

SUBMITTER: Barbazza E

PROVIDER: S-EPMC8360335 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundSince the outbreak of COVID-19, the development of dashboards as dynamic, visual tools for communicating COVID-19 data has surged worldwide. Dashboards can inform decision-making and support behavior change. To do so, they must be actionable. The features that constitute an actionable dashboard in the context of the COVID-19 pandemic have not been rigorously assessed.ObjectiveThe aim of this study is to explore the characteristics of public web-based COVID-19 dashboards by assessing their purpose and users ("why"), content and data ("what"), and analyses and displays ("how" they communicate COVID-19 data), and ultimately to appraise the common features of highly actionable dashboards.MethodsWe conducted a descriptive assessment and scoring using nominal group technique with an international panel of experts (n=17) on a global sample of COVID-19 dashboards in July 2020. The sequence of steps included multimethod sampling of dashboards; development and piloting of an assessment tool; data extraction and an initial round of actionability scoring; a workshop based on a preliminary analysis of the results; and reconsideration of actionability scores followed by joint determination of common features of highly actionable dashboards. We used descriptive statistics and thematic analysis to explore the findings by research question.ResultsA total of 158 dashboards from 53 countries were assessed. Dashboards were predominately developed by government authorities (100/158, 63.0%) and were national (93/158, 58.9%) in scope. We found that only 20 of the 158 dashboards (12.7%) stated both their primary purpose and intended audience. Nearly all dashboards reported epidemiological indicators (155/158, 98.1%), followed by health system management indicators (85/158, 53.8%), whereas indicators on social and economic impact and behavioral insights were the least reported (7/158, 4.4% and 2/158, 1.3%, respectively). Approximately a quarter of the dashboards (39/158, 24.7%) did not report their data sources. The dashboards predominately reported time trends and disaggregated data by two geographic levels and by age and sex. The dashboards used an average of 2.2 types of displays (SD 0.86); these were mostly graphs and maps, followed by tables. To support data interpretation, color-coding was common (93/158, 89.4%), although only one-fifth of the dashboards (31/158, 19.6%) included text explaining the quality and meaning of the data. In total, 20/158 dashboards (12.7%) were appraised as highly actionable, and seven common features were identified between them. Actionable COVID-19 dashboards (1) know their audience and information needs; (2) manage the type, volume, and flow of displayed information; (3) report data sources and methods clearly; (4) link time trends to policy decisions; (5) provide data that are "close to home"; (6) break down the population into relevant subgroups; and (7) use storytelling and visual cues.ConclusionsCOVID-19 dashboards are diverse in the why, what, and how by which they communicate insights on the pandemic and support data-driven decision-making. To leverage their full potential, dashboard developers should consider adopting the seven actionability features identified.

Project description:BackgroundWeb-based public reporting by means of dashboards has become an essential tool for governments worldwide to monitor COVID-19 information and communicate it to the public. The actionability of such dashboards is determined by their fitness for purpose-meeting a specific information need-and fitness for use-placing the right information into the right hands at the right time and in a manner that can be understood.ObjectiveThe aim of this study was to identify specific areas where the actionability of the Dutch government's COVID-19 dashboard could be improved, with the ultimate goal of enhancing public understanding of the pandemic.MethodsThe study was conducted from February 2020 to April 2021. A mixed methods approach was carried out, using (1) a descriptive checklist over time to monitor changes made to the dashboard, (2) an actionability scoring of the dashboard to pinpoint areas for improvement, and (3) a reflection meeting with the dashboard development team to contextualize findings and discuss areas for improvement.ResultsThe dashboard predominantly showed epidemiological information on COVID-19. It had been developed and adapted by adding more in-depth indicators, more geographic disaggregation options, and new indicator themes. It also changed in target audience from policy makers to the general public; thus, a homepage was added with the most important information, using news-like items to explain the provided indicators and conducting research to enhance public understanding of the dashboard. However, disaggregation options such as sex, socioeconomic status, and ethnicity and indicators on dual-track health system management and social and economic impact that have proven to give important insights in other countries are missing from the Dutch COVID-19 dashboard, limiting its actionability.ConclusionsThe Dutch COVID-19 dashboard developed over time its fitness for purpose and use in terms of providing epidemiological information to the general public as a target audience. However, to strengthen the Dutch health system's ability to cope with upcoming phases of the COVID-19 pandemic or future public health emergencies, we advise (1) establishing timely indicators relating to health system capacity, (2) including relevant data disaggregation options (eg, sex, socioeconomic status), and (3) enabling interoperability between social, health, and economic data sources.

Project description:Introduction: An essential aim of courses in evidence-based medicine (EBM) is to improve the skills for reading and interpreting medical literature adequately. Regarding the conceptual framework, it is important to consider different educational levels. Aim: Our primary aim was to investigate the applicability of different instruments for the assessment of methodological study quality by 3rd grade students after short courses in EBM. Our secondary outcomes were agreement with expert assessments and student's knowledge and competences. Methods: We conducted four short courses in EBM of 90 minutes each for health care management and medical students focused on critical appraisal of the literature. At the end, the students assessed five publications about randomized controlled trials (RCTs) using five different instruments; the results were compared to expert assessments. Results: In total, 167 students participated in our EBM courses. Students' assessments showed a non-systematic over- and underestimation of risk of bias compared to expert assessments with no clear direction. Agreement with expert assessments ranged between 66% to over 80%. Across RCTs, evidence was found that the choice of instrument had an impact on agreement rates between expert and student assessments (p=0.0158). Three RCTs showed an influence of the instrument on the agreement rate (p<0.05 each). Discussion: Our results contrast sharply with those of many other comparable evaluations. Reasons may be a lack of students' motivation due to the compulsory courses, and the comparison to a reference standard in addition to self-ratings causing objectivity. Conclusion: Undergraduates should become familiar with the principles of EBM, including research methods, and the reading of scientific papers as soon as possible. For a deeper understanding, clinical experience seems to be an indispensable precondition. Based on our results, we would recommend an integration of lectures about EBM and critical appraisal at least twice during studies and with greater intensity shortly before graduation.