Dataset Information

Developing a framework for evidence-based grading and assessment of predictive tools for clinical decision support.

ABSTRACT:

Background

Clinical predictive tools quantify contributions of relevant patient characteristics to derive likelihood of diseases or predict clinical outcomes. When selecting predictive tools for implementation at clinical practice or for recommendation in clinical guidelines, clinicians are challenged with an overwhelming and ever-growing number of tools, most of which have never been implemented or assessed for comparative effectiveness. To overcome this challenge, we have developed a conceptual framework to Grade and Assess Predictive tools (GRASP) that can provide clinicians with a standardised, evidence-based system to support their search for and selection of efficient tools.

Methods

A focused review of the literature was conducted to extract criteria along which tools should be evaluated. An initial framework was designed and applied to assess and grade five tools: LACE Index, Centor Score, Well's Criteria, Modified Early Warning Score, and Ottawa knee rule. After peer review, by six expert clinicians and healthcare researchers, the framework and the grading of the tools were updated.

Results

GRASP framework grades predictive tools based on published evidence across three dimensions: 1) Phase of evaluation; 2) Level of evidence; and 3) Direction of evidence. The final grade of a tool is based on the highest phase of evaluation, supported by the highest level of positive evidence, or mixed evidence that supports a positive conclusion. Ottawa knee rule had the highest grade since it has demonstrated positive post-implementation impact on healthcare. LACE Index had the lowest grade, having demonstrated only pre-implementation positive predictive performance.

Conclusion

GRASP framework builds on widely accepted concepts to provide standardised assessment and evidence-based grading of predictive tools. Unlike other methods, GRASP is based on the critical appraisal of published evidence reporting the tools' predictive performance before implementation, potential effect and usability during implementation, and their post-implementation impact. Implementing the GRASP framework as an online platform can enable clinicians and guideline developers to access standardised and structured reported evidence of existing predictive tools. However, keeping GRASP reports up-to-date would require updating tools' assessments and grades when new evidence becomes available, which can only be done efficiently by employing semi-automated methods for searching and processing the incoming information.

SUBMITTER: Khalifa M

PROVIDER: S-EPMC6820933 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundMany clinical predictive tools have been developed to diagnose traumatic brain injury among children and guide the use of computed tomography in the emergency department. It is not always feasible to compare tools due to the diversity of their development methodologies, clinical variables, target populations, and predictive performances. The objectives of this study are to grade and assess paediatric head injury predictive tools, using a new evidence-based approach, and to provide emergency clinicians with standardised objective information on predictive tools to support their search for and selection of effective tools.MethodsPaediatric head injury predictive tools were identified through a focused review of literature. Based on the critical appraisal of published evidence about predictive performance, usability, potential effect, and post-implementation impact, tools were evaluated using a new framework for grading and assessment of predictive tools (GRASP). A comprehensive analysis was conducted to explain why certain tools were more successful.ResultsFourteen tools were identified and evaluated. The highest-grade tool is PECARN; the only tool evaluated in post-implementation impact studies. PECARN and CHALICE were evaluated for their potential effect on healthcare, while the remaining 12 tools were only evaluated for predictive performance. Three tools; CATCH, NEXUS II, and Palchak, were externally validated. Three tools; Haydel, Atabaki, and Buchanich, were only internally validated. The remaining six tools; Da Dalt, Greenes, Klemetti, Quayle, Dietrich, and Güzel did not show sufficient internal validity for use in clinical practice.ConclusionsThe GRASP framework provides clinicians with a high-level, evidence-based, comprehensive, yet simple and feasible approach to grade, compare, and select effective predictive tools. Comparing the three main tools which were assigned the highest grades; PECARN, CHALICE and CATCH, to the remaining 11, we find that the quality of tools' development studies, the experience and credibility of their authors, and the support by well-funded research programs were correlated with the tools' evidence-based assigned grades, and were more influential, than the sole high predictive performance, on the wide acceptance and successful implementation of the tools. Tools' simplicity and feasibility, in terms of resources needed, technical requirements, and training, are also crucial factors for their success.

Project description:BackgroundWhile selecting predictive tools for implementation in clinical practice or for recommendation in clinical guidelines, clinicians and health care professionals are challenged with an overwhelming number of tools. Many of these tools have never been implemented or evaluated for comparative effectiveness. To overcome this challenge, the authors developed and validated an evidence-based framework for grading and assessment of predictive tools (the GRASP framework). This framework was based on the critical appraisal of the published evidence on such tools.ObjectiveThe aim of the study was to examine the impact of using the GRASP framework on clinicians' and health care professionals' decisions in selecting clinical predictive tools.MethodsA controlled experiment was conducted through a web-based survey. Participants were randomized to either review the derivation publications, such as studies describing the development of the predictive tools, on common traumatic brain injury predictive tools (control group) or to review an evidence-based summary, where each tool had been graded and assessed using the GRASP framework (intervention group). Participants in both groups were asked to select the best tool based on the greatest validation or implementation. A wide group of international clinicians and health care professionals were invited to participate in the survey. Task completion time, rate of correct decisions, rate of objective versus subjective decisions, and level of decisional conflict were measured.ResultsWe received a total of 194 valid responses. In comparison with not using GRASP, using the framework significantly increased correct decisions by 64%, from 53.7% to 88.1% (88.1/53.7=1.64; t193=8.53; P<.001); increased objective decision making by 32%, from 62% (3.11/5) to 82% (4.10/5; t189=9.24; P<.001); decreased subjective decision making based on guessing by 20%, from 49% (2.48/5) to 39% (1.98/5; t188=-5.47; P<.001); and decreased prior knowledge or experience by 8%, from 71% (3.55/5) to 65% (3.27/5; t187=-2.99; P=.003). Using GRASP significantly decreased decisional conflict and increased the confidence and satisfaction of participants with their decisions by 11%, from 71% (3.55/5) to 79% (3.96/5; t188=4.27; P<.001), and by 13%, from 70% (3.54/5) to 79% (3.99/5; t188=4.89; P<.001), respectively. Using GRASP decreased the task completion time, on the 90th percentile, by 52%, from 12.4 to 6.4 min (t193=-0.87; P=.38). The average System Usability Scale of the GRASP framework was very good: 72.5% and 88% (108/122) of the participants found the GRASP useful.ConclusionsUsing GRASP has positively supported and significantly improved evidence-based decision making. It has increased the accuracy and efficiency of selecting predictive tools. GRASP is not meant to be prescriptive; it represents a high-level approach and an effective, evidence-based, and comprehensive yet simple and feasible method to evaluate, compare, and select clinical predictive tools.

Project description:BackgroundPublic health professionals are increasingly expected to engage in evidence-informed decision making to inform practice and policy decisions. Evidence-informed decision making involves the use of research evidence along with expertise, existing public health resources, knowledge about community health issues, the local context and community, and the political climate. The National Collaborating Centre for Methods and Tools has identified a seven step process for evidence-informed decision making. Tools have been developed to support public health professionals as they work through each of these steps. This paper provides an overview of tools used in three Canadian public health departments involved in a study to develop capacity for evidence-informed decision making.MethodsAs part of a knowledge translation and exchange intervention, a Knowledge Broker worked with public health professionals to identify and apply tools for use with each of the steps of evidence-informed decision making. The Knowledge Broker maintained a reflective journal and interviews were conducted with a purposive sample of decision makers and public health professionals. This paper presents qualitative analysis of the perceived usefulness and usability of the tools.ResultsTools were used in the health departments to assist in: question identification and clarification; searching for the best available research evidence; assessing the research evidence for quality through critical appraisal; deciphering the 'actionable message(s)' from the research evidence; tailoring messages to the local context to ensure their relevance and suitability; deciding whether and planning how to implement research evidence in the local context; and evaluating the effectiveness of implementation efforts. Decision makers provided descriptions of how the tools were used within the health departments and made suggestions for improvement. Overall, the tools were perceived as valuable for advancing and sustaining evidence-informed decision making.ConclusionTools are available to support the process of evidence-informed decision making among public health professionals. The usability and usefulness of these tools for advancing and sustaining evidence-informed decision making are discussed, including recommendations for the tools' application in other public health settings beyond this study. Knowledge and awareness of these tools may assist other health professionals in their efforts to implement evidence-informed practice.

Project description:BackgroundUnique characteristics of nursing homes (NHs) contribute to high rates of inappropriate antibiotic use for asymptomatic bacteriuria (ASB), a benign condition. A mobile clinical decision support system (CDSS) may support NH staff in differentiating urinary tract infections (UTI) from ASB and reducing antibiotic days.ObjectivesWe used Goal-Directed Design to: 1) Characterize information needs for UTI identification and management in NHs; 2) Develop UTI Decide, a mobile CDSS prototype informed by personas and scenarios of use constructed from Aim 1 findings; 3) Evaluate the UTI Decide prototype with NH staff.MethodsFocus groups were conducted with providers and nurses in NHs in Denver, Colorado (n= 24). Qualitative descriptive analysis was applied to focus group transcripts to identify information needs and themes related to mobile clinical decision support for UTI identification and management. Personas representing typical end users were developed; typical clinical context scenarios were constructed using information needs as goals. Usability testing was performed using cognitive walk-throughs and a think-aloud protocol.ResultsFour information needs were identified including guidance regarding resident assessment; communication with providers; care planning; and urine culture interpretation. Design of a web-based application incorporating a published decision support algorithm for evidence-based UTI diagnoses proceeded with a focus on nursing information needs during resident assessment and communication with providers. Certified nursing assistant (CNA) and registered nurse (RN) personas were constructed in 4 context scenarios with associated key path scenarios. After field testing, a high fidelity prototype of UTI Decide was completed and evaluated by potential end users. Design recommendations and content recommendations were elicited.ConclusionsGoal-Directed Design informed the development of a mobile CDSS supporting participant-identified information needs for UTI assessment and communication in NHs. Future work will include iterative deployment and evaluation of UTI Decide in NHs to decrease inappropriate use of antibiotics for suspected UTI.

Project description:IntroductionShared decision-making intends to align care provision with individuals' preferences. However, the involvement of people living with dementia in decision-making about their care varies. We aimed to co-design the EMBED-Care Framework, to enhance shared decision-making between people affected by dementia and practitioners.MethodsA theory and evidence driven co-design study was conducted, using iterative workshops, informed by a theoretical model of shared decision-making and the EMBED-Care Framework (the intervention) for person-centred holistic palliative dementia care. The intervention incorporates a holistic outcome measure for assessment and review, linked with clinical decision-support tools to support shared decision-making. We drew on the Medical Research Council (MRC) guidance for developing and evaluating complex interventions. Participants included people with dementia of any type, current or bereaved family carers and practitioners. We recruited via established dementia groups and research and clinical networks. Data were analysed using reflexive thematic analysis to explore how and when the intervention could enhance communication and shared decision-making, and the requirements for use, presented as a logic model.ResultsFive co-design workshops were undertaken with participants comprising people affected by dementia (n = 18) and practitioners (n = 36). Three themes were generated, comprising: (1) 'knowing the person and personalisation of care', involving the person with dementia and/or family carer identifying the needs of the person using a holistic assessment. (2) 'engaging and considering the perspectives of all involved in decision-making' required listening to the person and the family to understand their priorities, and to manage multiple preferences. (3) 'Training and support activities' to use the Framework through use of animated videos to convey information, such as to understand the outcome measure used to assess symptoms.ConclusionsThe intervention developed sought to enhance shared decision-making with individuals affected by dementia and practitioners, through increased shared knowledge of individual priorities and choices for care and treatment. The workshops generated understanding to manage disagreements in determining priorities. Practitioners require face-to-face training on the intervention, and on communication to manage sensitive conversations about symptoms, care and treatment with individuals and their family. The findings informed the construction of a logic model to illustrate how the intervention is intended to work.

Project description:SUPPORT tools consist of 18 articles addressing the health policy-makers so that they can learn how to make evidence-informed health policies. These tools have been particularly recommended for developing countries. The present study tries to explain the process of evidence utilization for developing policy documents in the Iranian Ministry of Health and Medical Education (MoHME) and to compare the findings with those of SUPPORT tools.A qualitative research was conducted, using the framework analysis approach. Participants consisted of senior managers and technicians in MoHME. Purposeful sampling was done, with a maximum variety, for the selection of research participants: individuals having at least 5 years of experience in preparing evidence-based policy documents. Face-to-face interviews were conducted for data collection. As a guideline for the interviews, 'the Utilization of Evidence in Policy-Making Organizations' procedure was used. The data were analyzed through the analysis of the framework method using MAXQDA 10 software.The participants acquired the research evidence in a topic-based form, and they were less likely to search on the basis of the evidence pyramid. To assess the quality of evidence, they did not use standard critical tools; to adapt the evidence and interventions with the local setting, they did not use the ideas and experiences of all stakeholders, and in preparing the evidence-based policy documents, they did not take into consideration the window of opportunity, did not refrain from using highly technical terms, did not write user-friendly summaries, and did not present alternative policy options. In order to develop health policies, however, they used the following innovations: attention to the financial burden of policy issues on the agenda, sensitivity analysis of the preferred policy option on the basis of technical, sociopolitical, and economic feasibility, advocacy from other scholars, using the multi-criteria decision-making models for the prioritization of policy options, implementation of policy based on the degree of readiness of policy-implementing units, and the classification of policy documents on the basis of different conditions of policy-making (urgent, short-term, and long-term).Findings showed that the process of evidence utilization in IR-MoH enjoys some innovations for the support of health policy development. The present study provides IR-MoH with considerable opportunities for the improvement of evidence-informed health policy-making. Moreover, the SUPPORT process and tools are recommended to be used in developing countries.