Dataset Information

The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability.

ABSTRACT:

Background

Implementation strategies have flourished in an effort to increase integration of research evidence into clinical practice. Most strategies are complex, socially mediated processes. Many are complicated, expensive, and ultimately impractical to deliver in real-world settings. The field lacks methods to assess the extent to which strategies are usable and aligned with the needs and constraints of the individuals and contexts who will deliver or receive them. Drawn from the field of human-centered design, cognitive walkthroughs are an efficient assessment method with potential to identify aspects of strategies that may inhibit their usability and, ultimately, effectiveness. This article presents a novel walkthrough methodology for evaluating strategy usability as well as an example application to a post-training consultation strategy to support school mental health clinicians to adopt measurement-based care.

Method

The Cognitive Walkthrough for Implementation Strategies (CWIS) is a pragmatic, mixed-methods approach for evaluating complex, socially mediated implementation strategies. CWIS includes six steps: (1) determine preconditions; (2) hierarchical task analysis; (3) task prioritization; (4) convert tasks to scenarios; (5) pragmatic group testing; and (6) usability issue identification, classification, and prioritization. A facilitator conducted two group testing sessions with clinician users (N = 10), guiding participants through 6 scenarios and 11 associated subtasks. Clinicians reported their anticipated likelihood of completing each subtask and provided qualitative justifications during group discussion. Following the walkthrough sessions, users completed an adapted quantitative assessment of strategy usability.

Results

Average anticipated success ratings indicated substantial variability across participants and subtasks. Usability ratings (scale 0-100) of the consultation protocol averaged 71.3 (SD = 10.6). Twenty-one usability problems were identified via qualitative content analysis with consensus coding, and classified by severity and problem type. High-severity problems included potential misalignment between consultation and clinical service timelines as well as digressions during consultation processes.

Conclusions

CWIS quantitative usability ratings indicated that the consultation protocol was at the low end of the "acceptable" range (based on norms from the unadapted scale). Collectively, the 21 resulting usability issues explained the quantitative usability data and provided specific direction for usability enhancements. The current study provides preliminary evidence for the utility of CWIS to assess strategy usability and generate a blueprint for redesign.

SUBMITTER: Lyon AR

PROVIDER: S-EPMC8285864 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:Implementation science's focus on establishing implementation strategy effectiveness has overshadowed the need to understand differential performance of such strategies under various conditions. Methods allowing for assessment between implementation context and process can help address this gap. This paper provides a detailed description of a mixed method procedure for assessing factors related to the implementation context and process intersection, which was developed as part of the pilot study of the Housing First Technical Assistance and Training (HFTAT) Program, a multifaceted strategy designed to support Housing First model implementation. The HFTAT was pilot tested among a sample of three organizations. Our mixed method approach combines two tools often used in implementation research-the Stages of Implementation Completion and the Consolidated Framework for Implementation Research-in a novel way. Several stages to analysis were completed, starting with a separate analysis of data pertaining to each measure and then two levels of mixed method analysis. The approach provided a better understanding of the issues that impacted the implementation guided by the HFTAT, suggesting: (1) individual determinants seemed to have a bigger impact based on the number of SIC phases they affected, (2) implementation context and process were connected through climate-related factors in the inner setting that made the sites more or less responsive to addressing identified barriers, and (3) there is a need to better assess context factors to identify areas where implementation drivers should be better targeted to facilitate change, and this is supported by prior research. Understanding the underlying factors impacting a setting's performance related to a specific implementation strategy has potential to improve decision-making and optimize future implementation efforts. The approach likely be as successful combining the SIC with other determinant frameworks and should be utilized at the onset of an implementation project to maximize its usefulness.

Project description:BackgroundPeople often prefer evidence-based psychosocial interventions (EBPIs) for mental health care; however, these interventions frequently remain unavailable to people in nonspecialty or integrated settings, such as primary care and schools. Previous research has suggested that usability, a concept from human-centered design, could support an understanding of the barriers to and facilitators of the successful adoption of EBPIs and support the redesign of EBPIs and implementation strategies.ObjectiveThis study aimed to identify and categorize usability issues in EBPIs and their implementation strategies.MethodsWe adapted a usability issue analysis and reporting format from a human-centered design. A total of 13 projects supported by the National Institute of Mental Health-funded Accelerating the Reach and Impact of Treatments for Youth and Adults with Mental Illness Center at the University of Washington used this format to describe usability issues for EBPIs and implementation strategies with which they were working. Center researchers used iterative affinity diagramming and coding processes to identify usability issue categories. On the basis of these categories and the underlying issues, we propose heuristics for the design or redesign of EBPIs and implementation strategies.ResultsThe 13 projects reported a total of 90 usability issues, which we categorized into 12 categories, including complex and/or cognitively overwhelming, required time exceeding available time, incompatibility with interventionist preference or practice, incompatibility with existing workflow, insufficient customization to clients/recipients, intervention buy-in (value), interventionist buy-in (trust), overreliance on technology, requires unavailable infrastructure, inadequate scaffolding for client/recipient, inadequate training and scaffolding for interventionists, and lack of support for necessary communication. These issues range from minor inconveniences that affect a few interventionists or recipients to severe issues that prevent all interventionists or recipients in a setting from completing part or all of the intervention. We propose 12 corresponding heuristics to guide EBPIs and implementation strategy designers in preventing and addressing these usability issues.ConclusionsUsability issues were prevalent in the studied EBPIs and implementation strategies. We recommend using the lens of usability evaluation to understand and address barriers to the effective use and reach of EBPIs and implementation strategies.International registered report identifier (irrid)RR2-10.2196/14990.

Project description:BackgroundMore than two-thirds of youth experience trauma during childhood, and up to 1 in 5 of these youth develops posttraumatic stress symptoms that significantly impair their functioning. Although trauma-focused cognitive behavior therapy (TF-CBT) has a strong evidence base, it is rarely adopted, delivered with adequate fidelity, or evaluated in the most common setting where youth access mental health services-schools. Given that individual behavior change is ultimately required for successful implementation, even when organizational factors are firmly in place, focusing on individual-level processes represents a potentially parsimonious approach. Beliefs and Attitudes for Successful Implementation in Schools (BASIS) is a pragmatic, motivationally focused multifaceted strategy that augments training and consultation and is designed to target precise mechanisms of behavior change to produce enhanced implementation and youth clinical outcomes. This study protocol describes a hybrid type 2 effectiveness-implementation trial designed to concurrently evaluate the main effects, mediators, and moderators of both the BASIS implementation strategy on implementation outcomes and TF-CBT on youth mental health outcomes.MethodsUsing a cluster randomized controlled design, this trial will assign school-based mental health (SMH) clinicians and schools to one of three study arms: (a) enhanced treatment-as-usual (TAU), (b) attention control plus TF-CBT, or (c) BASIS+TF-CBT. With a proposed sample of 120 SMH clinicians who will each recruit 4-6 youth with a history of trauma (480 children), this project will gather data across 12 different time points to address two project aims. Aim 1 will evaluate, relative to an enhanced TAU condition, the effects of TF-CBT on identified mechanisms of change, youth mental health outcomes, and intervention costs and cost-effectiveness. Aim 2 will compare the effects of BASIS against an attention control plus TF-CBT condition on theoretical mechanisms of clinician behavior change and implementation outcomes, as well as examine costs and cost-effectiveness.DiscussionThis study will generate critical knowledge about the effectiveness and cost-effectiveness of BASIS-a pragmatic, theory-driven, and generalizable implementation strategy designed to enhance motivation-to increase the yield of evidence-based practice training and consultation, as well as the effectiveness of TF-CBT in a novel service setting.Trial registrationClinicalTrials.gov registration number NCT04451161 . Registered on June 30, 2020.

Project description:BackgroundValid and reliable measurement of implementation strategies is essential to advancing implementation science; however, this area lags behind the measurement of implementation outcomes and determinants. Clinical supervision is a promising and highly feasible implementation strategy in behavioral healthcare for which pragmatic measures are lacking. This research aimed to develop and psychometrically evaluate a pragmatic measure of clinical supervision conceptualized in terms of two broadly applicable, discrete clinical supervision techniques shown to improve providers' implementation of evidence-based psychosocial interventions-(1) audit and feedback and (2) active learning.MethodsItems were generated based on a systematic review of the literature and administered to a sample of 154 outpatient mental health clinicians serving youth and 181 community-based mental health providers serving adults. Scores were evaluated for evidence of reliability, structural validity, construct-related validity, and measurement invariance across the two samples.ResultsIn sample 1, confirmatory factor analysis (CFA) supported the hypothesized two-factor structure of scores on the Evidence-Based Clinical Supervision Strategies (EBCSS) scale (χ2=5.89, df=4, p=0.208; RMSEA=0.055, CFI=0.988, SRMR=0.033). In sample 2, CFA replicated the EBCSS factor structure and provided discriminant validity evidence relative to an established supervisory alliance measure (χ2=36.12, df=30, p=0.204; RMSEA=0.034; CFI=0.990; SRMR=0.031). Construct-related validity evidence was provided by theoretically concordant associations between EBCSS subscale scores and agency climate for evidence-based practice implementation in sample 1 (d= .47 and .55) as well as measures of the supervision process in sample 2. Multiple group CFA supported the configural, metric, and partial scalar invariance of scores on the EBCSS across the two samples.ConclusionsScores on the EBCSS provide a valid basis for inferences regarding the extent to which behavioral health providers experience audit and feedback and active learning as part of their clinical supervision in both clinic- and community-based behavioral health settings.Trial registrationClinicalTrials.gov NCT04096274 . Registered on 19 September 2019.

Project description:Regional-scale air quality models and observations at routine air quality monitoring sites are used to determine attainment/non-attainment of the ozone air quality standard in the United States. In current regulatory applications, a regional-scale air quality model is applied for a base year and a future year with reduced emissions using the same meteorological conditions as those in the base year. Because of the stochastic nature of the atmosphere, the same meteorological conditions would not prevail in the future year. Therefore, we use multi-decadal observations to develop a new method for estimating the confidence bounds for the future ozone design value (based on the 4th highest value in the daily maximum 8-hr ozone concentration time series, DM8HR) for each emission loading scenario along with the probability of the design value exceeding a given ozone threshold concentration at all monitoring sites in the contiguous United States. To this end, we spectrally decompose the observed DM8HR ozone time series covering the period from 1981 to 2014 using the Kolmogorov-Zurbenko (KZ) filter and examine the variability in the relative strengths of the short-term variations (induced by synoptic-scale weather fluctuations; referred to as synoptic component, SY) and the long-term component (dictated by changes in emissions, seasonality and other slow-changing processes such as climate change; referred to as baseline component, BL). Results indicate that combining the projected change in the ozone baseline level with the adjusted synoptic forcing in historical ozone observations enables us to provide a probabilistic assessment of the efficacy of a selected emissions control strategy in complying with the ozone standard in future years. In addition, attainment demonstration is illustrated with a real-world application of the proposed methodology by using air quality model simulations, thereby helping build confidence in the use of regional-scale air quality models for supporting regulatory policies.

Project description:BackgroundFor approximately one in five children who have social, emotional, and behavioral (SEB) challenges, accessible evidence-based prevention practices (EBPPs) are critical. In the USA, schools are the primary setting for children's SEB service delivery. Still, EBPPs are rarely adopted and implemented by front-line educators (e.g., teachers) with sufficient fidelity to see effects. Given that individual behavior change is ultimately required for successful implementation, focusing on individual-level processes holds promise as a parsimonious approach to enhance impact. Beliefs and Attitudes for Successful Implementation in Schools for Teachers (BASIS-T) is a pragmatic, multifaceted pre-implementation strategy targeting volitional and motivational mechanisms of educators' behavior change to enhance implementation and student SEB outcomes. This study protocol describes a hybrid type 3 effectiveness-implementation trial designed to evaluate the main effects, mediators, and moderators of the BASIS-T implementation strategy as applied to Positive Greetings at the Door, a universal school-based EBPP previously demonstrated to reduce student disruptive behavior and increase academic engagement.MethodsThis project uses a blocked randomized cohort design with an active comparison control (ACC) condition. We will recruit and include approximately 276 teachers from 46 schools randomly assigned to BASIS-T or ACC conditions. Aim 1 will evaluate the main effects of BASIS-T on proximal implementation mechanisms (attitudes, subjective norms, self-efficacy, intentions to implement, and maintenance self-efficacy), implementation outcomes (adoption, reach, fidelity, and sustainment), and child outcomes (SEB, attendance, discipline, achievement). Aim 2 will examine how, for whom, under what conditions, and how efficiently BASIS-T works, specifically by testing whether the effects of BASIS-T on child outcomes are (a) mediated via its putative mechanisms of behavior change, (b) moderated by teacher factors or school contextual factors, and (c) cost-effective.DiscussionThis study will provide a rigorous test of BASIS-T-a pragmatic, theory-driven, and generalizable implementation strategy designed to target theoretically-derived motivational mechanisms-to increase the yield of standard EBPP training and support strategies.Trial registrationClinicalTrials.gov ID: NCT05989568. Registered on May 30, 2023.

Project description:BackgroundLittle research has directly compared the effectiveness of implementation strategies in any setting, and we know of no prior trials directly comparing how effectively different combinations of strategies support implementation in community health centers. This paper outlines the protocol of the Study of Practices Enabling Implementation and Adaptation in the Safety Net (SPREAD-NET), a trial designed to compare the effectiveness of several common strategies for supporting implementation of an intervention and explore contextual factors that impact the strategies' effectiveness in the community health center setting.Methods/designThis cluster-randomized trial compares how three increasingly hands-on implementation strategies support adoption of an evidence-based diabetes quality improvement intervention in 29 community health centers, managed by 12 healthcare organizations. The strategies are as follows: (arm 1) a toolkit, presented in paper and electronic form, which includes a training webinar; (arm 2) toolkit plus in-person training with a focus on practice change and change management strategies; and (arm 3) toolkit, in-person training, plus practice facilitation with on-site visits. We use a mixed methods approach to data collection and analysis: (i) baseline surveys on study clinic characteristics, to explore how these characteristics impact the clinics' ability to implement the tools and the effectiveness of each implementation strategy; (ii) quantitative data on change in rates of guideline-concordant prescribing; and (iii) qualitative data on the "how" and "why" underlying the quantitative results. The outcomes of interest are clinic-level results, categorized using the Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) framework, within an interrupted time-series design with segmented regression models. This pragmatic trial will compare how well each implementation strategy works in "real-world" practices.DiscussionHaving a better understanding of how different strategies support implementation efforts could positively impact the field of implementation science, by comparing practical, generalizable methods for implementing clinical innovations in community health centers. Bridging this gap in the literature is a critical step towards the national long-term goal of effectively disseminating and implementing effective interventions into community health centers.Trial registrationClinicalTrials.gov, NCT02325531.

Project description:BackgroundEstablishing a diagnosis is a complex, iterative process involving patient data gathering, integration and interpretation. Premature closure is a fallacious cognitive tendency of closing the diagnostic process before sufficient data have been gathered. A proposed strategy to minimize premature closure is the use of a checklist to trigger metacognition (the process of monitoring one's own thinking). A number of studies have suggested the effectiveness of this strategy in classroom settings. This qualitative study examined the perception of usability of a metacognitive mnemonic checklist called TWED checklist (where the letter "T = Threat", "W = What if I am wrong? What else?", "E = Evidence" and "D = Dispositional influence") in a real clinical setting.MethodTwo categories of participants, i.e., medical doctors (n =?11) and final year medical students (Group 1, n =?5; Group 2, n =?10) participated in four separate focus group discussions. Nielsen's 5 dimensions of usability (i.e. learnability, effectiveness, memorability, errors, and satisfaction) and Pentland's narrative network were adapted as the framework to study the usability and the implementation of the checklist in a real clinical setting respectively.ResultsBoth categories (medical doctors and medical students) of participants found that the TWED checklist was easy to learn and effective in promoting metacognition. For medical student participants, items "T" and "W" were believed to be the two most useful aspects of the checklist, whereas for the doctor participants, it was item "D". Regarding its implementation, item "T" was applied iteratively, items "W" and "E" were applied when the outcomes did not turn out as expected, and item "D" was applied infrequently. The one checkpoint where all four items were applied was after the initial history taking and physical examination had been performed to generate the initial clinical impression.ConclusionA metacognitive checklist aimed to check cognitive errors may be a useful tool that can be implemented in the real clinical setting.