Dataset Information

Leveraging routine clinical materials and mobile technology to assess CBT fidelity: the Innovative Methods to Assess Psychotherapy Practices (imAPP) study.

ABSTRACT: BACKGROUND:Identifying scalable strategies for assessing fidelity is a key challenge in implementation science. However, for psychosocial interventions, the existing, reliable ways to test treatment fidelity quality are often labor intensive, and less burdensome strategies may not reflect actual clinical practice. Cognitive behavioral therapies (CBTs) provide clinicians with a set of effective core elements to help treat a multitude of disorders, which, evidence suggests, need to be delivered with fidelity to maximize potential client impact. The current "gold standard" for rating CBTs is rating recordings of therapy sessions, which is extremely time-consuming and requires a substantial amount of initial training. Although CBTs can vary based on the target disorder, one common element employed in most CBTs is the use of worksheets to identify specific behaviors and thoughts that affect a client's ability to recover. The present study will develop and evaluate an innovative new approach to rate CBT fidelity, by developing a universal CBT scoring system based on worksheets completed in therapy sessions. METHODS:To develop a scoring system for CBT worksheets, we will compile common CBT elements from a variety of CBT worksheets for a range of psychiatric disorders and create adherence and competence measures. We will collect archival worksheets from past studies to test the scoring system and assess test-retest reliability. To evaluate whether CBT worksheet scoring accurately reflects clinician fidelity, we will recruit clinicians who are engaged in a CBT for depression, anxiety, and/or posttraumatic stress disorder. Clinicians and clients will transmit routine therapy materials produced in session (e.g., worksheets, clinical notes, session recordings) to the study team after each session. We will compare observer-rated fidelity, clinical notes, and fidelity-rated worksheets to identify the most effective and efficient method to assess clinician fidelity. Clients will also be randomly assigned to either complete the CBT worksheets on paper forms or on a mobile application (app) to learn if worksheet format influences clinician and client experience or differs in terms of reflecting fidelity. DISCUSSION:Scoring fidelity using CBT worksheets may allow clinics to test fidelity in a short and effective manner, enhancing continuous quality improvement in the workplace. Clinicians and clinics can use such data to improve clinician fidelity in real time, leading to improved patient outcomes. TRIAL REGISTRATION:ClinicalTrials.gov NCT03479398 . Retrospectively registered March 20, 2018.

SUBMITTER: Wiltsey Stirman S

PROVIDER: S-EPMC5964900 | biostudies-literature | 2018 May

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Leveraging routine clinical materials and mobile technology to assess CBT fidelity: the Innovative Methods to Assess Psychotherapy Practices (imAPP) study.

Wiltsey Stirman Shannon S Marques Luana L Creed Torrey A TA Gutner Cassidy A CA DeRubeis Robert R Barnett Paul G PG Kuhn Eric E Suvak Michael M Owen Jason J Vogt Dawne D Jo Booil B Schoenwald Sonja S Johnson Clara C Mallard Kera K Beristianos Matthew M La Bash Heidi H

Implementation science : IS 20180522 1

<h4>Background</h4>Identifying scalable strategies for assessing fidelity is a key challenge in implementation science. However, for psychosocial interventions, the existing, reliable ways to test treatment fidelity quality are often labor intensive, and less burdensome strategies may not reflect actual clinical practice. Cognitive behavioral therapies (CBTs) provide clinicians with a set of effective core elements to help treat a multitude of disorders, which, evidence suggests, need to be deli ...[more]

PMID: 29789017

Similar Datasets

Project description:BACKGROUND:Even though 1 in 5 Americans experience some form of mental illness each year, 80% have been shown to discontinue psychotherapy prematurely. The traditional psychotherapy service delivery model, consisting of isolated clinical sessions, lacks the ability to keep patients engaged outside clinical sessions. Newer digital mental health platforms can address the clinical need for a robust tool that tracks mental well-being and improves engagement in patients with depressive symptoms. OBJECTIVE:The primary goals of this feasibility study were to (1) assess compliance among providers and their patients with a digital mental health platform protocol, and (2) examine the usability and fidelity of a mobile app through structured participant feedback. METHODS:A sample of 30 participants was recruited for a 5-week study from a community-based mental health clinic in Baltimore, Maryland, USA. Inclusion criteria were: aged 18 years or older, having access to a smartphone, and having at least mild-to-moderate depression and/or anxiety as measured by the Patient Health Questionnaire-9 (PHQ-9) and Generalized Anxiety Disorder-7 (GAD-7) scales, respectively. Eligible participants were randomized into one of two study arms: (1) the intervention arm or (2) the waitlist control arm. Participants in the intervention arm were asked to download the Rose app and were prompted to complete clinical assessments (PHQ-9 and GAD-7) every other week, daily mood and anxiety Likert scales, and daily journal entries. The participants in the waitlist arm served as controls for the study and completed the clinical assessments only. Both arms engaged in weekly psychotherapy sessions, with participant in-app input informing the psychotherapy process of the intervention arm, while those in the waitlist control arm continued their standard care. Outcomes of interest included adherence to completion of in-app assessments and usability of the Rose mobile app assessed through the modified Mobile Application Rating Scale. RESULTS:Over the study period, a sample of 30 participants used the Rose app 2834 times to complete clinical assessments. On average, 70% (21; 95% CI 61.14%-77.41%) of participants completed mood and anxiety daily check-ins and journal entries 5 days per week. Nearly all participants (29/30, 97%) completed all PHQ-9 and GAD-7 in-app scales during the study. Subjective impressions showed that 73% (22/30) of participants found the mobile app to be engaging and in line with their needs, and approximately 70% (21/30) of participants reported the app functionality and quality of information to be excellent. Additionally, more than two-thirds of the participants felt that their knowledge and awareness of depression and anxiety management improved through using the app. CONCLUSIONS:Steady compliance and high app ratings showcase the utility of the Rose mobile mental health app in augmenting the psychotherapy process for patients with mood disorders and improving mental health knowledge. Future studies are needed to further examine the impact of Rose on treatment outcomes. TRIAL REGISTRATION:ClinicalTrials.gov NCT04200170; https://clinicaltrials.gov/ct2/show/NCT04200170.

Project description:Rationale: As donor contributions for HIV/AIDS stagnate globally, national governments must seek ways to improve use of existing resources through interventions to drive efficiency at the facility level. But program managers lack routine information on unit expenditures at points of care, and higher-level planners are unable to assess resource use in the health system. Thus, managers cannot measure current levels of technical efficiency, and are unable to evaluate effectiveness of interventions to increase technical efficiency. Phased Implementation of REMS: FHI 360 developed the Routine Efficiency Monitoring System (REMS)-a relational database leveraging existing budget, expenditure and output data to produce quarterly site-level estimates of unit expenditure per service. Along with the Government of the Republic of Zambia (GRZ) and implementation partner Avencion, we configured REMS to measure technical efficiency of Ministry of Health resources used to deliver HIV/AIDS services in 326 facilities in 17 high-priority districts in Copperbelt and Central Provinces. REMS allocation algorithms were developed through facility assessments, and key informant interviews with MoH staff. Existing IFMIS and DHIS-2 data streams provide recurring flows of expenditure and output data needed to estimate service-specific unit expenditures. Trained users access REMS output through user-friendly dashboards delivered through a web-based application. REMS as a Solution: District health managers use REMS to identify "outlier" facilities to test performance improvement interventions. Provincial and national planners are using REMS to seek savings and ensure that resources are directed to geographic and programmatic areas with highest need. REMS can support reimbursement for social health insurance and provide time-series data on facility-level costs for modeling. Conclusions and Next Steps: REMS gives managers and planners substantially-improved data on how programs transform resources into services. The GRZ is seeking funding to expand REMS nationally, covering all major disease areas. Improved technical efficiency supports the goal of a sustainable HIV/AIDS response.

Project description:BackgroundBehavioral interventions such as psychotherapy are leading, evidence-based practices for a variety of problems (e.g., substance abuse), but the evaluation of provider fidelity to behavioral interventions is limited by the need for human judgment. The current study evaluated the accuracy of statistical text classification in replicating human-based judgments of provider fidelity in one specific psychotherapy--motivational interviewing (MI).MethodParticipants (n?=?148) came from five previously conducted randomized trials and were either primary care patients at a safety-net hospital or university students. To be eligible for the original studies, participants met criteria for either problematic drug or alcohol use. All participants received a type of brief motivational interview, an evidence-based intervention for alcohol and substance use disorders. The Motivational Interviewing Skills Code is a standard measure of MI provider fidelity based on human ratings that was used to evaluate all therapy sessions. A text classification approach called a labeled topic model was used to learn associations between human-based fidelity ratings and MI session transcripts. It was then used to generate codes for new sessions. The primary comparison was the accuracy of model-based codes with human-based codes.ResultsReceiver operating characteristic (ROC) analyses of model-based codes showed reasonably strong sensitivity and specificity with those from human raters (range of area under ROC curve (AUC) scores: 0.62 - 0.81; average AUC: 0.72). Agreement with human raters was evaluated based on talk turns as well as code tallies for an entire session. Generated codes had higher reliability with human codes for session tallies and also varied strongly by individual code.ConclusionTo scale up the evaluation of behavioral interventions, technological solutions will be required. The current study demonstrated preliminary, encouraging findings regarding the utility of statistical text classification in bridging this methodological gap.

Dataset Information

Leveraging routine clinical materials and mobile technology to assess CBT fidelity: the Innovative Methods to Assess Psychotherapy Practices (imAPP) study.

Publications

Leveraging routine clinical materials and mobile technology to assess CBT fidelity: the Innovative Methods to Assess Psychotherapy Practices (imAPP) study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets