Dataset Information

Scaling up the evaluation of psychotherapy: evaluating motivational interviewing fidelity via statistical text classification.

ABSTRACT:

Background

Behavioral interventions such as psychotherapy are leading, evidence-based practices for a variety of problems (e.g., substance abuse), but the evaluation of provider fidelity to behavioral interventions is limited by the need for human judgment. The current study evaluated the accuracy of statistical text classification in replicating human-based judgments of provider fidelity in one specific psychotherapy--motivational interviewing (MI).

Method

Participants (n?=?148) came from five previously conducted randomized trials and were either primary care patients at a safety-net hospital or university students. To be eligible for the original studies, participants met criteria for either problematic drug or alcohol use. All participants received a type of brief motivational interview, an evidence-based intervention for alcohol and substance use disorders. The Motivational Interviewing Skills Code is a standard measure of MI provider fidelity based on human ratings that was used to evaluate all therapy sessions. A text classification approach called a labeled topic model was used to learn associations between human-based fidelity ratings and MI session transcripts. It was then used to generate codes for new sessions. The primary comparison was the accuracy of model-based codes with human-based codes.

Results

Receiver operating characteristic (ROC) analyses of model-based codes showed reasonably strong sensitivity and specificity with those from human raters (range of area under ROC curve (AUC) scores: 0.62 - 0.81; average AUC: 0.72). Agreement with human raters was evaluated based on talk turns as well as code tallies for an entire session. Generated codes had higher reliability with human codes for session tallies and also varied strongly by individual code.

Conclusion

To scale up the evaluation of behavioral interventions, technological solutions will be required. The current study demonstrated preliminary, encouraging findings regarding the utility of statistical text classification in bridging this methodological gap.

SUBMITTER: Atkins DC

PROVIDER: S-EPMC4026152 | biostudies-literature | 2014 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Scaling up the evaluation of psychotherapy: evaluating motivational interviewing fidelity via statistical text classification.

Atkins David C DC Steyvers Mark M Imel Zac E ZE Smyth Padhraic P

Implementation science : IS 20140424

<h4>Background</h4>Behavioral interventions such as psychotherapy are leading, evidence-based practices for a variety of problems (e.g., substance abuse), but the evaluation of provider fidelity to behavioral interventions is limited by the need for human judgment. The current study evaluated the accuracy of statistical text classification in replicating human-based judgments of provider fidelity in one specific psychotherapy--motivational interviewing (MI).<h4>Method</h4>Participants (n = 148) ...[more]

PMID: 24758152

Similar Datasets

Project description:BackgroundHormone receptor-positive (HR +) breast cancer is the most common type of breast cancer in the USA but has excellent long-term outcomes in recent decades, in part due to effective oral endocrine therapy (ET). ET medications are typically prescribed for 5 to 10 years, depending on the risk of recurrence, and must be taken daily. One limiting factor to ET efficacy is nonadherence, with high-risk groups for nonadherence including younger women and Black women.MethodsThe Alliance for Clinical Trials in Oncology (Alliance) trial A191901 is an ongoing, four-arm (text message reminder (TMR), motivational interviewing (MI), TMR plus MI, or enhanced usual care) randomized clinical trial that tests the efficacy and effect of two interventions (TMR and/or MI) on improved ET adherence, patient-reported outcomes (PROs), and resource use requirements among HR + breast cancer survivors. Participants are randomized in a 1:1:1:1 ratio to the four arms. With an assumed loss to follow-up of approximately 11%, we plan to recruit 1180 participants. Randomization is stratified based on age and race to ensure balance between the arms, and we oversample younger and Black women, with each group representing 30% of the study population. Participants randomized to an intervention will actively participate in the intervention for 9 months, and all participants will be followed for adherence data and PRO endpoints, through the use of the Pillsy cap medication event monitoring system and Alliance ePRO survey app (i.e., Patient Cloud). The primary analysis will compare Pillsy-measured ET adherence among study arms at 12 months.DiscussionThis multisite study will not only define strategies to improve adherence to breast cancer oral therapies, but it will also potentially support strategies in large cooperative research groups that can increase delivery and tolerability of ET, involve diverse patient populations in clinical research, and engage patients effectively in interventional studies, using remote and cost-effective delivery methods.Trial registrationClinicaltrials.gov NCT04379570 . Registered on 7 May 2020.

Project description:Background and objectiveVaccine uptake during pregnancy remains low. Our objectives were to describe 1) development and adaptation of a clinician communication training intervention for maternal immunizations and 2) obstetrics and gynecology (ob-gyn) clinician and staff perspectives on the intervention and fit for the prenatal care context.MethodsDesign of the Motivational Interviewing for Maternal Immunizations (MI4MI) intervention was based on similar communication training interventions for pediatric settings and included presumptive initiation of vaccine recommendations ("You're due for two vaccines today") combined with motivational interviewing (MI) for hesitant patients. Interviews and focus group discussions were conducted with ob-gyn clinicians and staff in five Colorado clinics including settings with obstetric physicians, certified nurse midwives (CNMs), and clinician-trainees. Participants were asked about adapting training to the ob-gyn setting and their implementation experiences. Feedback was incorporated through iterative changes to training components.ResultsInterview and focus group discussion results from participants before (n = 3), during (n = 11) and after (n = 25) implementation guided intervention development and adaptation. Three virtual, asynchronous training components were created: a video and two interactive modules. This virtual format was favored due to challenges attending group meetings; however, participants noted opportunities to practice skills through role-play were lacking. Training modules were adapted to include common challenging vaccine conversations and live-action videos. Participants liked interactive training components and use of adult learning strategies. Some participants initially resisted the presumptive approach but later found it useful after applying it in their practices. Overall, participants reported that MI4MI training fit well with the prenatal context and recommended more inclusion of non-clinician staff.ConclusionsMI4MI training was viewed as relevant and useful for ob-gyn clinicians and staff. Suggestions included making training more interactive, and including more complex scenarios and non-clinician staff.

Dataset Information

Scaling up the evaluation of psychotherapy: evaluating motivational interviewing fidelity via statistical text classification.

Background

Method

Results

Conclusion

Publications

Scaling up the evaluation of psychotherapy: evaluating motivational interviewing fidelity via statistical text classification.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets