Dataset Information

Testing e-mail content to encourage physicians to access an audit and feedback tool: a factorial randomized experiment.

ABSTRACT: Background:In Ontario, an online audit and feedback tool that provides primary care physicians with detailed information about patients who are overdue for cancer screening is underused. In the present study, we aimed to examine the effect of messages operationalizing 3 behaviour change techniques on access to the audit and feedback tool and on cancer screening rates. Methods:During May-September 2017, a pragmatic 2×2×2 factorial experiment tested 3 behaviour change techniques: anticipated regret, material incentive, and problem-solving. Outcomes were assessed using routinely collected administrative data. A qualitative process evaluation explored how and why the e-mail messages did or did not support Screening Activity Report access. Results:Of 5449 primary care physicians randomly allocated to 1 of 8 e-mail messages, fewer than half opened the messages and fewer than 1 in 10 clicked through the messages. Messages with problem-solving content were associated with a 12.9% relative reduction in access to the tool (risk ratio: 0.871; 95% confidence interval: 0.791 to 0.958; p = 0.005), but a 0.3% increase in cervical cancer screening (rate ratio: 1.003; 95% confidence interval: 1.001 to 1.006; p = 0.003). If true, that association would represent 7568 more patients being screened. No other significant effects were observed. Conclusions:For audit and feedback to work, recipients must engage with the data; for e-mail messages to prompt activity, recipients must open and review the message content. This large factorial experiment demonstrated that small changes in the content of such e-mail messages might influence clinical behaviour. Future research should focus on strategies to make cancer screening more user-centred.

SUBMITTER: Vaisson G

PROVIDER: S-EPMC6588071 | biostudies-literature | 2019 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Testing e-mail content to encourage physicians to access an audit and feedback tool: a factorial randomized experiment.

Vaisson G G Witteman H O HO Chipenda-Dansokho S S Saragosa M M Bouck Z Z Bravo C A CA Desveaux L L Llovet D D Presseau J J Taljaard M M Umar S S Grimshaw J M JM Tinmouth J J Ivers N M NM

Current oncology (Toronto, Ont.) 20190601 3

<h4>Background</h4>In Ontario, an online audit and feedback tool that provides primary care physicians with detailed information about patients who are overdue for cancer screening is underused. In the present study, we aimed to examine the effect of messages operationalizing 3 behaviour change techniques on access to the audit and feedback tool and on cancer screening rates.<h4>Methods</h4>During May-September 2017, a pragmatic 2×2×2 factorial experiment tested 3 behaviour change techniques: an ...[more]

PMID: 31285667

Similar Datasets

Project description:BackgroundCancer Care Ontario's Screening Activity Report (SAR) is an online audit and feedback tool designed to help primary care physicians in Ontario, Canada, identify patients who are overdue for cancer screening or have abnormal results requiring follow-up. Use of the SAR is associated with increased screening rates. To encourage SAR use, Cancer Care Ontario sends monthly emails to registered primary care physicians announcing that updated data are available. However, analytics reveal that 50% of email recipients do not open the email and less than 7% click the embedded link to log in to their report.ObjectiveThe goal of the study is to determine whether rewritten emails result in increased log-ins. This manuscript describes how different user- and theory-informed messages intended to improve the impact of the monthly emails will be experimentally tested and how a process evaluation will explore why and how any effects observed were (or were not) achieved.MethodsA user-centered approach was used to rewrite the content of the monthly email, including messages operationalizing 3 behavior change techniques: anticipated regret, material incentive (behavior), and problem solving. A pragmatic, 2x2x2 factorial experiment within a multiphase optimization strategy will test the redesigned emails with an embedded qualitative process evaluation to understand how and why the emails may or may not have worked. Trial outcomes will be ascertained using routinely collected administrative data. Physicians will be recruited for semistructured interviews using convenience and snowball sampling.ResultsAs of April 2017, 5576 primary care physicians across the province of Ontario, Canada, had voluntarily registered for the SAR, and in so doing, signed up to receive the monthly email updates. From May to August 2017 participants received the redesigned monthly emails with content specific to their allocated experimental condition prompting use of the SAR. We have not yet begun analyses.ConclusionsThis study will inform how to communicate effectively with primary care providers by email and identify which behavior change techniques tested are most effective at encouraging engagement with an audit and feedback report.Trial registrationClinicalTrials.gov NCT03124316; https://clinicaltrials.gov/ct2/show/NCT03124316 (Archived by WebCite at http://www.webcitation.org/6w2MqDWGu).

Project description:BackgroundAudit and feedback aims to improve patient care by comparing healthcare performance against explicit standards. It is used to monitor and improve patient care, including through National Clinical Audit (NCA) programmes in the UK. Variability in effectiveness of audit and feedback is attributed to intervention design; separate randomised trials to address multiple questions about how to optimise effectiveness would be inefficient. We evaluated different feedback modifications to identify leading candidates for further "real-world" evaluation.MethodsUsing an online fractional factorial screening experiment, we randomised recipients of feedback from five UK NCAs to different combinations of six feedback modifications applied within an audit report excerpt: use effective comparators, provide multimodal feedback, recommend specific actions, provide optional detail, incorporate the patient voice, and minimise cognitive load. Outcomes, assessed immediately after exposure to the online modifications, included intention to enact audit standards (primary outcome, ranked on a scale of -3 to +3, tailored to the NCA), comprehension, user experience, and engagement.ResultsWe randomised 1241 participants (clinicians, managers, and audit staff) between April and October 2019. Inappropriate repeated participant completion occurred; we conservatively excluded participant entries during the relevant period, leaving a primary analysis population of 638 (51.4%) participants. None of the six feedback modifications had an independent effect on intention across the five NCAs. We observed both synergistic and antagonistic effects across outcomes when modifications were combined; the specific NCA and whether recipients had a clinical role had dominant influences on outcome, and there was an antagonistic interaction between multimodal feedback and optional detail. Among clinical participants, predicted intention ranged from 1.22 (95% confidence interval 0.72, 1.72) for the least effective combination in which multimodal feedback, optional detail, and reduced cognitive load were applied within the audit report, up to 2.40 (95% CI 1.88, 2.93) for the most effective combination including multimodal feedback, specific actions, patient voice, and reduced cognitive load.ConclusionPotentially important synergistic and antagonistic effects were identified across combinations of feedback modifications, audit programmes, and recipients, suggesting that feedback designers must explicitly consider how different features of feedback may interact to achieve (or undermine) the desired effects.Trial registrationInternational Standard Randomised Controlled Trial Number: ISRCTN41584028.

Project description:BackgroundProviding clinical performance data to health professionals, a process known as audit and feedback, can play an important role in health system improvement. However, audit and feedback tools can only be effective if the targeted health professionals access and actively review their data. Email is used by Cancer Care Ontario, a provincial cancer agency, to promote access to a Web-based audit and feedback tool called the Screening Activity Report (SAR); however, current emails that lack behavior change content have been ineffective at encouraging log-in to the SAR.ObjectiveThe objective of our study was to describe the process and experience of developing email content that incorporates user input and behavior change techniques (BCTs) to promote the use of the SAR among Ontario primary care providers.MethodsOur interdisciplinary research team first identified BCTs shown to be effective in other settings that could be adapted to promote use of the SAR. We then developed draft BCT-informed email content. Next, we conducted cocreation workshops with physicians who had logged in to the SAR more than once over the past year. Participants provided reactions to researcher-developed BCT-informed content and helped to develop an email that they believed would prompt their colleagues to use the SAR. Content from cocreation workshops was brought to focus groups with physicians who had not used the SAR in the past year. We analyzed notes from the cocreation workshops and focus groups to inform decisions about content. Finally, 8 emails were created to test BCT-informed content in a 2×2×2 factorial randomized experiment.ResultsWe identified 3 key tensions during the development of the email that required us to balance user input with scientific evidence, organizational policies, and our scientific objectives, which are as follows: conflict between user preference and scientific evidence, privacy constraints around personalizing unencrypted emails with performance data, and using cocreation methods in a study with the objective of developing an email that featured BCT-informed content.ConclusionsTeams tasked with developing content to promote health professional engagement with audit and feedback or other quality improvement tools might consider cocreation processes for developing communications that are informed by both users and BCTs. Teams should be cautious about making decisions solely based on user reactions because what users seem to prefer is not always the same as what works. Furthermore, implementing user recommendations may not always be feasible. Teams may face challenges when using cocreation methods to develop a product with the simultaneous goal of having clearly defined variables to test in later studies. The expected role of users, evidence, and the implementation context all warrant consideration to determine whether and how cocreation methods could help to achieve design and scientific objectives.

Project description:BackgroundAudit and feedback (A&F) often successfully enhances health professionals' intentions to improve quality of care but does not consistently lead to practice changes. Recipients often cite data credibility and limited resources as barriers impeding their ability to act upon A&F, suggesting the intention-to-action gap manifests while recipients are interacting with their data. While attention has been paid to the role feedback and contextual variables play in contributing to (or impeding) success, we lack a nuanced understanding of how healthcare professionals interact with and process clinical performance data.MethodsWe used qualitative, semi-structured interviews guided by Normalization Process Theory (NPT). Questions explored the role of data in quality improvement, experiences with the A&F report, perceptions of the data, and interpretations and reflections. Interviews were audio-recorded and transcribed verbatim. Data were analyzed using a combination of inductive and deductive strategies using reflexive thematic analysis informed by a constructivist paradigm.ResultsHealthcare professional characteristics (individual quality improvement capabilities and beliefs about data) seem to influence engagement with A&F to a greater degree than feedback variables (i.e., delivered by peers) and observed contextual factors (i.e., strong quality improvement culture). Most participants lacked the capabilities to interpret practice-level data in an actionable way despite a motivation to engage meaningfully. Reasons for the intention-to-action gap included challenges interpreting longitudinal data, appreciating the nuances of common data sources, understanding how aggregate data provides insights into individualized care, and identifying practice-level actions to improve quality. These factors limited effective cognitive participation and collective action, as outlined in NPT.ConclusionsA well-designed A&F intervention is necessary but not sufficient to inform practice changes. A&F initiatives must include co-interventions to address recipient characteristics (i.e., beliefs and capabilities) and context to optimize impact. Effective strategies to overcome the intention-to-action gap may include modelling how to use A&F to inform practice change, providing opportunities for social interaction relating to the A&F, and circulating examples of effective actions taken in response to A&F. More broadly, undergraduate medical education and post-graduate training must ensure physicians are equipped with QI capabilities, with an emphasis on the skills required to interpret and act on practice-level data.

Project description:BackgroundMedication non-adherence is a major contributor to poor outcomes in diabetes. Previous research has shown an association between use of mail order pharmacy delivery and better medication adherence, but little is known about the barriers and facilitators to mail order pharmacy use in diabetes patients. This qualitative study examined factors related to mail order pharmacy use versus traditional "brick and mortar" pharmacies to refill prescriptions.MethodsWe conducted four 90-min focus groups in 2016 among 28 diabetes patients in the Hawaii and Northern California regions of Kaiser Permanente, a large integrated health care delivery system. We queried participants on their preferred mode for refilling prescriptions and perceived barriers and facilitators of mail order pharmacy use. One researcher independently coded each focus group transcript, with two of these transcripts double-coded by a second researcher to promote reliability. We employed thematic analysis guided by the Capability, Opportunity, Motivation, and Behavior (COM-B) framework using NVivo 11 software.ResultsA total of 28 diabetes patients participated. Participants' average age was 64.1 years; 57% were female; and racial/ethnic backgrounds included Asian/Native Hawaiian/Pacific Islander (36%), Black/African-American (21%) Hispanic/Latino (7%), and non-Hispanic White (36%). Analysis uncovered 26 themes related to the decision to use mail order pharmacy, with each theme representing a barrier or facilitator mapped to the COM-B framework. Most themes (20/26) fell into the COM-B category of 'Opportunity.' Opportunity barriers to mail order pharmacy use included unpredictability of medication delivery date, concerns about mail security, and difficulty coordinating refill orders for multiple prescriptions. In contrast, facilitators included greater access and convenience (e.g., no need to wait in line or arrange transportation) compared to traditional pharmacies. Motivational facilitators to mail order pharmacy use included receiving a pharmacy benefit plan incentive of a free one-month supply of prescriptions.ConclusionsThis study found that while patients with diabetes may benefit from mail order pharmacy use, they perceive numerous barriers to using the service. These findings will inform the design of interventions and quality improvement initiatives to increase mail order pharmacy use, which in turn may improve medication adherence and outcomes in diabetes patients, across health care systems.

Project description:BackgroundMeaningful variations in physician performance are not always discernible from the medical record.ObjectiveWe used unannounced standardized patients to measure and provide feedback on care quality and fidelity of documentation, and examined downstream effects on reimbursement claims.DesignStatic group pre-post comparison study conducted between 2017 and 2019.SettingFourteen New Jersey primary care practice groups (22 practices) enrolled in Horizon BCBS's value-based program received the intervention. For claims analyses, we identified 14 additional comparison practice groups matched on county, practice size, and claims activity.ParticipantsFifty-nine of 64 providers volunteered to participate.InterventionUnannounced standardized patients (USPs) made 217 visits portraying patients with 1-2 focal conditions (diabetes, depression, back pain, smoking, or preventive cancer screening). After two baseline visits to a provider, we delivered feedback and conducted two follow-up visits.MeasurementsUSP-completed checklists of guideline-based provider care behaviors, visit audio recordings, and provider notes were used to measure behaviors performed and documentation errors pre- and post-feedback. We also compared changes in 3-month office-based claims by actual patients between the intervention and comparison practice groups before and after feedback.ResultsExpected clinical behaviors increased from 46% to 56% (OR = 1.53, 95% CI 1.29-1.83, p < 0.0001), with significant improvements in smoking cessation, back pain, and depression screening. Providers were less likely to document unperformed tasks after (16%) than before feedback (18%; OR = 0.74, 95% CI 0.62 to 0.90, p = 0.002). Actual claim costs increased significantly less in the study than comparison group for diabetes and depression but significantly more for smoking cessation, cancer screening, and low back pain.LimitationsSelf-selection of participating practices and lack of access to prescription claims.ConclusionDirect observation of care identifies hidden deficits in practice and documentation, and with feedback can improve both, with concomitant effects on costs.

Dataset Information

Testing e-mail content to encourage physicians to access an audit and feedback tool: a factorial randomized experiment.

Publications

Testing e-mail content to encourage physicians to access an audit and feedback tool: a factorial randomized experiment.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets