Dataset Information

Measuring ability to assess claims about treatment effects: the development of the 'Claim Evaluation Tools'.

ABSTRACT: OBJECTIVES:To describe the development of the Claim Evaluation Tools, a set of flexible items to measure people's ability to assess claims about treatment effects. SETTING:Methodologists and members of the community (including children) in Uganda, Rwanda, Kenya, Norway, the UK and Australia. PARTICIPANTS:In the iterative development of the items, we used purposeful sampling of people with training in research methodology, such as teachers of evidence-based medicine, as well as patients and members of the public from low-income and high-income countries. Development consisted of 4 processes: (1) determining the scope of the Claim Evaluation Tools and development of items; (2) expert item review and feedback (n=63); (3) cognitive interviews with children and adult end-users (n=109); and (4) piloting and administrative tests (n=956). RESULTS:The Claim Evaluation Tools database currently includes a battery of multiple-choice items. Each item begins with a scenario which is intended to be relevant across contexts, and which can be used for children (from age 10? and above), adult members of the public and health professionals. People with expertise in research methods judged the items to have face validity, and end-users judged them relevant and acceptable in their settings. In response to feedback from methodologists and end-users, we simplified some text, explained terms where needed, and redesigned formats and instructions. CONCLUSIONS:The Claim Evaluation Tools database is a flexible resource from which researchers, teachers and others can design measurement instruments to meet their own requirements. These evaluation tools are being managed and made freely available for non-commercial use (on request) through Testing Treatments interactive (testingtreatments.org). TRIAL REGISTRATION NUMBERS:PACTR201606001679337 and PACTR201606001676150; Pre-results.

SUBMITTER: Austvoll-Dahlgren A

PROVIDER: S-EPMC5777467 | biostudies-other | 2017 May

REPOSITORIES: biostudies-other

ACCESS DATA

Publications

Measuring ability to assess claims about treatment effects: the development of the 'Claim Evaluation Tools'.

Austvoll-Dahlgren Astrid A Semakula Daniel D Nsangi Allen A Oxman Andrew David AD Chalmers Iain I Rosenbaum Sarah S Guttersrud Øystein Ø

BMJ open 20170517 5

<h4>Objectives</h4>To describe the development of the Claim Evaluation Tools, a set of flexible items to measure people's ability to assess claims about treatment effects.<h4>Setting</h4>Methodologists and members of the community (including children) in Uganda, Rwanda, Kenya, Norway, the UK and Australia.<h4>Participants</h4>In the iterative development of the items, we used purposeful sampling of people with training in research methodology, such as teachers of evidence-based medicine, as well ...[more]

PMID: 28515181

Similar Datasets

Project description:IntroductionWe evaluated an intervention designed to teach 10- to 12-year-old primary school children to assess claims about the effects of treatments (any action intended to maintain or improve health). We report outcomes measured 1 year after the intervention.MethodsIn this cluster-randomised trial, we included primary schools in the central region of Uganda that taught year 5 children (aged 10 to 12?years). We randomly allocated a representative sample of eligible schools to either an intervention or control group. Intervention schools received the Informed Health Choices primary school resources (textbooks, exercise books and a teachers' guide). The primary outcomes, measured at the end of the school term and again after 1 year, were the mean score on a test with two multiple-choice questions for each of the 12 concepts and the proportion of children with passing scores.ResultsWe assessed 2960 schools for eligibility; 2029 were eligible, and a random sample of 170 were invited to recruitment meetings. After recruitment meetings, 120 eligible schools consented and were randomly assigned to either the intervention group (n =?60 schools; 76 teachers and 6383 children) or the control group (n =?60 schools; 67 teachers and 4430 children). After 1 year, the mean score in the multiple-choice test for the intervention schools was 68.7% compared with 53.0% for the control schools (adjusted mean difference 16.7%; 95% CI, 13.9 to 19.5; P <?0.00001). In the intervention schools, 3160 (80.1%) of 3943 children who completed the test after 1 year achieved a predetermined passing score (??13 of 24 correct answers) compared with 1464 (51.5%) of 2844 children in the control schools (adjusted difference, 39.5%; 95% CI, 29.9 to 47.5).ConclusionUse of the learning resources led to a large improvement in the ability of children to assess claims, which was sustained for at least 1 year.Trial registrationPan African Clinical Trial Registry (www.pactr.org), PACTR201606001679337. Registered on 13 June 2016.

Project description:Objective: Opioid surveillance in response to the opioid epidemic will benefit from scalable, automated algorithms for identifying patients with clinically documented signs of problem prescription opioid use. Existing algorithms lack accuracy. We sought to develop a high-sensitivity, high-specificity classification algorithm based on widely available structured health data to identify patients receiving chronic extended-release/long-acting (ER/LA) therapy with evidence of problem use to support subsequent epidemiologic investigations. Methods: Outpatient medical records of a probability sample of 2,000 Kaiser Permanente Washington patients receiving ≥60 days' supply of ER/LA opioids in a 90-day period from 1 January 2006 to 30 June 2015 were manually reviewed to determine the presence of clinically documented signs of problem use and used as a reference standard for algorithm development. Using 1,400 patients as training data, we constructed candidate predictors from demographic, enrollment, encounter, diagnosis, procedure, and medication data extracted from medical claims records or the equivalent from electronic health record (EHR) systems, and we used adaptive least absolute shrinkage and selection operator (LASSO) regression to develop a model. We evaluated this model in a comparable 600-patient validation set. We compared this model to ICD-9 diagnostic codes for opioid abuse, dependence, and poisoning. This study was registered with ClinicalTrials.gov as study NCT02667262 on 28 January 2016. Results: We operationalized 1,126 potential predictors characterizing patient demographics, procedures, diagnoses, timing, dose, and location of medication dispensing. The final model incorporating 53 predictors had a sensitivity of 0.582 at positive predictive value (PPV) of 0.572. ICD-9 codes for opioid abuse, dependence, and poisoning had a sensitivity of 0.390 at PPV of 0.599 in the same cohort. Conclusions: Scalable methods using widely available structured EHR/claims data to accurately identify problem opioid use among patients receiving long-term ER/LA therapy were unsuccessful. This approach may be useful for identifying patients needing clinical evaluation.

Project description:IntroductionEarlier, we designed and evaluated an educational mass media intervention for improving people's ability to think more critically and to assess the trustworthiness of claims (assertions) about the benefits and harms (effects) of treatments. The overall aims of this follow-up study were to evaluate the impact of our intervention 1 year after it was administered, and to assess retention of learning and behaviour regarding claims about treatments.MethodsWe randomly allocated consenting parents to listen to either the Informed Health Choices podcast (intervention) or typical public service announcements about health issues (control) over 7-10 weeks. Each intervention episode explained how the trustworthiness of treatment claims can be assessed by using relevant key concepts of evidence-informed decision-making. Participants listened to two episodes per week, delivered by research assistants. We evaluated outcomes immediately, and a year after the intervention. Primary outcomes were mean score and the proportion with a score indicating a basic ability to apply the key concepts (>?11 out of 18 correct answers) on a tool measuring people's ability to critically appraise the trustworthiness of treatment claims. Skills decay/retention was estimated by calculating the relative difference between the follow-up and initial results in the intervention group, adjusting for chance. Statistical analyses were performed using R (R Core Team, Vienna, Austria; version 3.4.3).ResultsAfter 1 year, the mean score for parents in the intervention group was 58.9% correct answers, compared to 52.6% in the control (adjusted mean difference of 6.7% (95% CI 3.3% to 10.1%)). In the intervention group, 47.2% of 267 parents had a score indicating a basic ability to assess treatment claims compared to 39.5% of 256 parents in the control (adjusted difference of 9.8% more parents (95% CI 0.9% to 18.9%). These represent relative reductions of 29% in the mean scores and 33% in the proportion of parents with a score indicating a basic ability to assess the trustworthiness of claims about treatment effects.ConclusionsAlthough listening to the Informed Health Choices podcast initially led to a large improvement in the ability of parents to assess claims about the effects of treatments, our findings show that these skills decreased substantially over 1 year. More active practice could address the substantial skills decay observed over 1 year.Trial registrationPan African Clinical Trial Registry (www.pactr.org), PACTR201606001676150. Registered on 12 June 2016.

Project description:We developed the Informed Health Choices podcast to improve people's ability to assess claims about the effects of treatments. We evaluated the effects of the podcast in a randomised trial.ObjectivesWe conducted this process evaluation to assess the fidelity of the intervention, identify factors that affected the implementation and impact of the intervention and could affect scaling up, and identify potential adverse and beneficial effects.SettingThe study was conducted in central Uganda in rural, periurban and urban settings.ParticipantsWe collected data on parents who were in the intervention arm of the Informed Health Choices study that evaluated an intervention to improve parents' ability to assess treatment effects.ProceduresWe conducted 84 semistructured interviews during the intervention, 19 in-depth interviews shortly after, two focus group discussions with parents, one focus group discussion with research assistants and two in-depth interviews with the principal investigators. We used framework analysis to manage qualitative data, assessed the certainty of the findings using the GRADE-CERQual (Grading of Recommendations, Assessment, Development and Evaluations-Confidence in the Evidence from Reviews of Qualitative Research) approach, and organised findings in a logic model.OutcomesProportion of participants listening to all episodes; factors influencing the implementation of the podcast; ways to scale up and any adverse and beneficial effects.ResultsAll participants who completed the study listened to the podcast as intended, perhaps because of the explanatory design and recruitment of parents with a positive attitude. This was also likely facilitated by the podcast being delivered by research assistants, and providing the participants with MP3 players. The podcast was reportedly clear, understandable, credible and entertaining, which motivated them to listen and eased implementation. No additional adverse effects were reported.ConclusionsParticipants experienced the podcast positively and were motivated to engage with it. These findings help to explain the short-term effectiveness of the intervention, but not the decrease in effectiveness over the following year.

Dataset Information

Measuring ability to assess claims about treatment effects: the development of the 'Claim Evaluation Tools'.

Publications

Measuring ability to assess claims about treatment effects: the development of the 'Claim Evaluation Tools'.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets