Dataset Information

Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors.

ABSTRACT:

Background

Clinical trials are commonly done without blinded outcome assessors despite the risk of bias. We wanted to evaluate the effect of nonblinded outcome assessment on estimated effects in randomized clinical trials with outcomes that involved subjective measurement scales.

Methods

We conducted a systematic review of randomized clinical trials with both blinded and nonblinded assessment of the same measurement scale outcome. We searched PubMed, EMBASE, PsycINFO, CINAHL, Cochrane Central Register of Controlled Trials, HighWire Press and Google Scholar for relevant studies. Two investigators agreed on the inclusion of trials and the outcome scale. For each trial, we calculated the difference in effect size (i.e., standardized mean difference between nonblinded and blinded assessments). A difference in effect size of less than 0 suggested that nonblinded assessors generated more optimistic estimates of effect. We pooled the differences in effect size using inverse variance random-effects meta-analysis and used metaregression to identify potential reasons for variation.

Results

We included 24 trials in our review. The main meta-analysis included 16 trials (involving 2854 patients) with subjective outcomes. The estimated treatment effect was more beneficial when based on nonblinded assessors (pooled difference in effect size -0.23 [95% confidence interval (CI) -0.40 to -0.06]). In relative terms, nonblinded assessors exaggerated the pooled effect size by 68% (95% CI 14% to 230%). Heterogeneity was moderate (I(2) = 46%, p = 0.02) and unexplained by metaregression.

Interpretation

We provide empirical evidence for observer bias in randomized clinical trials with subjective measurement scale outcomes. A failure to blind assessors of outcomes in such trials results in a high risk of substantial bias.

SUBMITTER: Hrobjartsson A

PROVIDER: S-EPMC3589328 | biostudies-literature | 2013 Mar

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors.

Hróbjartsson Asbjørn A Thomsen Ann Sofia Skou AS Emanuelsson Frida F Tendal Britta B Hilden Jørgen J Boutron Isabelle I Ravaud Philippe P Brorson Stig S

CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne 20130128 4

<h4>Background</h4>Clinical trials are commonly done without blinded outcome assessors despite the risk of bias. We wanted to evaluate the effect of nonblinded outcome assessment on estimated effects in randomized clinical trials with outcomes that involved subjective measurement scales.<h4>Methods</h4>We conducted a systematic review of randomized clinical trials with both blinded and nonblinded assessment of the same measurement scale outcome. We searched PubMed, EMBASE, PsycINFO, CINAHL, Coch ...[more]

PMID: 23359047

Similar Datasets

Project description:BackgroundPrevious studies of second opinions in the diagnosis of melanocytic skin lesions have examined blinded second opinions, which do not reflect usual clinical practice. The current study, conducted in the USA, investigated both blinded and nonblinded second opinions for their impact on diagnostic accuracy.MethodsIn total, 100 melanocytic skin biopsy cases, ranging from benign to invasive melanoma, were interpreted by 74 dermatopathologists. Subsequently, 151 dermatopathologists performed nonblinded second and third reviews. We compared the accuracy of single reviewers, second opinions obtained from independent, blinded reviewers and second opinions obtained from sequential, nonblinded reviewers. Accuracy was defined with respect to a consensus reference diagnosis.ResultsThe mean case-level diagnostic accuracy of single reviewers was 65.3% (95% CI 63.4-67.2%). Second opinions arising from sequential, nonblinded reviewers significantly improved accuracy to 69.9% (95% CI 68.0-71.7%; P < 0.001). Similarly, second opinions arising from blinded reviewers improved upon the accuracy of single reviewers (69.2%; 95% CI 68.0-71.7%). Nonblinded reviewers were more likely than blinded reviewers to give diagnoses in the same diagnostic classes as the first diagnosis. Nonblinded reviewers tended to be more confident when they agreed with previous reviewers, even with inaccurate diagnoses.ConclusionWe found that both blinded and nonblinded second reviewers offered a similar modest improvement in diagnostic accuracy compared with single reviewers. Obtaining second opinions with knowledge of previous reviews tends to generate agreement among reviews, and may generate unwarranted confidence in an inaccurate diagnosis. Combining aspects of both blinded and nonblinded review in practice may leverage the advantages while mitigating the disadvantages of each approach. Specifically, a second pathologist could give an initial diagnosis blinded to the results of the first pathologist, with subsequent nonblinded discussion between the two pathologists if their diagnoses differ.

Project description:BackgroundFrailty is a distinctive health state associated with a loss of physiological reserve that results in higher rates of perioperative complications and impaired return to pre-morbid functional status. It is prevalent in the vascular population; however routine assessment is not common despite national guidance to the contrary. We aimed to evaluate the reliability of the Clinical Frailty Scale in assessing frailty in the surgical vascular population.MethodsIn this prospective, observational, observer-blinded study, we compared assessment of frailty in patients scheduled for major vascular surgery attending the pre-operative assessment clinic using the Clinical Frailty Scale against the Edmonton Frailty Scale. The study investigator completed the Edmonton Frailty Scale assessment; this was compared to the Clinical Frailty Scale assessments performed by the pre-assessment consultant and pre-assessment nurse, who were blinded to the Edmonton Frailty Scale score. The inter-rater reliability of the Clinical Frailty Scale between the pre-assessment consultant and pre-assessment nurse was determined by comparing their frailty scores for each patient.ResultsNinety-seven patients were included in the analysis (median age 72 years, 84% male and 16% female). There was a moderate level of agreement between the Edmonton and Clinical Frailty Scale score for both consultants (87.6% agreement) and pre-assessment nurses (87.6% agreement). There was a substantial level of agreement between consultants and pre-assessment nurses for the Clinical Frailty Scale (89.7% agreement) CONCLUSIONS: The Clinical Frailty Scale is a useful tool to assess frailty in the vascular surgical population. It is more practical than the Edmonton Frailty Scale: quick to complete, requires minimal training and can be used when physical disability is present.Trial registrationThe study was approved by the Wales Health and Care Research Ethics Service (REC reference 17/WA/0160, IRAS 201173).Trial registrationNCT03403673. Registered 19 January 2018, https://clinicaltrials.gov/ct2/show/NCT03403673.

Project description:BackgroundThere is now clear systematic review evidence that measurement can affect the people being measured; much of this evidence focusses on how asking people to complete a questionnaire can result in changes in behaviour. Changes in measured behaviour and other outcomes due to this reactivity may introduce bias in otherwise well-conducted randomised controlled trials (RCTs), yielding incorrect estimates of intervention effects. Despite this, measurement reactivity is not currently adequately considered in risk of bias frameworks. The present research aims to produce a set of guidance statements on how best to avoid or minimise bias due to measurement reactivity in studies of interventions to improve health, with a particular focus on bias in RCTs.MethodsThe MERIT study consists of a series of systematic and rapid reviews, a Delphi study and an expert workshop to develop guidance on how to minimise bias in trials due to measurement reactivity. An existing systematic review on question-behaviour effects on health-related behaviours will be updated and three new rapid reviews will be conducted to identify (1) existing guidance on measurement reactivity; (2) systematic reviews of studies that have quantified the effects of measurement on outcomes relating to behaviour and affective outcomes in health and non-health contexts and (3) trials that have investigated the effects of objective measurements of behaviour on concurrent or subsequent behaviour itself. A Delphi procedure will be used to combine the views of experts with a view to reaching agreement on the scope of the guidance statements. Finally, a workshop will be held in autumn 2018, with the aim of producing a set of guidance statements that will form the central part of new MRC guidance on how best to avoid bias due to measurement reactivity in studies of interventions to improve health.DiscussionOur ambition is to produce MRC guidance on measurement reactions in trials which will be used by future trial researchers, leading to the development of trials that are less likely to be at risk of bias.

Dataset Information

Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors.

Background

Methods

Results

Interpretation

Publications

Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets