Dataset Information

Statistical methods used to test for agreement of medical instruments measuring continuous variables in method comparison studies: a systematic review.

ABSTRACT:

Background

Accurate values are a must in medicine. An important parameter in determining the quality of a medical instrument is agreement with a gold standard. Various statistical methods have been used to test for agreement. Some of these methods have been shown to be inappropriate. This can result in misleading conclusions about the validity of an instrument. The Bland-Altman method is the most popular method judging by the many citations of the article proposing this method. However, the number of citations does not necessarily mean that this method has been applied in agreement research. No previous study has been conducted to look into this. This is the first systematic review to identify statistical methods used to test for agreement of medical instruments. The proportion of various statistical methods found in this review will also reflect the proportion of medical instruments that have been validated using those particular methods in current clinical practice.

Methodology/findings

Five electronic databases were searched between 2007 and 2009 to look for agreement studies. A total of 3,260 titles were initially identified. Only 412 titles were potentially related, and finally 210 fitted the inclusion criteria. The Bland-Altman method is the most popular method with 178 (85%) studies having used this method, followed by the correlation coefficient (27%) and means comparison (18%). Some of the inappropriate methods highlighted by Altman and Bland since the 1980s are still in use.

Conclusions

This study finds that the Bland-Altman method is the most popular method used in agreement research. There are still inappropriate applications of statistical methods in some studies. It is important for a clinician or medical researcher to be aware of this issue because misleading conclusions from inappropriate analyses will jeopardize the quality of the evidence, which in turn will influence quality of care given to patients in the future.

SUBMITTER: Zaki R

PROVIDER: S-EPMC3360667 | biostudies-literature | 2012

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Statistical methods used to test for agreement of medical instruments measuring continuous variables in method comparison studies: a systematic review.

Zaki Rafdzah R Bulgiba Awang A Ismail Roshidi R Ismail Noor Azina NA

PloS one 20120525 5

<h4>Background</h4>Accurate values are a must in medicine. An important parameter in determining the quality of a medical instrument is agreement with a gold standard. Various statistical methods have been used to test for agreement. Some of these methods have been shown to be inappropriate. This can result in misleading conclusions about the validity of an instrument. The Bland-Altman method is the most popular method judging by the many citations of the article proposing this method. However, ...[more]

PMID: 22662248

Similar Datasets

Project description:BackgroundCoaching is emerging as a novel approach to guide medical students toward becoming competent, reflective physicians and master adaptive learners. However, no instruments currently exist to measure academic coaching at the undergraduate medical education level.ObjectiveTo describe the development and psychometric assessments of two instruments designed to assess academic coaching of medical students toward creating a robust measurement model of this educational paradigm.DesignObservational psychometric.ParticipantsAll medical students in the 2014 and 2015 cohorts and all their coaches were invited to complete the instruments being tested, which led to 662 medical student responses from 292 medical students and 468 coaching responses from 22 coaches being included in analyses. Medical student response rates were 75.7% from 2014 and 75.5% from 2015. Overall, the coaches' response rate was 71%.Main measuresTwo 31-item instruments were initially developed, one for medical students to assess their coach and one for faculty coaches to assess their students, both of which evaluated coaching based on definitions we formulated using existing literature. Each was administered to two cohorts of medical students and coaches in 2015 and 2016. An exploratory factor analysis using principal component analysis as the extraction method and Varimax with Kaiser normalization as the rotation method was conducted.Key resultsEighteen items reflecting four domains were retained with eigenvalues higher than 1.0 for medical students' assessment of coaching, which measured promoting self-monitoring, relationship building, promoting reflective behavior, and establishing foundational ground rules. Sixteen items reflecting two domains were retained for the faculty assessment of coaching with eigenvalues higher than 1.0: the Practice of Coaching and Relationship Formation.ConclusionsWe successfully developed and psychometrically validated surveys designed to measure key aspects of the coaching relationship, coaching processes, and reflective outcomes. The new validated instruments offer a robust measurement model for academic coaching.

Project description:BackgroundStudies of agreement examine the distance between readings made by different devices or observers measuring the same quantity. If the values generated by each device are close together most of the time then we conclude that the devices agree. Several different agreement methods have been described in the literature, in the linear mixed modelling framework, for use when there are time-matched repeated measurements within subjects.MethodsWe provide a tutorial to help guide practitioners when choosing among different methods of assessing agreement based on a linear mixed model assumption. We illustrate the use of five methods in a head-to-head comparison using real data from a study involving Chronic Obstructive Pulmonary Disease (COPD) patients and matched repeated respiratory rate observations. The methods used were the concordance correlation coefficient, limits of agreement, total deviation index, coverage probability, and coefficient of individual agreement.ResultsThe five methods generated similar conclusions about the agreement between devices in the COPD example; however, some methods emphasized different aspects of the between-device comparison, and the interpretation was clearer for some methods compared to others.ConclusionsFive different methods used to assess agreement have been compared in the same setting to facilitate understanding and encourage the use of multiple agreement methods in practice. Although there are similarities between the methods, each method has its own strengths and weaknesses which are important for researchers to be aware of. We suggest that researchers consider using the coverage probability method alongside a graphical display of the raw data in method comparison studies. In the case of disagreement between devices, it is important to look beyond the overall summary agreement indices and consider the underlying causes. Summarising the data graphically and examining model parameters can both help with this.

Project description:Empowerment is acknowledged as a process facilitating those who are less powerful to be engaged in their problem identification, decision making and actions to gain control over their life. This is an important goal for women during the perinatal period in their transition to motherhood. A concept analysis of women's empowerment during the perinatal period found that psychological and social dimensions play a role in women's perinatal empowerment and identified several defining attributes. The aim of this study was to identify robust validated instruments that measure all the attributes of women's empowerment during the perinatal period. We did a scoping review of scientific literature following the methodology of the JBI Reviewer's Manual. We searched the database MEDLINE, CINAHL, PsycINFO, PsycARTICLES and SocINDEX and selected papers meeting the inclusion criteria. Instruments measuring empowerment or related concepts were identified in the selected papers. Two authors independently cross referenced the items of each instrument against the defining attributes for empowerment. Our search resulted in 9771 unique hits of which 36 papers were finally included. Studies were from various countries with a wide variety of aims, demographics of cohorts and timepoints across the perinatal period. Twenty-one different instruments were used to measure empowerment, of which 11 were validated among women during the perinatal period. However, no identified instrument was developed specifically for women during the perinatal period or included all the dimensions of empowerment and the defining attributes. There is a need for a theoretically sound valid and reliable instrument measuring all the dimensions of empowerment of women during the perinatal period. Once developed this instrument needs testing with a broad range of women. Results from such a study will inform the development of appropriate interventions that have a coherent theoretical basis and are empirically informed to enhance women's empowerment during the perinatal period.

Project description:Despite the recognized importance of atrophy in multiple sclerosis (MS), methods for its quantification have been mostly restricted to the research domain. Recently, a CE labelled and FDA approved MS-specific atrophy quantification method, MSmetrix, has become commercially available. Here we perform a validation of MSmetrix against established methods in simulated and in vivo MRI data.Whole-brain and gray matter (GM) volume were measured with the cross-sectional pipeline of MSmetrix and compared to the outcomes of FreeSurfer (cross-sectional pipeline), SIENAX and SPM. For this comparison we investigated 20 simulated brain images, as well as in vivo data from 100 MS patients and 20 matched healthy controls. In fifty of the MS patients a second time point was available. In this subgroup, we additionally analyzed the whole-brain and GM volume change using the longitudinal pipeline of MSmetrix and compared the results with those of FreeSurfer (longitudinal pipeline) and SIENA.In the simulated data, SIENAX displayed the smallest average deviation compared with the reference whole-brain volume (+ 19.56 ± 10.34 mL), followed by MSmetrix (- 38.15 ± 17.77 mL), SPM (- 42.99 ± 17.12 mL) and FreeSurfer (- 78.51 ± 12.68 mL). A similar pattern was seen in vivo. Among the cross-sectional methods, Deming regression analyses revealed proportional errors particularly in MSmetrix and SPM. The mean difference percentage brain volume change (PBVC) was lowest between longitudinal MSmetrix and SIENA (+ 0.16 ± 0.91%). A strong proportional error was present between longitudinal percentage gray matter volume change (PGVC) measures of MSmetrix and FreeSurfer (slope = 2.48). All longitudinal methods were sensitive to the MRI hardware upgrade that occurred during the time of the study.MSmetrix, FreeSurfer, FSL and SPM show differences in atrophy measurements, even at the whole-brain level, that are large compared to typical atrophy rates observed in MS. Especially striking are the proportional errors between methods. Cross-sectional MSmetrix behaved similarly to SPM, both in terms of mean volume difference as well as proportional error. Longitudinal MSmetrix behaved most similar to SIENA. Our results indicate that brain volume measurement and normalization from T1-weighted images remains an unsolved problem that requires much more attention.

Dataset Information

Statistical methods used to test for agreement of medical instruments measuring continuous variables in method comparison studies: a systematic review.

Background

Methodology/findings

Conclusions

Publications

Statistical methods used to test for agreement of medical instruments measuring continuous variables in method comparison studies: a systematic review.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets