Dataset Information

Experimental design and data analysis in receiver operating characteristic studies: lessons learned from reports in radiology from 1997 to 2006.

ABSTRACT:

Purpose

To provide a broad perspective concerning the recent use of receiver operating characteristic (ROC) analysis in medical imaging by reviewing ROC studies published in Radiology between 1997 and 2006 for experimental design, imaging modality, medical condition, and ROC paradigm.

Materials and methods

Two hundred ninety-five studies were obtained by conducting a literature search with PubMed with two criteria: publication in Radiology between 1997 and 2006 and occurrence of the phrase "receiver operating characteristic." Studies returned by the query that were not diagnostic imaging procedure performance evaluations were excluded. Characteristics of the remaining studies were tabulated.

Results

Two hundred thirty-three (79.0%) of the 295 studies reported findings based on observers' diagnostic judgments or objective measurements. Forty-three (14.6%) did not include human observers, with most of these reporting an evaluation of a computer-aided diagnosis system or functional data obtained with computed tomography (CT) or magnetic resonance (MR) imaging. The remaining 19 (6.4%) studies were classified as reviews or meta-analyses and were excluded from our subsequent analysis. Among the various imaging modalities, MR imaging (46.0%) and CT (25.7%) were investigated most frequently. Approximately 60% (144 of 233) of ROC studies with human observers published in Radiology included three or fewer observers.

Conclusion

ROC analysis is widely used in radiologic research, confirming its fundamental role in assessing diagnostic performance. However, the ROC studies reported in Radiology were not always adequate to support clear and clinically relevant conclusions.

SUBMITTER: Shiraishi J

PROVIDER: S-EPMC2786192 | biostudies-literature | 2009 Dec

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Experimental design and data analysis in receiver operating characteristic studies: lessons learned from reports in radiology from 1997 to 2006.

Shiraishi Junji J Pesce Lorenzo L LL Metz Charles E CE Doi Kunio K

Radiology 20091028 3

<h4>Purpose</h4>To provide a broad perspective concerning the recent use of receiver operating characteristic (ROC) analysis in medical imaging by reviewing ROC studies published in Radiology between 1997 and 2006 for experimental design, imaging modality, medical condition, and ROC paradigm.<h4>Materials and methods</h4>Two hundred ninety-five studies were obtained by conducting a literature search with PubMed with two criteria: publication in Radiology between 1997 and 2006 and occurrence of t ...[more]

PMID: 19864510

Similar Datasets

Project description:BackgroundDisease prevalence is rarely explicitly considered in the early stages of the development of novel prognostic tests. Rather, researchers use the area under the receiver operating characteristic (AUROC) as the key metric to gauge and report predictive performance ability. Because this statistic does not account for disease prevalence, proposed tests may not appropriately address clinical requirements. This ultimately impedes the translation of prognostic tests into clinical practice.MethodsA method to express positive- and/or negative predictive value criteria (PPV, NPV) within the ROC space is presented. Equations are derived for so-called equi-PPV (and equi-NPV) lines. Herewith it is possible, for any given prevalence, to plot a series of sensitivity-specificity pairs which meet a specified PPV (or NPV) criterion onto the ROC space.This concept is introduced by firstly reviewing the well-established "mechanics", strengths and limitations of the ROC analysis in the context of developing prognostic models. Then, the use of PPV (and/or) NPV criteria to augment the ROC analysis is elaborated.Additionally, an interactive web tool was also created to enable people to explore the dynamics of lines of equi-predictive value in function of prevalence. The web tool also allows to gauge what ROC curve shapes best meet specific positive and/or negative predictive value criteria (http://d4ta.link/ppvnpv/).ResultsTo illustrate the merits and implications of this concept, an example on the prediction of pre-eclampsia risk in low-risk nulliparous pregnancies is elaborated.ConclusionsIn risk stratification, the clinical usefulness of a prognostic test can be expressed in positive- and negative predictive value criteria; the development of novel prognostic tests will be facilitated by the possibility to co-visualise such criteria together with ROC curves. To achieve clinically meaningful risk stratification, the development of separate tests to meet either a pre-specified positive value (rule-in) or a negative predictive value (rule-out) criteria should be considered: the characteristics of successful rule-in and rule-out tests may markedly differ.

Dataset Information

Experimental design and data analysis in receiver operating characteristic studies: lessons learned from reports in radiology from 1997 to 2006.

Purpose

Materials and methods

Results

Conclusion

Publications

Experimental design and data analysis in receiver operating characteristic studies: lessons learned from reports in radiology from 1997 to 2006.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets