Dataset Information

Threshold-free measures for assessing the performance of medical screening tests.

ABSTRACT:

Background

The area under the receiver operating characteristic curve (AUC) is frequently used as a performance measure for medical tests. It is a threshold-free measure that is independent of the disease prevalence rate. We evaluate the utility of the AUC against an alternate measure called the average positive predictive value (AP), in the setting of many medical screening programs where the disease has a low prevalence rate.

Methods

We define the two measures using a common notation system and show that both measures can be expressed as a weighted average of the density function of the diseased subjects. The weights for the AP include prevalence in some form, but those for the AUC do not. These measures are compared using two screening test examples under rare and common disease prevalence rates.

Results

The AP measures the predictive power of a test, which varies when the prevalence rate changes, unlike the AUC, which is prevalence independent. The relationship between the AP and the prevalence rate depends on the underlying screening/diagnostic test. Therefore, the AP provides relevant information to clinical researchers and regulators about how a test is likely to perform in a screening population.

Conclusion

The AP is an attractive alternative to the AUC for the evaluation and comparison of medical screening tests. It could improve the effectiveness of screening programs during the planning stage.

SUBMITTER: Yuan Y

PROVIDER: S-EPMC4403252 | biostudies-literature | 2015

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Threshold-free measures for assessing the performance of medical screening tests.

Yuan Yan Y Su Wanhua W Zhu Mu M

Frontiers in public health 20150420

<h4>Background</h4>The area under the receiver operating characteristic curve (AUC) is frequently used as a performance measure for medical tests. It is a threshold-free measure that is independent of the disease prevalence rate. We evaluate the utility of the AUC against an alternate measure called the average positive predictive value (AP), in the setting of many medical screening programs where the disease has a low prevalence rate.<h4>Methods</h4>We define the two measures using a common not ...[more]

PMID: 25941668

Similar Datasets

Project description:BackgroundOverall accuracy measures of medical tests are often used with unclear interpretations.ObjectivesTo develop methods of calculating the overall accuracy of medical tests in the patient population.MethodsAlgebraic equations based on Bayes' theorem.ResultsA new approach is proposed for calculating overall accuracy in the patient population. Examples and applications using published data are presented.ConclusionsThe overall accuracy is the proportion of the correct test results. We introduce a clear distinction between the overall accuracy measures of medical tests that are aimed at the detection of a disease in a screening of populations for public health purposes in the general population and the overall accuracy measures of tests aimed at determining a diagnosis in individuals in a clinical setting. We show that the overall detection accuracy measure is obtained in a specific study that explores test accuracy among persons with known diagnoses and may be useful for public health screening tests. It is different from the overall diagnostic accuracy that could be calculated in the clinical setting for the evaluation of medical tests aimed at determining the individual patients' diagnoses. We show that the overall detection accuracy is constant and is not affected by the prevalence of the disease. In contrast, the overall diagnostic accuracy changes and is dependent on the prevalence. Moreover, it ranges according to the ratio between the sensitivity and specificity. Thus, when the sensitivity is greater than the specificity, the overall diagnostic accuracy increases with increasing prevalence, and vice versa, that is, when the sensitivity is lower than the specificity, the overall diagnostic accuracy decreases with increasing prevalence so that another test might be more useful for diagnostic procedures. Our paper suggests a new and more appropriate methodology for estimating the overall diagnostic accuracy of any medical test. This may be important for helping clinicians avoid errors.

Project description:BackgroundClinical-performance measurement has helped improve the quality of health-care; yet success in attaining high levels of quality across multiple domains simultaneously still varies considerably. Although many sources of variability in care quality have been studied, the difficulty required to complete the clinical work itself has received little attention.ObjectiveWe present a task-based methodology for evaluating the difficulty of clinical-performance measures (CPMs) by assessing the complexity of their component requisite tasks.DesignUsing Functional Job Analysis (FJA), subject-matter experts (SMEs) generated task lists for 17 CPMs; task lists were rated on ten dimensions of complexity, and then aggregated into difficulty composites.ParticipantsEleven outpatient work SMEs; 133 VA Medical Centers nationwide.Main measuresClinical Performance: 17 outpatient CPMs (2000-2008) at 133 VA Medical Centers nationwide. Measure Difficulty: for each CPM, the number of component requisite tasks and the average rating across ten FJA complexity scales for the set of tasks comprising the measure.Key resultsMeasures varied considerably in the number of component tasks (M?=?10.56, SD?=?6.25, min?=?5, max?=?25). Measures of chronic care following acute myocardial infarction exhibited significantly higher measure difficulty ratings compared to diabetes or screening measures, but not to immunization measures ([Formula: see text]?=?0.45, -0.04, -0.05, and -0.06 respectively; F (3, 186)?=?3.57, p?=?0.015). Measure difficulty ratings were not significantly correlated with the number of component tasks (r?=?-0.30, p?=?0.23).ConclusionsEvaluating the difficulty of achieving recommended CPM performance levels requires more than simply counting the tasks involved; using FJA to assess the complexity of CPMs' component tasks presents an alternate means of assessing the difficulty of primary-care CPMs and accounting for performance variation among measures and performers. This in turn could be used in designing performance reward programs, or to match workflow to clinician time and effort.

Project description:Supraspan verbal list-learning tests, such as the Rey Auditory Verbal Learning Test (RAVLT), are classic neuropsychological tests for assessing verbal memory. In this study, we investigated the impact of the meaning of the words to be learned on three memory stages [short-term recall (STR), learning, and delayed recall (DR)] in a cohort of 447 healthy adults. First, we compared scores obtained from the RAVLT (word condition) to those of an alternative version of this test using phonologically similar but meaningless items (pseudoword condition) and observed how each score varied as a function of age and sex. Then, we collected the participants' self-reported strategies to retain the word and pseudoword lists and examined if these strategies mediated the age and sex effects on memory scores. The word condition resulted in higher memory scores than pseudoword condition at each memory stage and even canceled out, for the learning stage, the detrimental effect of age that was observed for the short-term and DR. When taking sex into account, the word advantage was observed only in women for STR. The self-reported strategies, which were similar for words and pseudowords, were based on the position of the item on the list (word: 53%, pseudoword: 37%) or the meaning of the item (word: 64%, pseudoword: 58%) and were used alone or in combination. The best memory performance was associated with the meaning strategy in the word condition and with the combination of the meaning and position strategies in the pseudoword condition. Finally, we found that the word advantage observed in women for STR was mediated by the use of the meaning strategy. The RAVLT scores were thus highly dependent on word meaning, notably because it allowed efficient semantic knowledge-based strategies. Within the framework of Tulving's declarative memory model, these results are at odds with the depiction of the RAVLT as a verbal episodic memory test as it is increasingly referred to in the literature.

Project description:BackgroundTo overcome the limitations of conventional malaria rapid diagnostic tests (cRDTs) in diagnosing malaria in patients with low parasitaemia, ultrasensitive malaria rapid diagnostic tests (uRDTs) have recently been developed, with promising results under laboratory conditions. The current study is the first meta-analysis comparing the overall sensitivity, and specificity of newly developed ultrasensitive Plasmodium falciparum malaria RDT (Alere™ Ultra-sensitive Malaria Ag P. falciparum RDT) with the cRDT conducted in the same field conditions.MethodsPubMed, EMBASE, Cochrane infectious diseases group specialized register, and African Journals Online (AJOL) were searched up to 20th April 2021. Studies with enough data to compute sensitivity and specificity of uRDT and cRDT were retrieved. A random-effect model for meta-analysis was used to obtain the pooled sensitivity and specificity.ResultsOverall, 15 data sets from 14 studies were included in the meta-analysis. The overall sensitivity of the Alere™ ultra-sensitive Malaria Ag P. falciparum RDT regardless of the reference test and the clinical presentation of participants, was 55.5% (95% confidence interval [CI]: 45.5; 65.0), while the sensitivity regardless of the reference test and the clinical presentation of participants, was 42.9% (95% CI: 31.5; 55.2) for the cRDT performed in the same field conditions. When PCR was used as reference test, the sensitivity of uRDT was 60.4% (95% CI: 50.8; 69.2), while the sensitivity was 49.4% (95% CI: 38.2; 60.6) for the cRDT. The pooled specificity of uRDT regardless of the reference test and the clinical presentation of participants was 98.6% (95% CI: 97.1; 99.4), and the pooled specificity of cRDT regardless of the reference test and the clinical presentation of participants was 99.3% (95% CI: 98.1; 99.7). When PCR was used as reference test the specificity of uRDT and cRDT was 97.5% (95% CI: 94.1; 98.9) and 98.2% (95% CI: 95.5; 99.3). Regardless of the reference test used, the sensitivity of Alere™ Ultra-sensitive Malaria Ag P. falciparum RDT in symptomatic patients was 72.1% (95%CI: 67.4; 76.4), while sensitivity of cRDT was 67.4% (95%CI: 57.6; 75.9).ConclusionFindings of the meta-analysis show that Alere™ Ultra-sensitive Malaria Ag P. falciparum RDT compared to cRDT performed in the same field conditions has higher sensitivity but lower specificity although the difference is not statistically significant.

Dataset Information

Threshold-free measures for assessing the performance of medical screening tests.

Background

Methods

Results

Conclusion

Publications

Threshold-free measures for assessing the performance of medical screening tests.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure