Dataset Information

Threshold-free measures for assessing the performance of medical screening tests.

ABSTRACT:

Background

The area under the receiver operating characteristic curve (AUC) is frequently used as a performance measure for medical tests. It is a threshold-free measure that is independent of the disease prevalence rate. We evaluate the utility of the AUC against an alternate measure called the average positive predictive value (AP), in the setting of many medical screening programs where the disease has a low prevalence rate.

Methods

We define the two measures using a common notation system and show that both measures can be expressed as a weighted average of the density function of the diseased subjects. The weights for the AP include prevalence in some form, but those for the AUC do not. These measures are compared using two screening test examples under rare and common disease prevalence rates.

Results

The AP measures the predictive power of a test, which varies when the prevalence rate changes, unlike the AUC, which is prevalence independent. The relationship between the AP and the prevalence rate depends on the underlying screening/diagnostic test. Therefore, the AP provides relevant information to clinical researchers and regulators about how a test is likely to perform in a screening population.

Conclusion

The AP is an attractive alternative to the AUC for the evaluation and comparison of medical screening tests. It could improve the effectiveness of screening programs during the planning stage.

SUBMITTER: Yuan Y

PROVIDER: S-EPMC4403252 | biostudies-literature | 2015

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Threshold-free measures for assessing the performance of medical screening tests.

Yuan Yan Y Su Wanhua W Zhu Mu M

Frontiers in public health 20150420

<h4>Background</h4>The area under the receiver operating characteristic curve (AUC) is frequently used as a performance measure for medical tests. It is a threshold-free measure that is independent of the disease prevalence rate. We evaluate the utility of the AUC against an alternate measure called the average positive predictive value (AP), in the setting of many medical screening programs where the disease has a low prevalence rate.<h4>Methods</h4>We define the two measures using a common not ...[more]

PMID: 25941668

Similar Datasets

Project description:Overall accuracy measures of medical tests are often used with unclear interpretations. To develop methods of calculating the overall accuracy of medical tests in the patient population. Algebraic equations based on Bayes' theorem. A new approach is proposed for calculating overall accuracy in the patient population. Examples and applications using published data are presented. The overall accuracy is the proportion of the correct test results. We introduce a clear distinction between the overall accuracy measures of medical tests that are aimed at the detection of a disease in a screening of populations for public health purposes in the general population and the overall accuracy measures of tests aimed at determining a diagnosis in individuals in a clinical setting. We show that the overall detection accuracy measure is obtained in a specific study that explores test accuracy among persons with known diagnoses and may be useful for public health screening tests. It is different from the overall diagnostic accuracy that could be calculated in the clinical setting for the evaluation of medical tests aimed at determining the individual patients' diagnoses. We show that the overall detection accuracy is constant and is not affected by the prevalence of the disease. In contrast, the overall diagnostic accuracy changes and is dependent on the prevalence. Moreover, it ranges according to the ratio between the sensitivity and specificity. Thus, when the sensitivity is greater than the specificity, the overall diagnostic accuracy increases with increasing prevalence, and vice versa, that is, when the sensitivity is lower than the specificity, the overall diagnostic accuracy decreases with increasing prevalence so that another test might be more useful for diagnostic procedures. Our paper suggests a new and more appropriate methodology for estimating the overall diagnostic accuracy of any medical test. This may be important for helping clinicians avoid errors.

Project description:BackgroundClinical-performance measurement has helped improve the quality of health-care; yet success in attaining high levels of quality across multiple domains simultaneously still varies considerably. Although many sources of variability in care quality have been studied, the difficulty required to complete the clinical work itself has received little attention.ObjectiveWe present a task-based methodology for evaluating the difficulty of clinical-performance measures (CPMs) by assessing the complexity of their component requisite tasks.DesignUsing Functional Job Analysis (FJA), subject-matter experts (SMEs) generated task lists for 17 CPMs; task lists were rated on ten dimensions of complexity, and then aggregated into difficulty composites.ParticipantsEleven outpatient work SMEs; 133 VA Medical Centers nationwide.Main measuresClinical Performance: 17 outpatient CPMs (2000-2008) at 133 VA Medical Centers nationwide. Measure Difficulty: for each CPM, the number of component requisite tasks and the average rating across ten FJA complexity scales for the set of tasks comprising the measure.Key resultsMeasures varied considerably in the number of component tasks (M = 10.56, SD = 6.25, min = 5, max = 25). Measures of chronic care following acute myocardial infarction exhibited significantly higher measure difficulty ratings compared to diabetes or screening measures, but not to immunization measures ([Formula: see text] = 0.45, -0.04, -0.05, and -0.06 respectively; F (3, 186) = 3.57, p = 0.015). Measure difficulty ratings were not significantly correlated with the number of component tasks (r = -0.30, p = 0.23).ConclusionsEvaluating the difficulty of achieving recommended CPM performance levels requires more than simply counting the tasks involved; using FJA to assess the complexity of CPMs' component tasks presents an alternate means of assessing the difficulty of primary-care CPMs and accounting for performance variation among measures and performers. This in turn could be used in designing performance reward programs, or to match workflow to clinician time and effort.

Project description:Supraspan verbal list-learning tests, such as the Rey Auditory Verbal Learning Test (RAVLT), are classic neuropsychological tests for assessing verbal memory. In this study, we investigated the impact of the meaning of the words to be learned on three memory stages [short-term recall (STR), learning, and delayed recall (DR)] in a cohort of 447 healthy adults. First, we compared scores obtained from the RAVLT (word condition) to those of an alternative version of this test using phonologically similar but meaningless items (pseudoword condition) and observed how each score varied as a function of age and sex. Then, we collected the participants' self-reported strategies to retain the word and pseudoword lists and examined if these strategies mediated the age and sex effects on memory scores. The word condition resulted in higher memory scores than pseudoword condition at each memory stage and even canceled out, for the learning stage, the detrimental effect of age that was observed for the short-term and DR. When taking sex into account, the word advantage was observed only in women for STR. The self-reported strategies, which were similar for words and pseudowords, were based on the position of the item on the list (word: 53%, pseudoword: 37%) or the meaning of the item (word: 64%, pseudoword: 58%) and were used alone or in combination. The best memory performance was associated with the meaning strategy in the word condition and with the combination of the meaning and position strategies in the pseudoword condition. Finally, we found that the word advantage observed in women for STR was mediated by the use of the meaning strategy. The RAVLT scores were thus highly dependent on word meaning, notably because it allowed efficient semantic knowledge-based strategies. Within the framework of Tulving's declarative memory model, these results are at odds with the depiction of the RAVLT as a verbal episodic memory test as it is increasingly referred to in the literature.

Project description:While force-velocity-power characteristics of resistance exercises, such as bench presses and squats, have been well documented, little attention has been paid to load, force, and power-velocity relationships in exercises engaging core muscles. Given that power produced during lifting tasks or trunk rotations plays an important role in most sport-specific and daily life activities, its measurement should represent an important part of the test battery in both athletes and the general population. The aim of this scoping review was 1) to map the literature related to testing methods assessing core muscle strength and stability in sport and rehabilitation, chiefly studies with particular focus on force-velocity-power characteristics of exercises involving the use of core muscles, 2) and to identify gaps in existing studies and suggest further research in this field. The literature search was conducted on Cochrane Library databases, Scopus, Web of Science, PubMed and MEDLINE, which was completed by SpringerLink, Google Scholar and Elsevier. The inclusion criteria were met in 37 articles. Results revealed that among a variety of studies investigating the core stability and core strength in sport and rehabilitation, only few of them analyzed force-velocity-power characteristics of exercises involving the use of core muscles. Most of them evaluated maximal isometric strength of the core and its endurance. However, there are some studies that assessed muscle power during lifting tasks at different loads performed either with free weights or using the Smith machine. Similarly, power and velocity were assessed during trunk rotations performed with different weights when standing or sitting. Nevertheless, there is still scant research investigating the power-velocity and force-velocity relationship during exercises engaging core muscles in able-bodied and para athletes with different demands on stability and strength of the core. Therefore, more research is needed to address this gap in the literature and aim research at assessing strength and power-related measures within cross-sectional and intervention studies. A better understanding of the power-force-velocity profiles during exercises with high demands on the core musculature has implications for designing sport training and rehabilitation programs for enhancement of athletes' performance and/or decrease their risk of back pain.

Project description:BackgroundSerological screening tests play a crucial role to diagnose gambiense human African trypanosomiasis (gHAT). Presently, they preselect individuals for microscopic confirmation, but in future "screen and treat" strategies they will identify individuals for treatment. Variability in reported specificities, the development of new rapid diagnostic tests (RDT) and the hypothesis that malaria infection may decrease RDT specificity led us to evaluate the specificity of 5 gHAT screening tests.MethodsDuring active screening, venous blood samples from 1095 individuals from Côte d'Ivoire and Guinea were tested consecutively with commercial (CATT, HAT Sero-K-SeT, Abbott Bioline HAT 2.0) and prototype (DCN HAT RDT, HAT Sero-K-SeT 2.0) gHAT screening tests and with a malaria RDT. Individuals with ≥ 1 positive gHAT screening test underwent microscopy and further immunological (trypanolysis with T.b. gambiense LiTat 1.3, 1.5 and 1.6; indirect ELISA/T.b. gambiense; T.b. gambiense inhibition ELISA with T.b. gambiense LiTat 1.3 and 1.5 VSG) and molecular reference laboratory tests (PCR TBRN3, 18S and TgsGP; SHERLOCK 18S Tids, 7SL Zoon, and TgsGP; Trypanozoon S2-RT-qPCR 18S2, 177T, GPI-PLC and TgsGP in multiplex; RT-qPCR DT8, DT9 and TgsGP in multiplex). Microscopic trypanosome detection confirmed gHAT, while other individuals were considered gHAT free. Differences in fractions between groups were assessed by Chi square and differences in specificity between 2 tests on the same individuals by McNemar.ResultsOne gHAT case was diagnosed. Overall test specificities (n = 1094) were: CATT 98.9% (95% CI: 98.1-99.4%); HAT Sero-K-SeT 86.7% (95% CI: 84.5-88.5%); Bioline HAT 2.0 82.1% (95% CI: 79.7-84.2%); DCN HAT RDT 78.2% (95% CI: 75.7-80.6%); and HAT Sero-K-SeT 2.0 78.4% (95% CI: 75.9-80.8%). In malaria positives, gHAT screening tests appeared less specific, but the difference was significant only in Guinea for Abbott Bioline HAT 2.0 (P = 0.03) and HAT Sero-K-Set 2.0 (P = 0.0006). The specificities of immunological and molecular laboratory tests in gHAT seropositives were 98.7-100% (n = 399) and 93.0-100% (n = 302), respectively. Among 44 reference laboratory test positives, only the confirmed gHAT patient and one screening test seropositive combined immunological and molecular reference laboratory test positivity.ConclusionsAlthough a minor effect of malaria cannot be excluded, gHAT RDT specificities are far below the 95% minimal specificity stipulated by the WHO target product profile for a simple diagnostic tool to identify individuals eligible for treatment. Unless specificity is improved, an RDT-based "screen and treat" strategy would result in massive overtreatment. In view of their inconsistent results, additional comparative evaluations of the diagnostic performance of reference laboratory tests are indicated for better identifying, among screening test positives, those at increased suspicion for gHAT.Trial registrationThe trial was retrospectively registered under NCT05466630 in clinicaltrials.gov on July 15 2022.

Project description:BackgroundTo overcome the limitations of conventional malaria rapid diagnostic tests (cRDTs) in diagnosing malaria in patients with low parasitaemia, ultrasensitive malaria rapid diagnostic tests (uRDTs) have recently been developed, with promising results under laboratory conditions. The current study is the first meta-analysis comparing the overall sensitivity, and specificity of newly developed ultrasensitive Plasmodium falciparum malaria RDT (Alere™ Ultra-sensitive Malaria Ag P. falciparum RDT) with the cRDT conducted in the same field conditions.MethodsPubMed, EMBASE, Cochrane infectious diseases group specialized register, and African Journals Online (AJOL) were searched up to 20th April 2021. Studies with enough data to compute sensitivity and specificity of uRDT and cRDT were retrieved. A random-effect model for meta-analysis was used to obtain the pooled sensitivity and specificity.ResultsOverall, 15 data sets from 14 studies were included in the meta-analysis. The overall sensitivity of the Alere™ ultra-sensitive Malaria Ag P. falciparum RDT regardless of the reference test and the clinical presentation of participants, was 55.5% (95% confidence interval [CI]: 45.5; 65.0), while the sensitivity regardless of the reference test and the clinical presentation of participants, was 42.9% (95% CI: 31.5; 55.2) for the cRDT performed in the same field conditions. When PCR was used as reference test, the sensitivity of uRDT was 60.4% (95% CI: 50.8; 69.2), while the sensitivity was 49.4% (95% CI: 38.2; 60.6) for the cRDT. The pooled specificity of uRDT regardless of the reference test and the clinical presentation of participants was 98.6% (95% CI: 97.1; 99.4), and the pooled specificity of cRDT regardless of the reference test and the clinical presentation of participants was 99.3% (95% CI: 98.1; 99.7). When PCR was used as reference test the specificity of uRDT and cRDT was 97.5% (95% CI: 94.1; 98.9) and 98.2% (95% CI: 95.5; 99.3). Regardless of the reference test used, the sensitivity of Alere™ Ultra-sensitive Malaria Ag P. falciparum RDT in symptomatic patients was 72.1% (95%CI: 67.4; 76.4), while sensitivity of cRDT was 67.4% (95%CI: 57.6; 75.9).ConclusionFindings of the meta-analysis show that Alere™ Ultra-sensitive Malaria Ag P. falciparum RDT compared to cRDT performed in the same field conditions has higher sensitivity but lower specificity although the difference is not statistically significant.

Dataset Information

Threshold-free measures for assessing the performance of medical screening tests.

Background

Methods

Results

Conclusion

Publications

Threshold-free measures for assessing the performance of medical screening tests.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets