Assessing diagnostic tests: how to correct for the combined effects of interpretation and reference standard.
Ontology highlight
ABSTRACT:
SUBMITTER: Omurtag A
PROVIDER: S-EPMC3530612 | biostudies-literature | 2012
REPOSITORIES: biostudies-literature
Publications
Assessing diagnostic tests: how to correct for the combined effects of interpretation and reference standard.
Omurtag Ahmet A Fenton Andre A AA
PloS one 20121226 12
We describe a general solution to the problem of determining diagnostic accuracy without the use of a perfect reference standard and in the presence of interpreter variability. The accuracy of a diagnostic test is typically determined by comparing its outcomes with those of an established reference standard. But the accuracy of the standard itself and those of the interpreters strongly influence such assessments. We use our solution to examine the effects of the properties of the standard, the r ...[more]
Similar Datasets
Project description:BACKGROUND: An outbreak of severe acute respiratory syndrome (SARS) began in Canada in February 2003. The initial diagnosis of SARS was based on clinical and epidemiological criteria. During the outbreak, molecular and serologic tests for the SARS-associated coronavirus (SARS-CoV) became available. However, without a "gold standard," it was impossible to determine the usefulness of these tests. We describe how these tests were used during the first phase of the SARS outbreak in Toronto and offer some recommendations that may be useful if SARS returns. METHODS: We examined the results of all diagnostic laboratory tests used in 117 patients admitted to hospitals in Toronto who met the Health Canada criteria for suspect or probable SARS. Focusing on tests for SARS-CoV, we attempted to determine the optimal specimen types and timing of specimen collection. RESULTS: Diagnostic test results for SARS-CoV were available for 110 of the 117 patients. SARS-CoV was detected by means of reverse-transcriptase polymerase chain reaction (RT-PCR) in at least one specimen in 59 (54.1%) of 109 patients. Serologic test results of convalescent samples were positive in 50 (96.2%) of 52 patients for whom paired serum samples were collected during the acute and convalescent phases of the illness. Of the 110 patients, 78 (70.9%) had specimens that tested positive by means of RT-PCR, serologic testing or both methods. The proportion of RT-PCR test results that were positive was similar between patients who met the criteria for suspect SARS (50.8%, 95% confidence interval [CI] 38.4%-63.2%) and those who met the criteria for probable SARS (58.0%, 95% CI 44.2%-70.7%). SARS-CoV was detected in nasopharyngeal swabs in 33 (32.4%) of 102 patients, in stool specimens in 19 (63.3%) of 30 patients, and in specimens from the lower respiratory tract in 10 (58.8%) of 17 patients. INTERPRETATION: These findings suggest that the rapid diagnostic tests in use at the time of the initial outbreak lack sufficient sensitivity to be used clinically to rule out SARS. As tests for SARS-CoV continue to be optimized, evaluation of the clinical presentation and elucidation of a contact history must remain the cornerstone of SARS diagnosis. In patients with SARS, specimens taken from the lower respiratory tract and stool samples test positive by means of RT-PCR more often than do samples taken from other areas.
| S-EPMC305313 | biostudies-literature
Project description:PurposeThe diagnostic performance of thyroid biopsy is influenced by several factors, including differences in the Bethesda categorization for malignancy, the inclusion or exclusion of non-diagnostic results, the definition used for the final diagnosis, and the definition of an inconclusive diagnosis. The purpose of this study was to provide an understanding of the factors influencing the diagnostic performance of thyroid biopsy.MethodsWe collected data retrospectively between January and December 2013 from a cohort of 6,762 thyroid nodules from 6,493 consecutive patients who underwent biopsy. In total, 4,822 nodules from 4,553 patients were included. We calculated the biopsy sensitivity according to the inclusion of different Bethesda categories in the numerator and the exclusion of non-diagnostic results, as well as the diagnostic accuracy according to different definitions of a benign diagnosis. We obtained the conclusive and inconclusive diagnosis rates.ResultsThe sensitivity increased when more Bethesda categories were included in the numerator and when non-diagnostic results were excluded. When a benign thyroid nodule diagnosis was defined as benign findings on surgical resection, concordant benign results on at least two occasions, or an initial benign biopsy result and follow-up for more than 12 months, the accuracy was higher than when the diagnosis was based on surgical resection alone (91.1% vs. 68.7%). A higher conclusive diagnosis rate was obtained when Bethesda categories I and III were considered inconclusive than when Bethesda categories I, III and IV were considered inconclusive (78.3% vs. 72.8%, P<0.001).ConclusionUnderstanding the concepts presented herein is important in order to appropriately interpret the diagnostic performance of thyroid biopsy.
| S-EPMC7994740 | biostudies-literature
Project description:BackgroundCrossmatching is used to prevent life-threatening transfusion reactions in horses. Laboratory methods are laborious and technically challenging, which is impractical during emergencies.Hypothesis/objectivesEvaluate agreement between a stall-side crossmatch kit (KIT) and a laboratory method (LAB) in horses with known and unknown blood types.AnimalsTwenty-four blood-typed and alloantibody-screened healthy adult horses (Aim 1) and 156 adult horses of unknown blood type (Aim 2).MethodsProspective, blinded study. Expected positive (n = 35) and negative (n = 36) crossmatches among 24 antibody and blood-typed horses were used to determine sensitivity and specificity of KIT and LAB against the reference method. Agreement in 156 untyped horses was evaluated by reciprocal crossmatch (n = 156).ResultsSensitivity (95% confidence interval [CI]) for LAB and KIT compared with expected reactions was 77.1% (59.9%-90.0%) and 91.4% (77.0%-98.2%), and specificity 77.8% (60.9%-89.9%) and 73.5% (55.6%-87.1%), respectively. The KIT was 100% sensitive for Aa reactions; LAB was 100% sensitive for Qab; and both were 100% sensitive for Ca. Cohen's κ agreement for LAB and KIT with expected positive and negative reactions (n = 71) was moderate (0.55 [0.36-0.74]) and substantial (0.65 [0.47-0.82]), respectively. Agreement was fair comparing LAB with KIT in Aim 1 (0.30 [0.08-0.52]) and in untyped horses in Aim 2 (0.26 [0.11-0.41]).Conclusions and clinical importanceAgreement between KIT and LAB with expected reactions was blood type dependent. Performance of both methods depends on blood type prevalence.
| S-EPMC7096648 | biostudies-literature
Project description:The C57BL/6N inbred lines of mice are widely used in genetic research. They are particularly favoured in large scale studies such as the International Mouse Phenotyping Consortium (IMPC), where C57BL/6N mice are genetically altered to generate a collection of null alleles (currently more than 8500 null alleles have been generated). In this project, mice carrying null alleles are subjected to a pipeline of broad-based phenotyping tests to produce wide ranging phenotyping data on each model. We have previously described the development of a Home Cage Analysis system that automatically tracks the activity of group housed mice from a microchip inserted in the groin. This platform allows assessment of multiple biologically relevant phenotypes over long periods of time without experimenter interference, and therefore is particularly suited for high through-put studies. To investigate the impact of microchips on other tests carried out in the IMPC pipeline, we inserted microchips in 12 male and 12 female C57BL/6Ntac mice at seven weeks of age. Starting at nine weeks of age these mice underwent standard phenotyping tests, concurrently with 20 unchipped C57BL/6Ntac mice (10 females, 10 males). Tissues from a subset of the microchipped mice (six males and six females), chosen at random, were also sent for histopathological examination at the end of the phenotyping pipeline. No significant impact of insertion of microchip was observed in any of the phenotyping tests apart from bone mineral density measurement at DEXA due to the nature of the microchip. We therefore recommend that the microchip be inserted during the DEXA procedure, after the measurement is taken but before the mouse has recovered from the anaesthetic. This would avoid multiple anaesthetic exposures and prevent the potential variability in DEXA analysis output.
| S-EPMC7194481 | biostudies-literature
Project description:Objective:We assessed the sensitivity and specificity of 8 electronic health record (EHR)-based phenotypes for diabetes mellitus against gold-standard American Diabetes Association (ADA) diagnostic criteria via chart review by clinical experts. Materials and Methods:We identified EHR-based diabetes phenotype definitions that were developed for various purposes by a variety of users, including academic medical centers, Medicare, the New York City Health Department, and pharmacy benefit managers. We applied these definitions to a sample of 173?503 patients with records in the Duke Health System Enterprise Data Warehouse and at least 1 visit over a 5-year period (2007-2011). Of these patients, 22?679 (13%) met the criteria of 1 or more of the selected diabetes phenotype definitions. A statistically balanced sample of these patients was selected for chart review by clinical experts to determine the presence or absence of type 2 diabetes in the sample. Results:The sensitivity (62-94%) and specificity (95-99%) of EHR-based type 2 diabetes phenotypes (compared with the gold standard ADA criteria via chart review) varied depending on the component criteria and timing of observations and measurements. Discussion and Conclusions:Researchers using EHR-based phenotype definitions should clearly specify the characteristics that comprise the definition, variations of ADA criteria, and how different phenotype definitions and components impact the patient populations retrieved and the intended application. Careful attention to phenotype definitions is critical if the promise of leveraging EHR data to improve individual and population health is to be fulfilled.
| S-EPMC6080723 | biostudies-literature
Project description:BackgroundStaquet et al. and Brenner both developed correction methods to estimate the sensitivity and specificity of a binary-response index test when the reference standard is imperfect and its sensitivity and specificity are known. However, to our knowledge, no study has compared the statistical properties of these methods, despite their long application in diagnostic accuracy studies.AimTo compare the correction methods developed by Staquet et al. and Brenner.MethodsSimulations techniques were employed to compare the methods under assumptions that the new test and the reference standard are conditionally independent or dependent given the true disease status of an individual. Three clinical datasets were analysed to understand the impact of using each method to inform clinical decision-making.ResultsUnder the assumption of conditional independence, the Staquet et al. correction method outperforms the Brenner correction method irrespective of the prevalence of disease and whether the performance of the reference standard is better or worse than the index test. However, when the prevalence of the disease is high (> 0.9) or low (< 0.1), the Staquet et al. correction method can produce illogical results (i.e. results outside [0,1]). Under the assumption of conditional dependence; both methods failed to estimate the sensitivity and specificity of the index test especially when the covariance terms between the index test and the reference standard is not close to zero.ConclusionWhen the new test and the imperfect reference standard are conditionally independent, and the sensitivity and specificity of the imperfect reference standard are known, the Staquet et al. correction method outperforms the Brenner method. However, where the prevalence of the target condition is very high or low or the two tests are conditionally dependent, other statistical methods such as latent class approaches should be considered.
| S-EPMC8040223 | biostudies-literature
Project description:Failure of arteriovenous fistulas (AVF) to mature and thrombosis in matured fistulas have been the major causes of morbidity and mortality in hemodialysis patients. Stenosis, which occurs due to adverse remodeling in AVFs, is one of the major underlying factors under both scenarios. Early diagnosis of a stenosis in an AVF can provide an opportunity to intervene in a timely manner for either assisting the maturation process or avoiding the thrombosis. The goal of surveillance strategies was to supplement the clinical evaluation (i.e., physical examination) of the AVF for better and earlier diagnosis of a developing stenosis. Surveillance strategies were mainly based on measurement of functional hemodynamic endpoints, including blood flow (Qa) to the vascular access and venous access pressure (VAP). As the changes in arterial pressure (MAP) affects the level of VAP, the ratio of VAP to MAP (VAPR = VAP/MAP) was used for diagnosis. A Qa < 400-500 mL/min or a VAPR > 0.55 is considered sign of significant stenosis, which requires immediate intervention. However, due to the complex nature of AVFs, the surveillance strategies have failed to consistently detect stenosis under different scenarios. VAPR has been primarily developed to detect outflow stenosis in arteriovenous grafts, and it hasn't been successful in accurate diagnosis of outflow lesions in AVFs. Similarly, AVFs can maintain relatively high blood flow despite the presence of a significant outflow stenosis and thus, Qa has been found to be a better predictor of only inflow lesions. Similar shortcomings have been reported in the detection of functional severity of coronary stenosis using diagnostic endpoints that were based on either flow or pressure. This limitation has been associated with the fact that both pressure and flow change in the presence of a stenosis and thus, hemodynamic diagnostic endpoints that employ only one of these parameters are inherently prone to inaccuracies. Recent attempts have resulted in development of new diagnostic endpoints that can combine the effects of pressure and flow. These new hemodynamic diagnostic endpoints have shown to be better predictors of functional severity of lesions as compared to either flow or pressure based counterparts. In this review article, we discussed the advantages and limitations of current functional and anatomical diagnostic endpoints in AVFs.
| S-EPMC4317629 | biostudies-literature
Project description:BackgroundWe observed that some patients with clinical leptospirosis supported by positive results of rapid tests were negative for leptospirosis on the basis of our diagnostic gold standard, which involves isolation of Leptospira species from blood culture and/or a positive result of a microscopic agglutination test (MAT). We hypothesized that our reference standard was imperfect and used statistical modeling to investigate this hypothesis.MethodsData for 1652 patients with suspected leptospirosis recruited during three observational studies and one randomized control trial that described the application of culture, MAT, immunofluorescence assay (IFA), lateral flow (LF) and/or PCR targeting the 16S rRNA gene were reevaluated using Bayesian latent class models and random-effects meta-analysis.ResultsThe estimated sensitivities of culture alone, MAT alone, and culture plus MAT (for which the result was considered positive if one or both tests had a positive result) were 10.5% (95% credible interval [CrI], 2.7%-27.5%), 49.8% (95% CrI, 37.6%-60.8%), and 55.5% (95% CrI, 42.9%-67.7%), respectively. These low sensitivities were present across all 4 studies. The estimated specificity of MAT alone (and of culture plus MAT) was 98.8% (95% CrI, 92.8%-100.0%). The estimated sensitivities and specificities of PCR (52.7% [95% CrI, 45.2%-60.6%] and 97.2% [95% CrI, 92.0%-99.8%], respectively), lateral flow test (85.6% [95% CrI, 77.5%-93.2%] and 96.2% [95% CrI, 87.7%-99.8%], respectively), and immunofluorescence assay (45.5% [95% CrI, 33.3%-60.9%] and 96.8% [95% CrI, 92.8%-99.8%], respectively) were considerably different from estimates in which culture plus MAT was considered a perfect gold standard test.ConclusionsOur findings show that culture plus MAT is an imperfect gold standard against which to compare alterative tests for the diagnosis of leptospirosis. Rapid point-of-care tests for this infection would bring an important improvement in patient care, but their future evaluation will require careful consideration of the reference test(s) used and the inclusion of appropriate statistical models.
| S-EPMC3393707 | biostudies-literature
Project description:ObjectiveThe Bayesian model plays an important role in diagnostic test evaluation in the absence of the gold standard, which used the external prior distribution of a parameter combined with sample data to yield the posterior distribution of the test characteristics. However, the correlation between diagnostic tests has always been a problem that cannot be ignored in the Bayesian model evaluation. This study will discuss how different Bayesian model, correlation scenarios, and prior distribution affect the outcome.MethodsThe data analyzed in this study was gathered during studies of patients presenting to the Nanjing Chest Hospital with suspected tuberculosis. The diagnostic character of T-SPOT.Tb and KD38 tuberculosis antibody test were evaluated in different Bayesian model, and discharge diagnosis as a gold standard was used to verify the model results in the end.ResultThe comparison of four models under the conditional independence situation found that Bayesian probabilistic constraint model was consistent with the Conditional Covariance Bayesian model. The results were mainly affected by prior information. The sensitivity and specificity of the two tests in Conditional Covariance Bayesian model in prior constraint situation were considerably higher than the Bayesian probabilistic constraint model in prior constraint situation. The results of the four models under the conditional dependence situation were similar to the conditional independence situation; pD was also negative with no prior constraint situation in both model Bayesian probabilistic constraint model and Conditional Covariance Bayesian model. The Deviance Information Criterion of Bayesian probabilistic constraint model was close to model Conditional Covariance Bayesian model, but pD of Conditional Covariance Bayesian model in Prior constraint situation (pD=2.40) was higher than the Bayesian probabilistic constraint model in Prior constraint situation (pD=1.66).ConclusionThe result of Conditional Covariance Bayesian model in prior constraint with conditional independence situation was closest to the result of gold standard evaluation in our data. Both of the two Bayesian methods are the feasible way for the evaluation of diagnostic test in the absence of the gold standard diagnostic. Prior source, priority number, and conditional dependencies should be considered in the method selection, the accuracy of posterior estimation mainly depending on the prior distribution.
| S-EPMC6720053 | biostudies-literature
Project description:The outbreak of novel coronavirus disease 2019 (COVID-19) caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has spread worldwide. To meet the urgent and massive demand for the screening and diagnosis of infected individuals, many in vitro diagnostic assays using nucleic acid tests (NATs) have been urgently authorized by regulators worldwide. A reference standard with a well-characterized concentration or titer is of the utmost importance for the study of limit of detection (LoD), which is a crucial feature for a diagnostic assay. Although several reference standards of plasmids or synthetic RNA have already been announced, a reference standard for inactivated virus particles with an accurate concentration is still needed to evaluate the complete procedure. Here, we performed a collaborative study to estimate the NAT-detectable units as a viral genomic equivalent quantity (GEQ) of an inactivated whole-virus SARS-CoV-2 reference standard candidate using digital PCR (dPCR) on multiple commercialized platforms. The median of the quantification results (4.6 × 105 ± 6.5 × 104 GEQ/mL) was treated as the consensus true value of GEQ of virus particles in the reference standard. This reference standard was then used to challenge the LoDs of six officially approved diagnostic assays. Our study demonstrates that an inactivated whole virus quantified by dPCR can serve as a reference standard and provides a unified solution for assay development, quality control, and regulatory surveillance.
| S-EPMC7737535 | biostudies-literature