Dataset Information

Inter-rater reliability and prospective validation of a clinical prediction rule for SARS-CoV-2 infection.

ABSTRACT:

Objectives

Accurate estimation of the risk of SARS-CoV-2 infection based on bedside data alone has importance to emergency department (ED) operations and throughput. The 13-item CORC (COVID [or coronavirus] Rule-out Criteria) rule had good overall diagnostic accuracy in retrospective derivation and validation. The objective of this study was to prospectively test the inter-rater reliability and diagnostic accuracy of the CORC score and rule (score ≤ 0 negative, > 0 positive) and compare the CORC rule performance with physician gestalt.

Methods

This noninterventional study was conducted at an urban academic ED from February 2021 to March 2021. Two practitioners were approached by research coordinators and asked to independently complete a form capturing the CORC criteria for their shared patient and their gestalt binary prediction of the SARS-CoV-2 test result and confidence (0%-100%). The criterion standard for SARS-CoV-2 was from reverse transcriptase polymerase chain reaction performed on a nasopharyngeal swab. The primary analysis was from weighted Cohen's kappa and likelihood ratios (LRs).

Results

For 928 patients, agreement between observers was good for the total CORC score, κ = 0.613 (95% confidence interval [CI] = 0.579-0.646), and for the CORC rule, κ = 0.644 (95% CI = 0.591-0.697). The agreement for clinician gestalt binary determination of SARs-CoV-2 status was κ = 0.534 (95% CI = 0.437-0.632) with median confidence of 76% (first-third quartile = 66-88.5). For 425 patients who had the criterion standard, a negative CORC rule (both observers scored CORC < 0), the sensitivity was 88%, and specificity was 51%, with a negative LR (LR-) of 0.24 (95% CI = 0.10-0.50). Among patients with a mean CORC score of >4, the prevalence of a positive SARS-CoV-2 test was 58% (95% CI = 28%-85%) and positive LR was 13.1 (95% CI = 4.5-37.2). Clinician gestalt demonstrated a sensitivity of 51% and specificity of 86% with a LR- of 0.57 (95% CI = 0.39-0.74).

Conclusion

In this prospective study, the CORC score and rule demonstrated good inter-rater reliability and reproducible diagnostic accuracy for estimating the pretest probability of SARs-CoV-2 infection.

SUBMITTER: Nevel AE

PROVIDER: S-EPMC8441807 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BACKGROUND:Focus on frailty status has become increasingly important when determining care plans within and across health care sectors. A standardized frailty measure applicable for both primary and secondary health care sectors is needed to provide a common reference point. The aim of this study was to translate the Clinical Frailty Scale (CFS) into Danish (CFS-DK) and test inter-rater reliability for key health care professionals in the primary and secondary sectors using the CFS-DK. METHODS:The Clinical Frailty Scale was translated into Danish using the ISPOR principles for translation and cultural adaptation that included forward and back translation, review by the original developer, and cognitive debriefing. For the validation exercise, 40 participants were asked to rate 15 clinical case vignettes using the CFS-DK. The raters were distributed across several health care professions: primary care physicians (n?=?10), community nurses (n?=?10), hospital doctors from internal medicine (n?=?10) and intensive care (n?=?10). Inter-rater reliability was assessed using intraclass correlation coefficients (ICC), and sensitivity analysis was performed using multilevel random effects linear regression. RESULTS:The Clinical Frailty Scale was translated and culturally adapted into Danish and is presented in this paper in its final form. Inter-rater reliability in the four professional groups ranged from ICC 0.81 to 0.90. Sensitivity analysis showed no significant impact of professional group or length of clinical experience. The health care professionals considered the CFS-DK to be relevant for their own area of work and for cross-sectoral collaboration. CONCLUSION:The Clinical Frailty Scale was translated and culturally adapted into Danish. The inter-rater reliability was high in all four groups of health care professionals involved in cross-sectoral collaborations. However, the use of case vignettes may reduce the generalizability of the reliability findings to real-life settings. The CFS has the potential to serve as a common reference tool when treating and rehabilitating older patients.

Project description:BackgroundThe statistical validities of the official Japanese classifications of activities of daily living (ADLs), including bedriddenness ranks (BR) and cognitive function scores (CFS), have yet to be assessed. To this aim, we evaluated the ability of BR and CFS to assess ADLs using inter-rater reliability and criterion-related validity.MethodsNew inpatients aged ≥75 years were enrolled in this hospital-based prospective observational study. BR and CFS were assessed once by an attending nurse, and then by a social worker/medical clerk. We evaluated inter-rater reliability between different professions by calculating the concordance rate, kappa coefficient, Cronbach's α, and intraclass correlation coefficient. We also estimated the relationship of the Barthel Index and Katz Index with the BR and CFS using Spearman's correlation coefficients.ResultsFor the 271 patients enrolled, BR at the first assessment revealed 66 normal, 10 of J1, 15 of J2, 18 of A1, 31 of A2, 37 of B1, 35 of B2, 22 of C1, and 32 of C2. The concordance rate between the two BR assessments was 68.6%, with a kappa coefficient of 0.61, Cronbach's α of 0.91, and an intraclass correlation coefficient of 0.83, thus showing good inter-rater reliability. BR was negatively correlated with the Barthel Index (r = - 0.848, p < 0.001) and Katz Index (r = - 0.820, p < 0.001), showing justifiable criterion-related validity. Meanwhile, CFS at the first assessment revealed 92 normal, 47 of 1, 19 of 2a, 30 of 2b, 60 of 3a, 8 of 3b, 8 of 4, and 0 of M. The concordance rate between the two CFS assessments was 70.1%, with a kappa coefficient of 0.62, Cronbach's α of 0.87, and an intraclass correlation coefficient 0.78, thus also showing good inter-rater reliability. CFS was negatively correlated with the Barthel Index (r = - 0.667, p < 0.001) and Katz Index (r = - 0.661, p < 0.001), showing justifiable criterion-related validity.ConclusionsBR and CFS could be reliable and easy-to-use grading scales of ADLs in acute clinical practice or large-scale screening, with high inter-rater reliabilities among different professions and significant correlations with well-established, though complicated to use, instruments to assess ADLs.Trial registrationUMIN000041051 (2020/7/10).

Project description:Background: There has been a groundswell of national support for transparent tracking and dissemination of PhD career outcomes. In 2017, individuals from multiple institutions and professional organizations met to create the Unified Career Outcomes Taxonomy (UCOT 2017), a three-tiered taxonomy to help institutions uniformly classify career outcomes of PhD graduates. Early adopters of UCOT 2017, noted ambiguity in some categories of the career taxonomy, raising questions about its consistent application within and across institutions. Methods: To test and evaluate the consistency of UCOT 2017, we calculated inter-rater reliability across two rounds of iterative refinement of the career taxonomy, classifying over 800 PhD alumni records via nine coders. Results: We identified areas of discordance in the taxonomy, and progressively refined UCOT 2017 and an accompanying Guidance Document to improve inter-rater reliability across all three tiers of the career taxonomy. However, differing interpretations of the classifications, especially for faculty classifications in the third tier, resulted in continued discordance among the coders. We addressed this discordance with clarifying language in the Guidance Document, and proposed the addition of a flag system for identification of the title, rank, and prefix of faculty members. This labeling system provides the additional benefit of highlighting the granularity and the intersectionality of faculty job functions, while maintaining the ability to sort by - and report data on - faculty and postdoctoral trainee roles, as is required by some national and federal reporting guidelines. We provide specific crosswalk guidance for how a user may choose to incorporate our suggestions while maintaining the ability to report in accordance with UCOT 2017. Conclusions: Our findings underscore the importance of detailed guidance documents, coder training, and periodic collaborative review of career outcomes taxonomies as PhD careers evolve in the global workforce. Implications for coder-training and use of novice coders are also discussed.

Project description:PurposeFirst, to evaluate inter-rater reliability when human raters estimate the reading performance of visually impaired individuals using the MNREAD acuity chart. Second, to evaluate the agreement between computer-based scoring algorithms and compare them with human rating.MethodsReading performance was measured for 101 individuals with low vision, using the Portuguese version of the MNREAD test. Seven raters estimated the maximum reading speed (MRS) and critical print size (CPS) of each individual MNREAD curve. MRS and CPS were also calculated automatically for each curve using two different algorithms: the original standard deviation method (SDev) and a non-linear mixed effects (NLME) modeling. Intra-class correlation coefficients (ICC) were used to estimate absolute agreement between raters and/or algorithms.ResultsAbsolute agreement between raters was 'excellent' for MRS (ICC = 0.97; 95%CI [0.96, 0.98]) and 'moderate' to 'good' for CPS (ICC = 0.77; 95%CI [0.69, 0.83]). For CPS, inter-rater reliability was poorer among less experienced raters (ICC = 0.70; 95%CI [0.57, 0.80]) when compared to experienced ones (ICC = 0.82; 95%CI [0.76, 0.88]). Absolute agreement between the two algorithms was 'excellent' for MRS (ICC = 0.96; 95%CI [0.91, 0.98]). For CPS, the best possible agreement was found for CPS defined as the print size sustaining 80% of MRS (ICC = 0.77; 95%CI [0.68, 0.84]). Absolute agreement between raters and automated methods was 'excellent' for MRS (ICC = 0.96; 95% CI [0.88, 0.98] for SDev; ICC = 0.97; 95% CI [0.95, 0.98] for NLME). For CPS, absolute agreement between raters and SDev ranged from 'poor' to 'good' (ICC = 0.66; 95% CI [0.3, 0.80]), while agreement between raters and NLME was 'good' (ICC = 0.83; 95% CI [0.76, 0.88]).ConclusionFor MRS, inter-rater reliability is excellent, even considering the possibility of noisy and/or incomplete data collected in low-vision individuals. For CPS, inter-rater reliability is lower. This may be problematic, for instance in the context of multisite investigations or follow-up examinations. The NLME method showed better agreement with the raters than the SDev method for both reading parameters. Setting up consensual guidelines to deal with ambiguous curves may help improve reliability. While the exact definition of CPS should be chosen on a case-by-case basis depending on the clinician or researcher's motivations, evidence suggests that estimating CPS as the smallest print size sustaining about 80% of MRS would increase inter-rater reliability.