Dataset Information

Consensus conference follow-up: inter-rater reliability assessment of the Best Evidence in Emergency Medicine (BEEM) rater scale, a medical literature rating tool for emergency physicians.

ABSTRACT: Studies published in general and specialty medical journals have the potential to improve emergency medicine (EM) practice, but there can be delayed awareness of this evidence because emergency physicians (EPs) are unlikely to read most of these journals. Also, not all published studies are intended for or ready for clinical practice application. The authors developed "Best Evidence in Emergency Medicine" (BEEM) to ameliorate these problems by searching for, identifying, appraising, and translating potentially practice-changing studies for EPs. An initial step in the BEEM process is the BEEM rater scale, a novel tool for EPs to collectively evaluate the relative clinical relevance of EM-related studies found in more than 120 journals. The BEEM rater process was designed to serve as a clinical relevance filter to identify those studies with the greatest potential to affect EM practice. Therefore, only those studies identified by BEEM raters as having the highest clinical relevance are selected for the subsequent critical appraisal process and, if found methodologically sound, are promoted as the best evidence in EM.The primary objective was to measure inter-rater reliability (IRR) of the BEEM rater scale. Secondary objectives were to determine the minimum number of EP raters needed for the BEEM rater scale to achieve acceptable reliability and to compare performance of the scale against a previously published evidence rating system, the McMaster Online Rating of Evidence (MORE), in an EP population.The authors electronically distributed the title, conclusion, and a PubMed link for 23 recently published studies related to EM to a volunteer group of 134 EPs. The volunteers answered two demographic questions and rated the articles using one of two randomly assigned seven-point Likert scales, the BEEM rater scale (n = 68) or the MORE scale (n = 66), over two separate administrations. The IRR of each scale was measured using generalizability theory.The IRR of the BEEM rater scale ranged between 0.90 (95% confidence interval [CI] = 0.86 to 0.93) to 0.92 (95% CI = 0.89 to 0.94) across administrations. Decision studies showed a minimum of 12 raters is required for acceptable reliability of the BEEM rater scale. The IRR of the MORE scale was 0.82 to 0.84.The BEEM rater scale is a highly reliable, single-question tool for a small number of EPs to collectively rate the relative clinical relevance within the specialty of EM of recently published studies from a variety of medical journals. It compares favorably with the MORE system because it achieves a high IRR despite simply requiring raters to read each article's title and conclusion.

SUBMITTER: Worster A

PROVIDER: S-EPMC3240997 | biostudies-literature | 2011 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Consensus conference follow-up: inter-rater reliability assessment of the Best Evidence in Emergency Medicine (BEEM) rater scale, a medical literature rating tool for emergency physicians.

Worster Andrew A Kulasegaram Kulamakan K Carpenter Christopher R CR Vallera Teresa T Upadhye Suneel S Sherbino Jonathan J Haynes R Brian RB

Academic emergency medicine : official journal of the Society for Academic Emergency Medicine 20111101 11

<h4>Background</h4>Studies published in general and specialty medical journals have the potential to improve emergency medicine (EM) practice, but there can be delayed awareness of this evidence because emergency physicians (EPs) are unlikely to read most of these journals. Also, not all published studies are intended for or ready for clinical practice application. The authors developed "Best Evidence in Emergency Medicine" (BEEM) to ameliorate these problems by searching for, identifying, appra ...[more]

PMID: 22092904

Similar Datasets

Project description:Background: Arterial Doppler Ultrasound waveform (DW) analysis allows the detection and evaluation of lower extremity peripheral artery disease. The high heterogeneity of the reported description of DW is reduced by the use of classification. However, the reliability of these classifications is either unknown or low to moderate and practices of vascular caregivers regarding the use of these classifications remain unknown. Aims: This study aims to assess the inter-observer reliability of the Saint-Bonnet classification, a 13-category DW classification. The secondary objective was to determine the utilization rate of the most common classifications and the ability of these classifications to describe DW. Methods: A national survey was conducted among all vascular physicians of French society of vascular medicine. They were invited by email to describe on a website 20 DW without and with the display of the Saint-Bonnet classification. The reliability of this classification was estimated by Fleiss' Kappa expressed with [95% confidence interval]. A semantic analysis allowed us to classify the physicians' responses according to the terms used. Finally we have evaluated for each classification the rate of misuse, i.e., the addition of a complementary term to the defined categories. Results: One hundred and ten physicians participated and only 5% of these were familiar with Saint-Bonnet classification. Fifty-four percent of vascular physicians used no classification at all. Vascular physicians used the Spronk (four-category), Descotes (five-category) and Saint-Bonnet (13-category) classifications for respectively, 31, 10, and 5%. Kappa coefficient of Fleiss (κ) was 0.546 [0.544-0.547] (p < 0.001). Reliability by category ranges from κ of 0.075 to 0.864. In multivariate analysis, the use of a classification was associated with fewer years of experience and was dependent on geographic location. Misuse rate by classification was 88, 82, and 5% using Spronk, Descotes and Saint-Bonnet classifications respectively. Conclusion: The reliability of Saint-Bonnet classification is weak to moderate by vascular physicians who are not familiar with its use. However, unlike the other classifications, it seems to be sufficiently precise so that the user does not need to complete its description. There is a significant heterogeneity in the use of arterial Doppler classifications in France.

Project description:IntroductionAs frailty among the elderly is receiving increasing attention in emergency departments (EDs) around the world, the use of frailty assessment tools is growing. An often used such tool is the Clinical Frailty Scale (CFS), whose inter-rater reliability has been sparingly investigated in ED settings. No inter-rater reliability study regarding CFS has previously been performed within the Scandinavian ED context. The primary aim of this study was to evaluate the inter-rater reliability of the CFS in a Swedish ED setting.MethodsThis was a prospective observational study conducted at three Swedish EDs. Patients ≥65 years were independently assessed with CFS by their responsible physician, registered nurse, and assistant nurse. Demographic information for each assessor was collected, along with frailty status (frail/not frail) on the basis of clinical judgment. Inter-rater reliability was calculated using intraclass correlation coefficient (ICC), whereas agreement of frailty assessed by CFS (dichotomized between frail/not frail, cutoff at ≥5 points) versus solely by clinical judgment was calculated by using cross-tabulation.ResultsOne-hundred patients were included. We found inter-rater reliability to be moderate to good (ICC 0.78, 95% confidence interval [CI] 0.72-0.84), regardless of whether the care team included an emergency physician (ICC 0.74, 95% CI 0.62-0.83) or an intern/resident from another specialty (ICC 0.83, 95% CI 0.74-0.89). The agreement of clinically judged frailty compared to frailty according to CFS was 84%. In the opposing cases, staff tended to assess patients as frail to a higher extent using clinical judgment than by applying CFS on the same patient.ConclusionsThe CFS appears to have a moderate to good inter-rater reliability when used in a clinical ED setting. When guiding clinical decisions, we advise that the CFS score should be discussed within the team. Further research needs to be performed on the accuracy of clinical judgment to identify frailty in ED patients.

Project description:Background: There has been a groundswell of national support for transparent tracking and dissemination of PhD career outcomes. In 2017, individuals from multiple institutions and professional organizations met to create the Unified Career Outcomes Taxonomy (UCOT 2017), a three-tiered taxonomy to help institutions uniformly classify career outcomes of PhD graduates. Early adopters of UCOT 2017, noted ambiguity in some categories of the career taxonomy, raising questions about its consistent application within and across institutions. Methods: To test and evaluate the consistency of UCOT 2017, we calculated inter-rater reliability across two rounds of iterative refinement of the career taxonomy, classifying over 800 PhD alumni records via nine coders. Results: We identified areas of discordance in the taxonomy, and progressively refined UCOT 2017 and an accompanying Guidance Document to improve inter-rater reliability across all three tiers of the career taxonomy. However, differing interpretations of the classifications, especially for faculty classifications in the third tier, resulted in continued discordance among the coders. We addressed this discordance with clarifying language in the Guidance Document, and proposed the addition of a flag system for identification of the title, rank, and prefix of faculty members. This labeling system provides the additional benefit of highlighting the granularity and the intersectionality of faculty job functions, while maintaining the ability to sort by - and report data on - faculty and postdoctoral trainee roles, as is required by some national and federal reporting guidelines. We provide specific crosswalk guidance for how a user may choose to incorporate our suggestions while maintaining the ability to report in accordance with UCOT 2017. Conclusions: Our findings underscore the importance of detailed guidance documents, coder training, and periodic collaborative review of career outcomes taxonomies as PhD careers evolve in the global workforce. Implications for coder-training and use of novice coders are also discussed.

Project description:PurposeFirst, to evaluate inter-rater reliability when human raters estimate the reading performance of visually impaired individuals using the MNREAD acuity chart. Second, to evaluate the agreement between computer-based scoring algorithms and compare them with human rating.MethodsReading performance was measured for 101 individuals with low vision, using the Portuguese version of the MNREAD test. Seven raters estimated the maximum reading speed (MRS) and critical print size (CPS) of each individual MNREAD curve. MRS and CPS were also calculated automatically for each curve using two different algorithms: the original standard deviation method (SDev) and a non-linear mixed effects (NLME) modeling. Intra-class correlation coefficients (ICC) were used to estimate absolute agreement between raters and/or algorithms.ResultsAbsolute agreement between raters was 'excellent' for MRS (ICC = 0.97; 95%CI [0.96, 0.98]) and 'moderate' to 'good' for CPS (ICC = 0.77; 95%CI [0.69, 0.83]). For CPS, inter-rater reliability was poorer among less experienced raters (ICC = 0.70; 95%CI [0.57, 0.80]) when compared to experienced ones (ICC = 0.82; 95%CI [0.76, 0.88]). Absolute agreement between the two algorithms was 'excellent' for MRS (ICC = 0.96; 95%CI [0.91, 0.98]). For CPS, the best possible agreement was found for CPS defined as the print size sustaining 80% of MRS (ICC = 0.77; 95%CI [0.68, 0.84]). Absolute agreement between raters and automated methods was 'excellent' for MRS (ICC = 0.96; 95% CI [0.88, 0.98] for SDev; ICC = 0.97; 95% CI [0.95, 0.98] for NLME). For CPS, absolute agreement between raters and SDev ranged from 'poor' to 'good' (ICC = 0.66; 95% CI [0.3, 0.80]), while agreement between raters and NLME was 'good' (ICC = 0.83; 95% CI [0.76, 0.88]).ConclusionFor MRS, inter-rater reliability is excellent, even considering the possibility of noisy and/or incomplete data collected in low-vision individuals. For CPS, inter-rater reliability is lower. This may be problematic, for instance in the context of multisite investigations or follow-up examinations. The NLME method showed better agreement with the raters than the SDev method for both reading parameters. Setting up consensual guidelines to deal with ambiguous curves may help improve reliability. While the exact definition of CPS should be chosen on a case-by-case basis depending on the clinician or researcher's motivations, evidence suggests that estimating CPS as the smallest print size sustaining about 80% of MRS would increase inter-rater reliability.

Dataset Information

Consensus conference follow-up: inter-rater reliability assessment of the Best Evidence in Emergency Medicine (BEEM) rater scale, a medical literature rating tool for emergency physicians.

Publications

Consensus conference follow-up: inter-rater reliability assessment of the Best Evidence in Emergency Medicine (BEEM) rater scale, a medical literature rating tool for emergency physicians.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets