Dataset Information

Inter-rater reliability of the American Society of Anesthesiologists physical status rating for emergency gastrointestinal surgery.

ABSTRACT: Aim:The American Society of Anesthesiologists Physical Status (ASA-PS) classification system is used worldwide and has also been incorporated into various prediction rules. However, concerns have been raised regarding inter-rater agreement in various surgical fields. Although emergency gastrointestinal surgery is relatively common and associated with high postoperative mortality, a reliability study has not yet been undertaken in this field. The aim of the present study was to investigate the inter-rater reliability of ASA-PS for emergency gastrointestinal surgery. Methods:Three sets of scenarios were generated for each ASA-PS class (2E, 3E, and 4E) in emergency gastrointestinal surgery, resulting in nine scenarios. These scenarios described the preoperative profiles of patients in one hospital. Two or three anesthesiologists from 18 other hospitals provided scores for ASA-PS for each scenario. Results:Fifty anesthesiologists scored the ASA-PS class. Between 66% and 90% of these anesthesiologists assigned the same ratings as the reference ratings for the individual scenarios. Inter-rater reliability was assessed using Fleiss' kappa (95% confidence interval) of 0.55 (0.54-0.56, P < 0.001) and an intraclass correlation coefficient (95% confidence interval) of 0.79 (0.63-0.93, P < 0.001). Conclusion:The results of the present study revealed the consistency of ASA-PS ratings between anesthesiologists for emergency gastrointestinal surgery. The ASA-PS may serve as a reliable variable in the prediction rules for this field.

SUBMITTER: Shichino T

PROVIDER: S-EPMC5667267 | biostudies-literature | 2017 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Inter-rater reliability of the American Society of Anesthesiologists physical status rating for emergency gastrointestinal surgery.

Shichino Tsutomu T Hirao Motohiro M Haga Yoshio Y

Acute medicine & surgery 20160913 2

<h4>Aim</h4>The American Society of Anesthesiologists Physical Status (ASA-PS) classification system is used worldwide and has also been incorporated into various prediction rules. However, concerns have been raised regarding inter-rater agreement in various surgical fields. Although emergency gastrointestinal surgery is relatively common and associated with high postoperative mortality, a reliability study has not yet been undertaken in this field. The aim of the present study was to investigat ...[more]

PMID: 29123855

Similar Datasets

Project description:BackgroundStudies published in general and specialty medical journals have the potential to improve emergency medicine (EM) practice, but there can be delayed awareness of this evidence because emergency physicians (EPs) are unlikely to read most of these journals. Also, not all published studies are intended for or ready for clinical practice application. The authors developed "Best Evidence in Emergency Medicine" (BEEM) to ameliorate these problems by searching for, identifying, appraising, and translating potentially practice-changing studies for EPs. An initial step in the BEEM process is the BEEM rater scale, a novel tool for EPs to collectively evaluate the relative clinical relevance of EM-related studies found in more than 120 journals. The BEEM rater process was designed to serve as a clinical relevance filter to identify those studies with the greatest potential to affect EM practice. Therefore, only those studies identified by BEEM raters as having the highest clinical relevance are selected for the subsequent critical appraisal process and, if found methodologically sound, are promoted as the best evidence in EM.ObjectivesThe primary objective was to measure inter-rater reliability (IRR) of the BEEM rater scale. Secondary objectives were to determine the minimum number of EP raters needed for the BEEM rater scale to achieve acceptable reliability and to compare performance of the scale against a previously published evidence rating system, the McMaster Online Rating of Evidence (MORE), in an EP population.MethodsThe authors electronically distributed the title, conclusion, and a PubMed link for 23 recently published studies related to EM to a volunteer group of 134 EPs. The volunteers answered two demographic questions and rated the articles using one of two randomly assigned seven-point Likert scales, the BEEM rater scale (n = 68) or the MORE scale (n = 66), over two separate administrations. The IRR of each scale was measured using generalizability theory.ResultsThe IRR of the BEEM rater scale ranged between 0.90 (95% confidence interval [CI] = 0.86 to 0.93) to 0.92 (95% CI = 0.89 to 0.94) across administrations. Decision studies showed a minimum of 12 raters is required for acceptable reliability of the BEEM rater scale. The IRR of the MORE scale was 0.82 to 0.84.ConclusionsThe BEEM rater scale is a highly reliable, single-question tool for a small number of EPs to collectively rate the relative clinical relevance within the specialty of EM of recently published studies from a variety of medical journals. It compares favorably with the MORE system because it achieves a high IRR despite simply requiring raters to read each article's title and conclusion.

Project description:IntroductionAs frailty among the elderly is receiving increasing attention in emergency departments (EDs) around the world, the use of frailty assessment tools is growing. An often used such tool is the Clinical Frailty Scale (CFS), whose inter-rater reliability has been sparingly investigated in ED settings. No inter-rater reliability study regarding CFS has previously been performed within the Scandinavian ED context. The primary aim of this study was to evaluate the inter-rater reliability of the CFS in a Swedish ED setting.MethodsThis was a prospective observational study conducted at three Swedish EDs. Patients ≥65 years were independently assessed with CFS by their responsible physician, registered nurse, and assistant nurse. Demographic information for each assessor was collected, along with frailty status (frail/not frail) on the basis of clinical judgment. Inter-rater reliability was calculated using intraclass correlation coefficient (ICC), whereas agreement of frailty assessed by CFS (dichotomized between frail/not frail, cutoff at ≥5 points) versus solely by clinical judgment was calculated by using cross-tabulation.ResultsOne-hundred patients were included. We found inter-rater reliability to be moderate to good (ICC 0.78, 95% confidence interval [CI] 0.72-0.84), regardless of whether the care team included an emergency physician (ICC 0.74, 95% CI 0.62-0.83) or an intern/resident from another specialty (ICC 0.83, 95% CI 0.74-0.89). The agreement of clinically judged frailty compared to frailty according to CFS was 84%. In the opposing cases, staff tended to assess patients as frail to a higher extent using clinical judgment than by applying CFS on the same patient.ConclusionsThe CFS appears to have a moderate to good inter-rater reliability when used in a clinical ED setting. When guiding clinical decisions, we advise that the CFS score should be discussed within the team. Further research needs to be performed on the accuracy of clinical judgment to identify frailty in ED patients.

Project description:BackgroundElectronic health records (EHR) contain large volumes of unstructured free-form text notes that richly describe a patient's health and medical comorbidities. It is unclear if perioperative risk stratification can be performed directly from these notes without manual data extraction. We conduct a feasibility study using natural language processing (NLP) to predict the American Society of Anesthesiologists Physical Status Classification (ASA-PS) as a surrogate measure for perioperative risk. We explore prediction performance using four different model types and compare the use of different note sections versus the whole note. We use Shapley values to explain model predictions and analyze disagreement between model and human anesthesiologist predictions.MethodsSingle-center retrospective cohort analysis of EHR notes from patients undergoing procedures with anesthesia care spanning all procedural specialties during a 5 year period who were not assigned ASA VI and also had a preoperative evaluation note filed within 90 days prior to the procedure. NLP models were trained for each combination of 4 models and 8 text snippets from notes. Model performance was compared using area under the receiver operating characteristic curve (AUROC) and area under the precision recall curve (AUPRC). Shapley values were used to explain model predictions. Error analysis and model explanation using Shapley values was conducted for the best performing model.ResultsFinal dataset includes 38,566 patients undergoing 61,503 procedures with anesthesia care. Prevalence of ASA-PS was 8.81% for ASA I, 31.4% for ASA II, 43.25% for ASA III, and 16.54% for ASA IV-V. The best performing models were the BioClinicalBERT model on the truncated note task (macro-average AUROC 0.845) and the fastText model on the full note task (macro-average AUROC 0.865). Shapley values reveal human-interpretable model predictions. Error analysis reveals that some original ASA-PS assignments may be incorrect and the model is making a reasonable prediction in these cases.ConclusionsText classification models can accurately predict a patient's illness severity using only free-form text descriptions of patients without any manual data extraction. They can be an additional patient safety tool in the perioperative setting and reduce manual chart review for medical billing. Shapley feature attributions produce explanations that logically support model predictions and are understandable to clinicians.

Dataset Information

Inter-rater reliability of the American Society of Anesthesiologists physical status rating for emergency gastrointestinal surgery.

Publications

Inter-rater reliability of the American Society of Anesthesiologists physical status rating for emergency gastrointestinal surgery.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets