An interrater reliability study on the Gothenburg obstetric triage system- a new obstetric triage system.
Ontology highlight
ABSTRACT: Background
Triage, identifying patients with critical and time-sensitive disorders, is an integrated process in general emergency medicine. Obstetric triage is more specialised, requiring assessment of both woman, fetus and labour status. Failure to identify severely ill obstetric patients has repeatedly led to maternal morbidity and mortality. Reliable triage systems, adapted to obstetric patients as well as local conditions, are thus essential. The study aims to assess the interrater reliability (IRR) of the Gothenburg Obstetric Triage System (GOTS).Methods
Midwives (n = 6) and registered nurses with no experience in managing obstetric patients (n = 7), assessed 30 paper cases based on actual real-life cases, using the GOTS. Furthermore, a reference group consisting of two midwives and two obstetricians, with extensive experience in obstetric care, determined the correct triage level in order to enable analysis of over- and undertriage. IRR was assessed, both with percentage of absolute agreement and with intra-class correlation coefficients (ICC) with 95% confidence intervals (CI).Results
A total of 388 assessments were performed, comprising all five levels of acuity in the GOTS. Absolute agreement was found in 69.6% of the assessments. The overall IRR was good, with a Kappa value of 0.78 (0.69-0.87, 95% CI) for final triage level. Comparison with reference group assessments established that over- and undertriage had occurred in 9% and 21% of the cases, respectively. The main reasons for undertriage were "not acknowledging abnormal vital sign parameters" and "limitations in study design".Conclusion
The GOTS is a reliable tool for triaging obstetric patients. It enables a standardized triage process unrelated to the assessors' level of experience in assessing and managing obstetric patients and is applicable for triaging obstetric patients presenting for emergency care at obstetric or emergency units.
SUBMITTER: Lindroos L
PROVIDER: S-EPMC8487102 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Background Failure to identify severely ill obstetric patients seeking acute care, and hence delaying treatment, can lead to maternal morbidity and mortality. Triage is the prioritization of patients seeking emergency care, based on clinical decision-making tools assessing medical urgency. While triage has been applied in general emergency medicine for 30 years, there are only a few obstetric triage systems (OTS) and obstetric triage has hitherto been unknown in Sweden. Obstetric triage is more complex than general triage since both mother and fetus require assessment, and pregnancy-related physiological changes must be taken into account. This paper aims to describe the development and an initial evaluation of the first OTS in Sweden. Methods A multidisciplinary team surveyed reasons to seek acute obstetric care and the current patient flow at the largest obstetric unit in Scandinavia, Sahlgrenska University Hospital, Gothenburg, Sweden, with about 10,000 deliveries/year. A semi-structured literature review on obstetric triage was undertaken. Based on the survey and the literature review the first Swedish OTS was developed and implemented. Patient satisfaction was followed by electronical questionnaires. Initial validity evaluation was performed, defined by the system’s ability to identify patients with need for hospital admission, stratified by acuity level. Results The Gothenburg Obstetrical Triage System (GOTS) addresses the patient to one of five acuity levels based on both vital signs and 14 chief complaint algorithms. It entails recommendations for initial procedures of care as well as an acuity form for documentation. Initial evaluation of the system indicates good correlation between need for admission and acuity level. The implementation has provided the staff with an improved medical overview of the patients and patient flow and enabled the unit to monitor emergency care in a structured way. Implementation came along with increased patient and staff satisfaction. Conclusion The GOTS is the first OTS developed in and for Sweden and implementation has improved management of obstetric patients seeking acute care. Patients are now prioritized according to level of acuity and the time to assessment and treatment of severely ill patients can be structurally evaluated. Both patients and staff express improved satisfaction with obstetric triage. Supplementary Information The online version contains supplementary material available at 10.1186/s12913-021-07210-9.
| S-EPMC8572438 | biostudies-literature
Project description:BackgroundInterrater reliability, aka intercoder reliability, is defined as true agreement between raters, aka coders, without chance agreement. It is used across many disciplines including medical and health research to measure the quality of ratings, coding, diagnoses, or other observations and judgements. While numerous indices of interrater reliability are available, experts disagree on which ones are legitimate or more appropriate. Almost all agree that percent agreement (ao), the oldest and the simplest index, is also the most flawed because it fails to estimate and remove chance agreement, which is produced by raters' random rating. The experts, however, disagree on which chance estimators are legitimate or better. The experts also disagree on which of the three factors, rating category, distribution skew, or task difficulty, an index should rely on to estimate chance agreement, or which factors the known indices in fact rely on. The most popular chance-adjusted indices, according to a functionalist view of mathematical statistics, assume that all raters conduct intentional and maximum random rating while typical raters conduct involuntary and reluctant random rating. The mismatches between the assumed and the actual rater behaviors cause the indices to rely on mistaken factors to estimate chance agreement, leading to the numerous paradoxes, abnormalities, and other misbehaviors of the indices identified by prior studies.MethodsWe conducted a 4 × 8 × 3 between-subject controlled experiment with 4 subjects per cell. Each subject was a rating session with 100 pairs of rating by two raters, totaling 384 rating sessions as the experimental subjects. The experiment tested seven best-known indices of interrater reliability against the observed reliabilities and chance agreements. Impacts of the three factors, i.e., rating category, distribution skew, and task difficulty, on the indices were tested.ResultsThe most criticized index, percent agreement (ao), showed as the most accurate predictor of reliability, reporting directional r2 = .84. It was also the third best approximator, overestimating observed reliability by 13 percentage points on average. The three most acclaimed and most popular indices, Scott's π, Cohen's κ and Krippendorff's α, underperformed all other indices, reporting directional r2 = .312 and underestimated reliability by 31.4 ~ 31.8 points. The newest index, Gwet's AC1, emerged as the second-best predictor and the most accurate approximator. Bennett et al's S ranked behind AC1, and Perreault and Leigh's Ir ranked the fourth both for prediction and approximation. The reliance on category and skew and failure to rely on difficulty explain why the six chance-adjusted indices often underperformed ao, which they were created to outperform. The evidence corroborated the notion that the chance-adjusted indices assume intentional and maximum random rating while the raters instead exhibited involuntary and reluctant random rating.ConclusionThe authors call for more empirical studies and especially more controlled experiments to falsify or qualify this study. If the main findings are replicated and the underlying theories supported, new thinking and new indices may be needed. Index designers may need to refrain from assuming intentional and maximum random rating, and instead assume involuntary and reluctant random rating. Accordingly, the new indices may need to rely on task difficulty, rather than distribution skew or rating category, to estimate chance agreement.
| S-EPMC9426226 | biostudies-literature
Project description:ObjectiveTo evaluate the interrater reliability of the new International Behavioural Variant FTD Criteria Consortium (FTDC) criteria for behavioral variant frontotemporal dementia (bvFTD).MethodsTwenty standardized clinical case modules were developed for patients with a range of neurodegenerative diagnoses, including bvFTD, primary progressive aphasia (nonfluent, semantic, and logopenic variant), Alzheimer disease, and Lewy body dementia. Eighteen blinded raters reviewed the modules and 1) rated the presence or absence of core diagnostic features for the FTDC criteria, and 2) provided an overall diagnostic rating. Interrater reliability was determined by ? statistics for multiple raters with categorical ratings.ResultsThe mean ? value for diagnostic agreement was 0.81 for possible bvFTD and 0.82 for probable bvFTD ("almost perfect agreement"). Interrater reliability for 4 of the 6 core features had "substantial" agreement (behavioral disinhibition, perseverative/compulsive, sympathy/empathy, hyperorality; ? = 0.61-0.80), whereas 2 had "moderate" agreement (apathy/inertia, neuropsychological; ? = 0.41-0.6). Clinician years of experience did not significantly influence rater accuracy.ConclusionsThe FTDC criteria show promise for improving the diagnostic accuracy and reliability of clinicians and researchers. As disease-altering therapies are developed, accurate differential diagnosis between bvFTD and other neurodegenerative diseases will become increasingly important.
| S-EPMC3716343 | biostudies-literature
Project description:Aim:The purpose of this study was to explain the challenges of the obstetric triage structure. Design:The present qualitative research was conducted with directed content analysis approach on 21 members of the triage team and the key informant using purposeful sampling in 2018. Methods:The method of data collection was semi-structured interviews. Then, the accuracy and rigour of the qualitative data were examined. Results:In this study, the most important challenges in the structure of obstetric triage were identified as pattern and standard, equipment, physical space, human resource and triage procedure and process. Correction and revision in the obstetric triage structure is important to provide high-quality services. Therefore, the quality of the structure can be developed and maintained accounting for the corresponding challenges.
| S-EPMC7308674 | biostudies-literature
Project description:BACKGROUND:Medical coding is essential for standardized communication and integration of clinical data. The Unified Medical Language System by the National Library of Medicine is the largest clinical terminology system for medical coders and Natural Language Processing tools. However, the abundance of ambiguous codes leads to low rates of uniform coding among different coders. OBJECTIVE:The objective of our study was to measure uniform coding among different medical experts in terms of interrater reliability and analyze the effect on interrater reliability using an expert- and Web-based code suggestion system. METHODS:We conducted a quasi-experimental study in which 6 medical experts coded 602 medical items from structured quality assurance forms or free-text eligibility criteria of 20 different clinical trials. The medical item content was selected on the basis of mortality-leading diseases according to World Health Organization data. The intervention comprised using a semiautomatic code suggestion tool that is linked to a European information infrastructure providing a large medical text corpus of >300,000 medical form items with expert-assigned semantic codes. Krippendorff alpha (Kalpha) with bootstrap analysis was used for the interrater reliability analysis, and coding times were measured before and after the intervention. RESULTS:The intervention improved interrater reliability in structured quality assurance form items (from Kalpha=0.50, 95% CI 0.43-0.57 to Kalpha=0.62 95% CI 0.55-0.69) and free-text eligibility criteria (from Kalpha=0.19, 95% CI 0.14-0.24 to Kalpha=0.43, 95% CI 0.37-0.50) while preserving or slightly reducing the mean coding time per item for all 6 coders. Regardless of the intervention, precoordination and structured items were associated with significantly high interrater reliability, but the proportion of items that were precoordinated significantly increased after intervention (eligibility criteria: OR 4.92, 95% CI 2.78-8.72; quality assurance: OR 1.96, 95% CI 1.19-3.25). CONCLUSIONS:The Web-based code suggestion mechanism improved interrater reliability toward moderate or even substantial intercoder agreement. Precoordination and the use of structured versus free-text data elements are key drivers of higher interrater reliability.
| S-EPMC6231825 | biostudies-other
Project description:BackgroundThere is a huge number of health-related apps available, and the numbers are growing fast. However, many of them have been developed without any kind of quality control. In an attempt to contribute to the development of high-quality apps and enable existing apps to be assessed, several guides have been developed.ObjectiveThe main aim of this study was to study the interrater reliability of a new guide - the Mobile App Development and Assessment Guide (MAG) - and compare it with one of the most used guides in the field, the Mobile App Rating Scale (MARS). Moreover, we also focused on whether the interrater reliability of the measures is consistent across multiple types of apps and stakeholders.MethodsIn order to study the interrater reliability of the MAG and MARS, we evaluated the 4 most downloaded health apps for chronic health conditions in the medical category of IOS and Android devices (ie, App Store and Google Play). A group of 8 reviewers, representative of individuals that would be most knowledgeable and interested in the use and development of health-related apps and including different types of stakeholders such as clinical researchers, engineers, health care professionals, and end users as potential patients, independently evaluated the quality of the apps using the MAG and MARS. We calculated the Krippendorff alpha for every category in the 2 guides, for each type of reviewer and every app, separately and combined, to study the interrater reliability.ResultsOnly a few categories of the MAG and MARS demonstrated a high interrater reliability. Although the MAG was found to be superior, there was considerable variation in the scores between the different types of reviewers. The categories with the highest interrater reliability in MAG were "Security" (α=0.78) and "Privacy" (α=0.73). In addition, 2 other categories, "Usability" and "Safety," were very close to compliance (health care professionals: α=0.62 and 0.61, respectively). The total interrater reliability of the MAG (ie, for all categories) was 0.45, whereas the total interrater reliability of the MARS was 0.29.ConclusionsThis study shows that some categories of MAG have significant interrater reliability. Importantly, the data show that the MAG scores are better than the ones provided by the MARS, which is the most commonly used guide in the area. However, there is great variability in the responses, which seems to be associated with subjective interpretation by the reviewers.
| S-EPMC8094021 | biostudies-literature
Project description:PurposeFor optimal management of ductal carcinoma in situ (DCIS), reproducible histopathological assessment is essential to distinguish low-risk from high-risk DCIS. Therefore, we analyzed interrater reliability of histopathological DCIS features and assessed their associations with subsequent ipsilateral invasive breast cancer (iIBC) risk.MethodsUsing a case-cohort design, reliability was assessed in a population-based, nationwide cohort of 2767 women with screen-detected DCIS diagnosed between 1993 and 2004, treated by breast-conserving surgery with/without radiotherapy (BCS ± RT) using Krippendorff's alpha (KA) and Gwet's AC2 (GAC2). Thirty-eight raters scored histopathological DCIS features including grade (2-tiered and 3-tiered), growth pattern, mitotic activity, periductal fibrosis, and lymphocytic infiltrate in 342 women. Using majority opinion-based scores for each feature, their association with subsequent iIBC risk was assessed using Cox regression.ResultsInterrater reliability of grade using various classifications was fair to moderate, and only substantial for grade 1 versus 2 + 3 when using GAC2 (0.78). Reliability for growth pattern (KA 0.44, GAC2 0.78), calcifications (KA 0.49, GAC2 0.70) and necrosis (KA 0.47, GAC2 0.70) was moderate using KA and substantial using GAC2; for (type of) periductal fibrosis and lymphocytic infiltrate fair to moderate estimates were found and for mitotic activity reliability was substantial using GAC2 (0.70). Only in patients treated with BCS-RT, high mitotic activity was associated with a higher iIBC risk in univariable analysis (Hazard Ratio (HR) 2.53, 95% Confidence Interval (95% CI) 1.05-6.11); grade 3 versus 1 + 2 (HR 2.64, 95% CI 1.35-5.14) and a cribriform/solid versus flat epithelial atypia/clinging/(micro)papillary growth pattern (HR 3.70, 95% CI 1.34-10.23) were independently associated with a higher iIBC risk.ConclusionsUsing majority opinion-based scores, DCIS grade, growth pattern, and mitotic activity are associated with iIBC risk in patients treated with BCS-RT, but interrater variability is substantial. Semi-quantitative grading, incorporating and separately evaluating nuclear pleomorphism, growth pattern, and mitotic activity, may improve the reliability and prognostic value of these features.
| S-EPMC7497690 | biostudies-literature
Project description:This study assesses the interrater reliability of Ayurvedic pulse (nadi), tongue (jivha), and body constitution (prakriti) assessments. Fifteen registered Ayurvedic doctors with 3-15 years of experience independently examined twenty healthy subjects. Subjects completed self-assessment questionnaires and software analyses for prakriti assessment. Weighted kappa statistics for all 105 pairs of doctors were computed for the pulse, tongue, and prakriti data sets. According to the Landis-Koch scale, the pairwise kappas ranged from poor to slight, slight to fair, and fair to moderate for pulse, tongue, and prakriti assessments, respectively. The average pairwise kappa for pulse, tongue, and prakriti was 0.07, 0.17, and 0.28, respectively. For each data set and pair of doctors, the null hypothesis of random rating was rejected for just twelve pairs of doctors for prakriti, one pair of doctors for pulse examination, and no pairs of doctors for tongue assessment. Thus, the results demonstrate a low level of reliability for all types of assessment made by doctors. There was significant evidence against random rating by software and questionnaire use and by the diagnosis preferred by the majority of doctors. Prakriti assessment appears reliable when questionnaire and software assessment are used, while other diagnostic methods have room for improvement.
| S-EPMC3803118 | biostudies-literature
Project description:IntroductionClinical clerkships that rely on community preceptors as faculty and assessors are challenged by the variability in their assessments of student performance. This workshop was designed to improve the interrater reliability of preceptors' assessment of student performance.MethodsThis workshop uses a series of videos showing standardized student interactions with standardized patients and a preceptor. The videos feature a borderline satisfactory student-patient encounter and a superior student encounter. For each case, there is a video of the student providing an oral presentation of the encounter to a preceptor. The concept of a plus/delta card is also presented. Participants are asked to note positive elements of the interactions and elements needing improvement on the plus/delta card while watching the videos. Next, participants share their assessment of the students with the group using an audience response system. Group discussion allows for consensus and leads to decreased variation in preceptor assessments of each student's performance. This workshop has been delivered to preceptors in our institution as a 1-hour workshop. Additionally, a version has been presented twice at national medical teaching conferences as a faculty development workshop, instructing faculty how to facilitate the workshop.ResultsParticipant evaluations from the workshop show an increase in preceptors' awareness of specific student behaviors to observe as well as increased confidence with assessing more consistently across various student performances.DiscussionUsing videos with standardized patients, students, and preceptors demonstrating various qualities of performance is effective in improving precision in preceptors' assessment of students in clinical settings.
| S-EPMC6338197 | biostudies-literature
Project description:ObjectiveIdentification of EEG waveforms is critical for diagnosing Lennox-Gastaut Syndrome (LGS) but is complicated by the progressive nature of the disease. Here, we assess the interrater reliability (IRR) among pediatric epileptologists for classifying EEG waveforms associated with LGS.MethodsA novel automated algorithm was used to objectively identify epochs of EEG with transient high power, which were termed events of interest (EOIs). The algorithm was applied to EEG from 20 LGS subjects and 20 healthy controls during NREM sleep, and 1350 EOIs were identified. Three raters independently reviewed the EOIs within isolated 15-second EEG segments in a randomized, blinded fashion. For each EOI, the raters assigned a waveform label (spike and slow wave, generalized paroxysmal fast activity, seizure, spindle, vertex, muscle, artifact, nothing, or other) and indicated the perceived subject type (LGS or control).ResultsLabeling of subject type had 85% accuracy across all EOIs and an IRR of κ =0.790, suggesting that brief segments of EEG containing high-power waveforms can be reliably classified as pathological or normal. Waveform labels were less consistent, with κ =0.558, and the results were highly variable for different categories of waveforms. Label mismatches typically occurred when one reviewer selected "nothing," suggesting that reviewers had different thresholds for applying named labels.SignificanceClassification of EEG waveforms associated with LGS has weak IRR, due in part to varying thresholds applied during visual review. Computational methods to objectively define EEG biomarkers of LGS may improve IRR and aid clinical decision-making.
| S-EPMC10839292 | biostudies-literature