Dataset Information

The use of global rating scales for OSCEs in veterinary medicine.

ABSTRACT: OSCEs (Objective Structured Clinical Examinations) are widely used in health professions to assess clinical skills competence. Raters use standardized binary checklists (CL) or multi-dimensional global rating scales (GRS) to score candidates performing specific tasks. This study assessed the reliability of CL and GRS scores in the assessment of veterinary students, and is the first study to demonstrate the reliability of GRS within veterinary medical education. Twelve raters from two different schools (6 from University of Calgary [UCVM] and 6 from Royal (Dick) School of Veterinary Studies [R(D)SVS] were asked to score 12 students (6 from each school). All raters assessed all students (video recordings) during 4 OSCE stations (bovine haltering, gowning and gloving, equine bandaging and skin suturing). Raters scored students using a CL, followed by the GRS. Novice raters (6 R(D)SVS) were assessed independently of expert raters (6 UCVM). Generalizability theory (G theory), analysis of variance (ANOVA) and t-tests were used to determine the reliability of rater scores, assess any between school differences (by student, by rater), and determine if there were differences between CL and GRS scores. There was no significant difference in rater performance with use of the CL or the GRS. Scores from the CL were significantly higher than scores from the GRS. The reliability of checklist scores were .42 and .76 for novice and expert raters respectively. The reliability of the global rating scale scores were .7 and .86 for novice and expert raters respectively. A decision study (D-study) showed that once trained using CL, GRS could be utilized to reliably score clinical skills in veterinary medicine with both novice and experienced raters.

SUBMITTER: Read EK

PROVIDER: S-EPMC4379077 | biostudies-literature | 2015

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

The use of global rating scales for OSCEs in veterinary medicine.

Read Emma K EK Bell Catriona C Rhind Susan S Hecker Kent G KG

PloS one 20150330 3

OSCEs (Objective Structured Clinical Examinations) are widely used in health professions to assess clinical skills competence. Raters use standardized binary checklists (CL) or multi-dimensional global rating scales (GRS) to score candidates performing specific tasks. This study assessed the reliability of CL and GRS scores in the assessment of veterinary students, and is the first study to demonstrate the reliability of GRS within veterinary medical education. Twelve raters from two different s ...[more]

PMID: 25822258

Similar Datasets

Project description:BackgroundConsensus-orientated Delphi studies are increasingly used in various areas of medical research using a variety of different rating scales and criteria for reaching consensus. We explored the influence of using three different rating scales and different consensus criteria on the results for reaching consensus and assessed the test-retest reliability of these scales within a study aimed at identification of global treatment goals for total knee arthroplasty (TKA).MethodsWe conducted a two-stage study consisting of two surveys and consecutively included patients scheduled for TKA from five German hospitals. Patients were asked to rate 19 potential treatment goals on different rating scales (three-point, five-point, nine-point). Surveys were conducted within a 2 week period prior to TKA, order of questions (scales and treatment goals) was randomized.ResultsEighty patients (mean age 68?±?10?years; 70% females) completed both surveys. Different rating scales (three-point, five-point and nine-point rating scale) lead to different consensus despite moderate to high correlation between rating scales (r?=?0.65 to 0.74). Final consensus was highly influenced by the choice of rating scale with 14 (three-point), 6 (five-point), 15 (nine-point) out of 19 treatment goals reaching the pre-defined 75% consensus threshold. The number of goals reaching consensus also highly varied between rating scales for other consensus thresholds. Overall, concordance differed between the three-point (percent agreement [p]?=?88.5%, weighted kappa [k]?=?0.63), five-point (p?=?75.3%, k?=?0.47) and nine-point scale (p?=?67.8%, k?=?0.78).ConclusionThis study provides evidence that consensus depends on the rating scale and consensus threshold within one population. The test-retest reliability of the three rating scales investigated differs substantially between individual treatment goals. This variation in reliability can become a potential source of bias in consensus studies. In our setting aimed at capturing patients' treatment goals for TKA, the three-point scale proves to be the most reasonable choice, as its translation into the clinical context is the most straightforward among the scales. Researchers conducting Delphi studies should be aware that final consensus is substantially influenced by the choice of rating scale and consensus criteria.

Project description:Introduction: The Clinical Global Impressions-Improvement (CGI-I) scale is widely used in clinical research to assess symptoms and functioning in the context of treatment. The correlates of the CGI-I with efficacy scales for adolescent major depressive disorder are poorly understood. This study focused on benchmarking CGI-I scores with changes in the Children's Depression Rating Scale-Revised (CDRS-R) and the Quick Inventory of Depressive Symptomatology-Adolescent (17-item) Self-Report (QIDS-A17-SR). Methods: We examined three datasets with the clinician-rated CDRS-R to ascertain equivalent percent changes in total scores and CGI-I ratings. Exploratory analyses examined corresponding percentage changes in the QIDS-A17-SR and the CGI-I ratings. The CGI-I was the reference scale for nonparametric equipercentile linking with the Equate package in R. Results: CGI-I scores of 1 mapped to ≥78%-95% change in CDRS-R scores at 4-6 weeks across three datasets. CGI-I scores of 2 mapped to 56%-94% change in CDRS-R scores at 4-6 weeks across three studies. CGI-I scores of 3 mapped to 30%-68% changes in CDRS-R scores at 4-6 weeks across three studies. CGI-I scores of 4 mapped to a range of 29%-44% at 4-6 weeks across three studies. There was no significant difference (p ≥ 0.6) between treatment groups in both the Treatment of Adolescents with Depression and Treatment of Resistant Depression in Adolescents studies, for each CGI-I score ( = 1, or = 2 or = 3, or ≥4), associated mapping of total depression severity score, or associated percent change from baseline for corresponding follow-up visits. There was no significant sex difference (p > 0.2) in CGI-I linkages to CDRS-R total or percentage changes. Conclusions: These findings establish clear relationships among CGI-I scores and the CDRS-R and the QIDS-A17-SR. These benchmarks have utility for clinical trial study design, inter-rater reliability training, and clinical implementation.

Project description:Rating scales are popular methods for generating quantitative data directly by persons rather than automated technologies. But scholars increasingly challenge their foundations. This article contributes epistemological and methodological analyses of the processes involved in person-generated quantification. They are crucial for measurement because data analyses can reveal information about study phenomena only if relevant properties were encoded systematically in the data. The Transdisciplinary Philosophy-of-Science Paradigm for Research on Individuals (TPS-Paradigm) is applied to explore psychological and social-science concepts of measurement and quantification, including representational measurement theory, psychometric theories and their precursors in psychophysics. These are compared to theories from metrology specifying object-dependence of measurement processes and subject-independence of outcomes as key criteria, which allow tracing data to the instances measured and the ways they were quantified. Separate histories notwithstanding, the article's basic premise is that general principles of scientific measurement and quantification should apply to all sciences. It elaborates principles by which these metrological criteria can be implemented also in psychology and social sciences, while considering their research objects' peculiarities. Application of these principles is illustrated by quantifications of individual-specific behaviors ('personality'). The demands rating methods impose on data-generating persons are deconstructed and compared with the demands involved in other quantitative methods (e.g., ethological observations). These analyses highlight problematic requirements for raters. Rating methods sufficiently specify neither the empirical study phenomena nor the symbolic systems used as data nor rules of assignment between them. Instead, pronounced individual differences in raters' interpretation and use of items and scales indicate considerable subjectivity in data generation. Together with recoding scale categories into numbers, this introduces a twofold break in the traceability of rating data, compromising interpretability of findings. These insights question common reliability and validity concepts for ratings and provide novel explanations for replicability problems. Specifically, rating methods standardize only data formats but not the actual data generation. Measurement requires data generation processes to be adapted to the study phenomena's properties and the measurement-executing persons' abilities and interpretations, rather than to numerical outcome formats facilitating statistical analyses. Researchers must finally investigate how people actually generate ratings to specify the representational systems underlying rating data.

Project description:Researchers are finding limitations of currently available disease-focused questionnaire tools for outcome studies in complementary and alternative medicine/integrative medicine (CAM/IM).Three substudies investigated the new one-item visual analogue Arizona Integrative Outcomes Scale (AIOS), which assesses self-rated global sense of spiritual, social, mental, emotional, and physical well-being over the past 24 hours and the past month. The first study tested the scale's ability to discriminate unhealthy individuals (n = 50) from healthy individuals (n = 50) in a rehabilitation outpatient clinic sample. The second study examined the concurrent validity of the AIOS by comparing ratings of global well-being to degree of psychological distress as measured by the Brief Symptom Inventory (BSI) in undergraduate college students (N = 458). The third study evaluated the relationships between the AIOS and positively- and negatively-valenced tools (Positive and Negative Affect Scale and the Positive States of Mind Scale) in a different sample of undergraduate students (N = 62).Substudy (i) Rehabilitation patients scored significantly lower than the healthy controls on both forms of the AIOS and a current global health rating. The AIOS 24-hours correlated moderately and significantly with global health (patients r = 0.50; controls r = 0.45). AIOS 1-month correlations with global health were stronger within the controls (patients r = 0.36; controls r = 0.50). Controls (r = 0.64) had a higher correlation between the AIOS 24-hour and 1-month forms than did the patients (r = 0.33), which is consistent with the presumptive improvement in the patients' condition over the previous 30 days in rehabilitation. Substudy (ii) In undergraduate students, AIOS scores were inversely related to distress ratings, as measured by the global severity index on the BSI (rAIOS24h = -0.42, rAIOS1month = -0.40). Substudy (iii) AIOS scores were significantly correlated with positive affect (rAIOS24h = 0.56, rAIOS1month = 0.57) and positive states of mind (rAIOS24h = 0.42, rAIOS1month = 0.45), and inversely correlated with negative affect (rAIOS24h = -0.41, rAIOS1month = -0.59).The AIOS is able to distinguish relatively sicker from relatively healthier individuals; and correlates in expected directions with a measure of distress and indicators of positive and negative affect and positive states of mind. The AIOS offers a tool for CAM/IM research that extends beyond a disease emphasis.

Dataset Information

The use of global rating scales for OSCEs in veterinary medicine.

Publications

The use of global rating scales for OSCEs in veterinary medicine.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets