Examining the Impacts of Rater Effects in Performance Assessments.
Ontology highlight
ABSTRACT:
SUBMITTER: Wind SA
PROVIDER: S-EPMC6376535 | biostudies-literature | 2019 Mar
REPOSITORIES: biostudies-literature
Publications
Examining the Impacts of Rater Effects in Performance Assessments.
Applied psychological measurement 20180805 2
Rater effects such as severity, centrality, and misfit are recurrent concerns in performance assessments. Despite their persistence in operational assessment settings and frequent discussion in research, researchers have not fully explored the impacts of rater effects as they relate to estimates of student achievement. The purpose of this study is to explore the impacts of rater severity, centrality, and misfit on student achievement estimates and on classification decisions. The results suggest ...[more]
Similar Datasets
Project description:BackgroundEntrustable Professional Activities (EPAs) assessments measure learners' competence with an entrustment or supervisory scale. Designed for workplace-based assessment EPA assessments have also been proposed for undergraduate medical education (UME), where assessments frequently occur outside the workplace and may be less intuitive, raising validity concerns. This study explored how assessors make entrustment determinations in UME, with additional specific comparison based on familiarity with prior performance in the context of longitudinal student-assessor relationships.MethodsA qualitative approach using think-alouds was employed. Assessors assessed two students (familiar and unfamiliar) completing a history and physical examination using a supervisory scale and then thought-aloud after each assessment. We conducted a thematic analysis of assessors' response processes and compared them based on their familiarity with a student.ResultsFour themes and fifteen subthemes were identified. The most prevalent theme related to "student performance." The other three themes included "frame of reference," "assessor uncertainty," and "the patient." "Previous student performance" and "affective reactions" were subthemes more likely to inform scoring when faculty were familiar with a student, while unfamiliar faculty were more likely to reference "self" and "lack confidence in their ability to assess."ConclusionsStudent performance appears to be assessors' main consideration for all students, providing some validity evidence for the response process in EPA assessments. Several problematic themes could be addressed with faculty development while others appear to be inherent to entrustment and may be more challenging to mitigate. Differences based on assessor familiarity with student merits further research on how trust develops over time.
| S-EPMC9077896 | biostudies-literature
Project description:IntroductionReliable team assessment has become a priority because of growing emphasis on interprofessional education and team-based care. Objective rating scales are needed to evaluate interprofessional student teams and individuals and provide real-time feedback.MethodsIn response to a need for behavioral rating scales, we modified the McMaster-Ottawa Scale from a 9-point to a 3-point scale and added descriptive behavioral anchors to define three levels of competency (i.e., below, at, and above expected). This modification is intended to provide consistent rating of individuals and teams in patient settings. We then developed a demonstration video using actors representing four professions to demonstrate the three levels of performance within the team. Our faculty rater tool, consisting of the modified scale and video, is designed to provide standardized ratings in interprofessional educational settings that involve patient care.ResultsWe conducted training sessions with 40 faculty members from seven professions (medicine, dentistry, occupational therapy, nursing, pharmacy, physician assistant, and psychology) over a 2-year period. Immediately after each training session, two trained faculty observers rated interprofessional student teams as they conducted history and assessments on standardized patients. Observer scores were compared with one another and with standard expert ratings of the same teams. Trained observer ratings were consistent across the pairs. The observer training can be conducted within 60-90 minutes with the tool.DiscussionResults of our implementation of the faculty rater tool confirm that the modified McMaster-Ottawa Scale is feasible to administer in clinical settings and that the demonstration video can be easily adopted for standardizing observer ratings.
| S-EPMC6464413 | biostudies-literature
Project description:Performance appraisal is a complex process by which an organization can determine the extent to which employees are performing their work effectively. However, this appraisal may not be accurate if there is no reduction in the impact of problems caused by possibly subjective rater judgements. The main objective of this work is to check the effectiveness-separately and jointly-of the following four training programmes in the extant literature aimed at improving the accuracy of performance assessment: 1) Performance Dimension Training, 2) Frame-of-Reference, 3) Rater Error Training, and 4) Behavioural Observation Training. Based on these training strategies, three programmes were designed and applied separately. A fourth programme was a combination of the other three. We analyzed two studies using different samples (85 students and 42 employees) for the existence of differences in the levels of knowledge of performance and its dimensions, rater errors, observational accuracy, and accuracy of task and citizenship performance appraisal, according to the type of training raters receive. First, the main results show that training based on performance dimensions and the creation of a common framework, in addition to the training that includes the four programmes (Training_4_programmes), increases the level of knowledge of performance and its dimensions. Second, groups that receive training in rater error score higher in knowledge of biases than the other groups, whether or not they have received training. Third, participants' observational accuracy improves with each new moment measure (post-training and follow-up), though not because of the type of training received. Fourth, participants who receive training through the programme that combine the other four gave a task performance appraisal that was closer to the one undertaken by the judges-experts than the other groups. And finally, students' citizenship performance appraisal does not vary according to type of training or to different moment measures, whereas the group of employees who received all four types of training gave a more accurate citizenship performance assessment.
| S-EPMC6752840 | biostudies-literature
Project description:Background: Despite a growing literature and commercial market, the effectiveness of transcranial direct current stimulation (tDCS) remains questionable. Notably, studies rarely examine factors such as expectations of outcomes, which may influence tDCS response through placebo-like effects. Here we sought to determine whether expectations could influence the behavioral outcomes of a tDCS intervention. Methods: Through an initial study and self-replication, we recruited 121 naïve young adults 18-34 years of age (M = 21.14, SD = 3.58; 88 women). We evaluated expectations of tDCS and of motor and cognitive performance at three times: (i) at baseline; (ii) after being primed to have High or Low expectations of outcomes; and (iii) after a single session of sham-controlled anodal tDCS over the left or right motor cortex. Before and after stimulation, participants performed the Grooved Pegboard Test and a choice reaction time task as measures of motor dexterity, response time, and response inhibition. Results: Repeated measures ANOVA revealed that participants had varying, largely uncertain, expectations regarding tDCS effectiveness at baseline. Expectation ratings significantly increased or decreased in response to High or Low priming, respectively, and decreased following the intervention. Response times and accuracy on motor and cognitive measures were largely unaffected by expectation or stimulation conditions. Overall, our analysis revealed no effect attributable to baseline expectations, belief of group assignment, or experimental condition on behavioral outcomes. Subjective experience did not differ based on expectation or stimulation condition. Conclusions: Our results suggest no clear effects of tDCS or of expectations on our performance measures, highlighting the need for further investigations of such stimulation methods.
| S-EPMC6330301 | biostudies-literature
Project description:Non-additive (dominance and epistasis) effects have remarkable influences on hybrid performance, e.g., via heterosis. Nevertheless, only additive effects are often considered in genomic predictions (GP). In this study, we demonstrated the importance of dominance effects in the prediction of hybrid performance in bioenergy sorghum [Sorghum bicolor (L.) Moench]. The dataset contained more than 400 hybrids between 200 inbred lines and two testers. The hybrids exhibited considerable heterosis in culm length and fresh weight, and the degree of heterosis was consistent with the genetic distance from the corresponding tester. The degree of heterosis was further different among subpopulations. Conversely, Brix exhibited limited heterosis. Regarding GP, we examined three statistical models and four training dataset types. In most of the dataset types, genomic best linear unbiased prediction (GBLUP) with additive effects had lower prediction accuracy than GBLUP with additive and dominance effects (GBLUP-AD) and Gaussian kernel regression (GK). The superiority of GBLUP-AD and GK depended on the level of dominance variance, which was high for culm length and fresh weight, and low for Brix. Considering subpopulations, the influence of dominance was more complex. Our findings highlight the importance of considering dominance effects in GP models for sorghum hybrid breeding.
| S-EPMC7878944 | biostudies-literature
Project description:Ocean acidification is increasingly recognized as a component of global change that could have a wide range of impacts on marine organisms, the ecosystems they live in, and the goods and services they provide humankind. Assessment of these potential socio-economic impacts requires integrated efforts between biologists, chemists, oceanographers, economists and social scientists. But because ocean acidification is a new research area, significant knowledge gaps are preventing economists from estimating its welfare impacts. For instance, economic data on the impact of ocean acidification on significant markets such as fisheries, aquaculture and tourism are very limited (if not non-existent), and non-market valuation studies on this topic are not yet available. Our paper summarizes the current understanding of future OA impacts and sets out what further information is required for economists to assess socio-economic impacts of ocean acidification. Our aim is to provide clear directions for multidisciplinary collaborative research.
| S-EPMC3873077 | biostudies-literature
Project description:PurposeCerebral Performance Category (CPC) scores are often an outcome measure for post-arrest neurologic function, collected worldwide to compare performance, evaluate therapies, and formulate recommendations. At most institutions, no formal training is offered in their determination, potentially leading to misclassification.Materials and methodsWe identified 171 patients at 2 hospitals between 5/10/2005 and 8/31/2012 with two CPC scores at hospital discharge recorded independently - in an in-house quality improvement database and as part of a national registry. Scores were abstracted retrospectively from the same electronic medical record by two separate non-clinical researchers. These scores were compared to assess inter-rater reliability and stratified based on whether the score was concordant or discordant among reviewers to determine factors related to discordance.ResultsThirty-nine CPC scores (22.8%) were discordant (kappa: 0.66), indicating substantial agreement. When dichotomized into "favorable" neurologic outcome (CPC 1-2)/"unfavorable" neurologic outcome (CPC 3-5), 20 (11.7%) scores were discordant (kappa: 0.70), also indicating substantial agreement. Patients discharged home (as opposed to nursing/other care facility) and patients with suspected cardiac etiology of arrest were statistically more likely to have concordant scores. For the quality improvement database, patients with discordant scores had a statistically higher median CPC score than those with concordant scores. The registry had statistically lower median CPC score (CPC 1) than the quality improvement database (CPC 2); p<0.01 for statistical significance.ConclusionsCPC scores have substantial inter-rater reliability, which is reduced in patients who have worse outcomes, have a non-cardiac etiology of arrest, and are discharged to a location other than home.
| S-EPMC5990009 | biostudies-literature
Project description:Quality control is a critical step in the processing and analysis of functional magnetic resonance imaging data. Its purpose is to remove problematic data that could otherwise lead to downstream errors in the analysis and reporting of results. The manual inspection of data can be a laborious and error-prone process that is susceptible to human error. The development of automated tools aims to mitigate these issues. One such tool is pyfMRIqc, which we previously developed as a user-friendly method for assessing data quality. Yet, these methods still generate output that requires subjective interpretations about whether the quality of a given dataset meets an acceptable standard for further analysis. Here we present a quality control protocol using pyfMRIqc and assess the inter-rater reliability of four independent raters using this protocol for data from the fMRI Open QC project (https://osf.io/qaesm/). Data were classified by raters as either "include," "uncertain," or "exclude." There was moderate to substantial agreement between raters for "include" and "exclude," but little to no agreement for "uncertain." In most cases only a single rater used the "uncertain" classification for a given participant's data, with the remaining raters showing agreement for "include"/"exclude" decisions in all but one case. We suggest several approaches to increase rater agreement and reduce disagreement for "uncertain" cases, aiding classification consistency.
| S-EPMC9936142 | biostudies-literature
Project description:BackgroundThe thoracic spine (TS) has been neglected in the study of the spine despite its essential role in the stability and posture of the entire spinal complex. Therefore, there is an inevitable need to investigate the reproducibility of different thoracic spinal posture measures used in subjects with TS pain.MethodsThirty-two subjects (16 females and 16 males, mean age 39 years) were evaluated by two physiotherapists on the same day to gauge inter-rater reliability and on two consecutive days to gauge intra-rater reliability. TS posture was assessed by observation, and thoracic spine mobility was measured by manual assessment of segmental flexion and extension mobility in a seated position. Additionally, posterior-to-anterior accessory mobility in a prone position was assessed manually. Moreover, cervicothoracic flexion in a seated position, thoracic posture, and thoracic flexion and extension mobility in a standing position were assessed with a tape measure, and flexion and extension mobility in a seated position and TS posture in seated and standing positions were measured with an inclinometer. The intraclass correlation coefficient (ICC), standard error of measurement (SEM), mean difference (MD), Bland-Altman (B&A) plot features and coefficient of repeatability (CR) were calculated.ResultsThe mean and standard deviation (SD) of the duration of TS pain was 22 (SD 45) months, with the intensity of pain being rated at 27 (SD 21) mm on a visual analogue scale (VAS). Intra-rater reliability was very strong (ICC ≥ 0.80) for the evaluation of seated and standing upper TS posture, standing whole TS posture and seated lower TS posture with an inclinometer. Moreover, TS posture evaluation with a measuring tape, posture inspection in a seated position, and manual assessment of segmental extension were found to have very strong intra-rater reliability. Inter-rater reliability was very strong for inclinometer measurements of standing and seated upper TS posture as well as standing whole TS posture.ConclusionIntra-rater reliability was higher than inter-rater reliability in most of the evaluated measurements. Overall, posture measurements with an inclinometer were more reliable than mobility measurements with the same instrument. The manual assessments can be used reliably when same evaluator performs the examination.Trial registrationClinical Trials, NCT01884818. Registered 24 June 2013, https://clinicaltrials.gov/ct2/show/NCT01884818?cond=thoracic+spine&cntry=FI&rank=1.
| S-EPMC7418198 | biostudies-literature
Project description:When constructed response items are administered repeatedly, "trend scoring" can be used to test for rater drift. In trend scoring, raters rescore responses from the previous administration. Two simulation studies evaluated the utility of Stuart's Q measure of marginal homogeneity as a way of evaluating rater drift when monitoring trend scoring. In the first study, data were generated based on trend scoring tables obtained from an operational assessment. The second study tightly controlled table margins to disentangle certain features present in the empirical data. In addition to Q, the paired t test was included as a comparison, because of its widespread use in monitoring trend scoring. Sample size, number of score categories, interrater agreement, and symmetry/asymmetry of the margins were manipulated. For identical margins, both statistics had good Type I error control. For a unidirectional shift in margins, both statistics had good power. As expected, when shifts in the margins were balanced across categories, the t test had little power. Q demonstrated good power for all conditions and identified almost all items identified by the t test. Q shows substantial promise for monitoring of trend scoring.
| S-EPMC5978607 | biostudies-literature