A Bayesian hierarchical latent trait model for estimating rater bias and reliability in large-scale performance assessment.
Ontology highlight
ABSTRACT:
SUBMITTER: Zupanc K
PROVIDER: S-EPMC5882162 | biostudies-literature | 2018
REPOSITORIES: biostudies-literature
Publications
A Bayesian hierarchical latent trait model for estimating rater bias and reliability in large-scale performance assessment.
Zupanc Kaja K Štrumbelj Erik E
PloS one 20180403 4
We propose a novel approach to modelling rater effects in scoring-based assessment. The approach is based on a Bayesian hierarchical model and simulations from the posterior distribution. We apply it to large-scale essay assessment data over a period of 5 years. Empirical results suggest that the model provides a good fit for both the total scores and when applied to individual rubrics. We estimate the median impact of rater effects on the final grade to be ± 2 points on a 50 point scale, while ...[more]
Similar Datasets
Project description:The diagnostic accuracy of a test or rater has a crucial impact on clinical decision making. The assessment of diagnostic accuracy for multiple tests or raters also merits much attention. A Bayesian hierarchical conditional independence latent class model for estimating sensitivities and specificities for a large group of tests or raters is proposed, which is applicable to both with-gold-standard and without-gold-standard situations. Through the hierarchical structure, not only are the sensitivities and specificities of individual tests estimated, but also the diagnostic performance of the whole group of tests. For a small group of tests or raters, the proposed model is further extended by introducing pairwise covariances between tests to improve the fitting and to allow for more modeling flexibility. Correlation residual analysis is applied to detect any significant covariance between multiple tests. Just Another Gibbs Sampler (JAGS) implementation is efficiently adopted for both models. Three real data sets from literature are analyzed to explicitly illustrate the proposed methods.
| S-EPMC6884669 | biostudies-literature
Project description:This dataset belongs to a large-scale randomized controlled trial (RCT) in educational research targeting English learning students and their teachers' instructional capacity. The dataset includes ratings conducted through classroom observations of 45-minute English as a Second language (ESL) blocks. Each coder rated 60 recorded video segments collected from each teacher. During the 20-second segment, ratings of six domains of teachers' instruction (i.e., ESL Strategies, Group, Activity Structure, Mode, Language Content, Language of Teacher, Language of Student) were collected. The dataset is organized by teacher, by coder, and by domain, for researchers to analyze inter-rater reliability among coders by domain and/or cross-domain. This data article is related to the research article Tong et al. [3] on "The determination of appropriate coefficient indices for inter-rater reliability: using classroom observation instruments as fidelity measures in large-scale randomized research".
| S-EPMC7044508 | biostudies-literature
Project description:PurposeThere is limited knowledge on the reliability of risk of bias (ROB) tools for assessing internal validity in systematic reviews of exposure and frequency studies. We aimed to identify and then compare the inter-rater reliability (IRR) of six commonly used tools for frequency (Loney scale, Gyorkos checklist, American Academy of Neurology [AAN] tool) and exposure (Newcastle-Ottawa scale, SIGN50 checklist, AAN tool) studies.MethodsSix raters independently assessed the ROB of 30 frequency and 30 exposure studies using the three respective ROB tools. Articles were rated as low, intermediate, or high ROB. We calculated an intraclass correlation coefficient (ICC) for each tool and category of ROB tool. We compared the IRR between ROB tools and tool type by inspection of overlapping ICC 95% CIs and by comparing their coefficients after transformation to Fisher's Z values. We assessed the criterion validity of the AAN ROB tools by calculating an ICC for each rater in comparison with the original ratings from the AAN.ResultsAll individual ROB tools had an IRR in the substantial range or higher (ICC point estimates between 0.61 and 0.80). The IRR was almost perfect (ICC point estimate > 0.80) for the AAN frequency tool and the SIGN50 checklist. All tools were comparable in IRR, except for the AAN frequency tool which had a significantly higher ICC than the Gyorkos checklist (p = 0.021) and trended towards a higher ICC when compared to the Loney scale (p = 0.085). When examined by category of ROB tool, scales, and checklists had a substantial IRR, whereas the AAN tools had an almost perfect IRR. For the criterion validity of the AAN ROB tools, the average agreement between our raters and the original AAN ratings was moderate.ConclusionAll tools had substantial IRRs except for the AAN frequency tool and the SIGN50 checklist, which both had an almost perfect IRR. The AAN ROB tools were the only category of ROB tools to demonstrate an almost perfect IRR. This category of ROB tools had fewer and simpler criteria. Overall, parsimonious tools with clear instructions, such as those from the AAN, may provide more reliable ROB assessments.
| S-EPMC10702000 | biostudies-literature
Project description:BackgroundThe Approximate Bayesian Computation (ABC) approach has been used to infer demographic parameters for numerous species, including humans. However, most applications of ABC still use limited amounts of data, from a small number of loci, compared to the large amount of genome-wide population-genetic data which have become available in the last few years.ResultsWe evaluated the performance of the ABC approach for three 'population divergence' models - similar to the 'isolation with migration' model - when the data consists of several hundred thousand SNPs typed for multiple individuals by simulating data from known demographic models. The ABC approach was used to infer demographic parameters of interest and we compared the inferred values to the true parameter values that was used to generate hypothetical "observed" data. For all three case models, the ABC approach inferred most demographic parameters quite well with narrow credible intervals, for example, population divergence times and past population sizes, but some parameters were more difficult to infer, such as population sizes at present and migration rates. We compared the ability of different summary statistics to infer demographic parameters, including haplotype and LD based statistics, and found that the accuracy of the parameter estimates can be improved by combining summary statistics that capture different parts of information in the data. Furthermore, our results suggest that poor choices of prior distributions can in some circumstances be detected using ABC. Finally, increasing the amount of data beyond some hundred loci will substantially improve the accuracy of many parameter estimates using ABC.ConclusionsWe conclude that the ABC approach can accommodate realistic genome-wide population genetic data, which may be difficult to analyze with full likelihood approaches, and that the ABC can provide accurate and precise inference of demographic parameters from these data, suggesting that the ABC approach will be a useful tool for analyzing large genome-wide datasets.
| S-EPMC3368717 | biostudies-literature
Project description:Study designCross sectional.ObjectivesTo determine the inter-rater and intra-rater reliability of the Spanish version of the Berg Balance Scale with incomplete spinal cord injured subjects.SettingCINER Rehabilitation Center.MethodsWe administered and video recorded the Spanish version of the Berg Balance Scale to 20 incomplete SCI patients. Two raters scored the videos on two different occasions at least three weeks apart. We used intraclass correlation coefficient (ICC) and a confidence interval (CI) of 95% to evaluate the inter-rater and intra-rater (test-retest) reliability of the BBS total scores.ResultsICC values for inter-rater reliability at first and second observation were 0.99 (95% CI 0.97-1.00) and 0.99 (95% CI 0.99-1.00), respectively. Intra-rater ICC for rater 1 was 1.00 (95% CI 1.00-1.00) and for rater 2 was 1.00 (95% CI 0.99-1.00). All of them were excellent.ConclusionsThe results indicate that the Spanish version of the Berg Balance Scale is a reliable tool to evaluate spinal cord injured patients' balance.
| S-EPMC7189380 | biostudies-literature
Project description:Introduction: To ensure the quality of clinical trial safety data, universal data standards are required. In 2019 the International Neonatal Consortium (INC) published a neonatal adverse event severity scale (NAESS) to standardize the reporting of adverse event (AE) severity. In this study the reliability of AE severity grading with INC NAESS was prospectively assessed in a real-world setting. Methods: Severity of AEs was assessed by two independent observers at each of four centers across the world. In each center two series of 30 neonatal adverse events were assessed by both observers: in a first phase with a generic (Common Terminology Criteria for Adverse Events, CTCAE) severity scale not specific to neonates, and in a second phase with INC NAESS (after a structured training). Intraclass correlation coefficients (ICC) were calculated to express inter-rater agreement in both phases, and bootstrap sampling was used to compare them. Results: 120 AEs were included in each of both phases. The ICC with the use of INC NAESS in phase 2 was 0.69. This represents a significant but modest improvement in comparison to the initial ICC of 0.66 in phase 1 (confidence interval of ratio of ICC in phase 2 to phase 1 = 1.005-1.146; excludes 1). The ICC was higher for those AEs for which a diagnosis specific AE severity table was available in INC NAESS (ICC 0.80). Discussion: Good inter-rater reliability of the INC NAESS was demonstrated in four neonatal intensive care units (NICUs) across the globe. The ICC is comparable to what is reported for scales with similar purposes in different populations. There is a modest, but significant, improvement in inter-rater agreement in comparison to the naïve phase without INC NAESS. The better performance when reviewers use AE-specific NAESS tables highlights the need to expand the number of AEs that are covered by specific criteria in the current version of INC NAESS.
| S-EPMC10512550 | biostudies-literature
Project description:Decisions about typhoid fever prevention and control are based on estimates of typhoid incidence and their uncertainty. Lack of specific clinical diagnostic criteria, poorly sensitive diagnostic tests, and scarcity of accurate and complete datasets contribute to difficulties in calculating age-specific population-level typhoid incidence. Using data from the Strategic Typhoid Alliance across Africa and Asia program, we integrated demographic censuses, healthcare utilization surveys, facility-based surveillance, and serological surveillance from Malawi, Nepal, and Bangladesh to account for under-detection of cases. We developed a Bayesian approach that adjusts the count of reported blood-culture-positive cases for blood culture detection, blood culture collection, and healthcare seeking-and how these factors vary by age-while combining information from prior published studies. We validated the model using simulated data. The ratio of observed to adjusted incidence rates was 7.7 (95% credible interval [CrI]: 6.0-12.4) in Malawi, 14.4 (95% CrI: 9.3-24.9) in Nepal, and 7.0 (95% CrI: 5.6-9.2) in Bangladesh. The probability of blood culture collection led to the largest adjustment in Malawi, while the probability of seeking healthcare contributed the most in Nepal and Bangladesh; adjustment factors varied by age. Adjusted incidence rates were within or below the seroincidence rate limits of typhoid infection. Estimates of blood-culture-confirmed typhoid fever without these adjustments results in considerable underestimation of the true incidence of typhoid fever. Our approach allows each phase of the reporting process to be synthesized to estimate the adjusted incidence of typhoid fever while correctly characterizing uncertainty, which can inform decision-making for typhoid prevention and control.
| S-EPMC7613088 | biostudies-literature
Project description:Decisions about typhoid fever prevention and control are based on estimates of typhoid incidence and their uncertainty. Lack of specific clinical diagnostic criteria, poorly sensitive diagnostic tests, and scarcity of accurate and complete datasets contribute to difficulties in calculating age-specific population-level typhoid incidence. Using data from the Strategic Typhoid Alliance across Africa and Asia program, we integrated demographic censuses, healthcare utilization surveys, facility-based surveillance, and serological surveillance from Malawi, Nepal, and Bangladesh to account for under-detection of cases. We developed a Bayesian approach that adjusts the count of reported blood-culture-positive cases for blood culture detection, blood culture collection, and healthcare seeking-and how these factors vary by age-while combining information from prior published studies. We validated the model using simulated data. The ratio of observed to adjusted incidence rates was 7.7 (95% credible interval [CrI]: 6.0-12.4) in Malawi, 14.4 (95% CrI: 9.3-24.9) in Nepal, and 7.0 (95% CrI: 5.6-9.2) in Bangladesh. The probability of blood culture collection led to the largest adjustment in Malawi, while the probability of seeking healthcare contributed the most in Nepal and Bangladesh; adjustment factors varied by age. Adjusted incidence rates were within or below the seroincidence rate limits of typhoid infection. Estimates of blood-culture-confirmed typhoid fever without these adjustments results in considerable underestimation of the true incidence of typhoid fever. Our approach allows each phase of the reporting process to be synthesized to estimate the adjusted incidence of typhoid fever while correctly characterizing uncertainty, which can inform decision-making for typhoid prevention and control.
| S-EPMC9291985 | biostudies-literature
Project description:PurposeDevelop a hierarchical longitudinal regression model for estimating local rates of change of macular ganglion cell complex (GCC) measurements with optical coherence tomography (OCT).MethodsWe enrolled 112 eyes with four or more macular OCT images and ≥2 years of follow-up. GCC thickness measurements within central 6 × 6 superpixels were extracted from macular volume scans. We fit data from each superpixel separately with several hierarchical Bayesian random-effects models. Models were compared with the Watanabe-Akaike information criterion. For our preferred model, we estimated population and individual slopes and intercepts (baseline thickness) and their correlation.ResultsMean (SD) follow-up time and median (interquartile range) baseline 24-2 visual field mean deviation were 3.6 (0.4) years and -6.8 (-12.2 to -4.3) dB, respectively. The random intercepts and slopes model with random residual variance was the preferred model. While more individual and population negative slopes were observed in the paracentral and papillomacular superpixels, superpixels in the superotemporal and inferior regions displayed the highest correlation between baseline thickness and rates of change (r = -0.43 to -0.50 for the top five correlations).ConclusionsA Bayesian linear hierarchical model with random intercepts/slopes and random variances is an optimal initial model for estimating GCC slopes at population and individual levels. This novel model is an efficient method for estimating macular rates of change and probability of glaucoma progression locally.Translational relevanceThe proposed Bayesian hierarchical model can be applied to various macular outcomes from different OCT devices and to superpixels of variable sizes to estimate local rates of change and progression probability.
| S-EPMC8054624 | biostudies-literature
Project description:Myriad human activities increasingly threaten the existence of many species. A variety of conservation interventions such as habitat restoration, protected areas, and captive breeding have been used to prevent extinctions. Evaluating the effectiveness of these interventions requires appropriate statistical methods, given the quantity and quality of available data. Historically, analysis of variance has been used with some form of predetermined before-after control-impact design to estimate the effects of large-scale experiments or conservation interventions. However, ad hoc retrospective study designs or the presence of random effects at multiple scales may preclude the use of these tools. We evaluated the effects of a large-scale supplementation program on the density of adult Chinook salmon Oncorhynchus tshawytscha from the Snake River basin in the northwestern United States currently listed under the U.S. Endangered Species Act. We analyzed 43 years of data from 22 populations, accounting for random effects across time and space using a form of Bayesian hierarchical time-series model common in analyses of financial markets. We found that varying degrees of supplementation over a period of 25 years increased the density of natural-origin adults, on average, by 0-8% relative to nonsupplementation years. Thirty-nine of the 43 year effects were at least two times larger in magnitude than the mean supplementation effect, suggesting common environmental variables play a more important role in driving interannual variability in adult density. Additional residual variation in density varied considerably across the region, but there was no systematic difference between supplemented and reference populations. Our results demonstrate the power of hierarchical Bayesian models to detect the diffuse effects of management interventions and to quantitatively describe the variability of intervention success. Nevertheless, our study could not address whether ecological factors (e.g., competition) were more important than genetic considerations (e.g., inbreeding depression) in determining the response to supplementation.
| S-EPMC4449763 | biostudies-literature