Deflation-Corrected Estimators of Reliability.
Ontology highlight
ABSTRACT:
SUBMITTER: Metsamuuronen J
PROVIDER: S-EPMC8781775 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:The reliability of a test score is discussed from the viewpoint of underestimation of and, specifically, deflation in estimates or reliability. Many widely used estimators are known to underestimate reliability. Empirical cases have shown that estimates by widely used estimators such as alpha, theta, omega, and rho may be deflated by up to 0.60 units of reliability or even more, with certain types of datasets. The reason for this radical deflation lies in the item-score correlation (Rit) embedded in the estimators: because the estimates by Rit are deflated when the number of categories in scales are far from each other, as is always the case with item and score, the estimates of reliability are deflated as well. A short-cut method to reach estimates closer to the true magnitude, new types of estimators, and deflation-corrected estimators of reliability (DCERs), are studied in the article. The empirical section is a study on the characteristics of combinations of DCERs formed by different bases for estimators (alpha, theta, omega, and rho), different alternative estimators of correlation as the linking factor between item and the score variable, and different conditions. Based on the simulation, an initial typology of the families of DCERs is presented: some estimators are better with binary items and some with polytomous items; some are better with small sample sizes and some with larger ones.
| S-EPMC9341485 | biostudies-literature
Project description:BackgroundInterrater reliability, aka intercoder reliability, is defined as true agreement between raters, aka coders, without chance agreement. It is used across many disciplines including medical and health research to measure the quality of ratings, coding, diagnoses, or other observations and judgements. While numerous indices of interrater reliability are available, experts disagree on which ones are legitimate or more appropriate. Almost all agree that percent agreement (ao), the oldest and the simplest index, is also the most flawed because it fails to estimate and remove chance agreement, which is produced by raters' random rating. The experts, however, disagree on which chance estimators are legitimate or better. The experts also disagree on which of the three factors, rating category, distribution skew, or task difficulty, an index should rely on to estimate chance agreement, or which factors the known indices in fact rely on. The most popular chance-adjusted indices, according to a functionalist view of mathematical statistics, assume that all raters conduct intentional and maximum random rating while typical raters conduct involuntary and reluctant random rating. The mismatches between the assumed and the actual rater behaviors cause the indices to rely on mistaken factors to estimate chance agreement, leading to the numerous paradoxes, abnormalities, and other misbehaviors of the indices identified by prior studies.MethodsWe conducted a 4 × 8 × 3 between-subject controlled experiment with 4 subjects per cell. Each subject was a rating session with 100 pairs of rating by two raters, totaling 384 rating sessions as the experimental subjects. The experiment tested seven best-known indices of interrater reliability against the observed reliabilities and chance agreements. Impacts of the three factors, i.e., rating category, distribution skew, and task difficulty, on the indices were tested.ResultsThe most criticized index, percent agreement (ao), showed as the most accurate predictor of reliability, reporting directional r2 = .84. It was also the third best approximator, overestimating observed reliability by 13 percentage points on average. The three most acclaimed and most popular indices, Scott's π, Cohen's κ and Krippendorff's α, underperformed all other indices, reporting directional r2 = .312 and underestimated reliability by 31.4 ~ 31.8 points. The newest index, Gwet's AC1, emerged as the second-best predictor and the most accurate approximator. Bennett et al's S ranked behind AC1, and Perreault and Leigh's Ir ranked the fourth both for prediction and approximation. The reliance on category and skew and failure to rely on difficulty explain why the six chance-adjusted indices often underperformed ao, which they were created to outperform. The evidence corroborated the notion that the chance-adjusted indices assume intentional and maximum random rating while the raters instead exhibited involuntary and reluctant random rating.ConclusionThe authors call for more empirical studies and especially more controlled experiments to falsify or qualify this study. If the main findings are replicated and the underlying theories supported, new thinking and new indices may be needed. Index designers may need to refrain from assuming intentional and maximum random rating, and instead assume involuntary and reluctant random rating. Accordingly, the new indices may need to rely on task difficulty, rather than distribution skew or rating category, to estimate chance agreement.
| S-EPMC9426226 | biostudies-literature
Project description:In the context of multidimensional structures, with the presence of a common factor and multiple specific or group factors, estimates of reliability require specific estimators. The use of classical procedures such as the alpha coefficient or omega total that ignore structural complexity are not appropriate, since they can lead to strongly biased estimates. Through a simulation study, the bias of six estimators of reliability in multidimensional measures was evaluated and compared. The study is complemented by an empirical illustration that exemplifies the procedure. Results showed that the estimators with the lowest bias in the estimation of the total reliability parameter are omega total, the two versions of greatest lower bound (GLB) and the alpha coefficient, which in turn are also those that produce the highest overestimation of the reliability of the general factor. Nevertheless, the most appropriate estimators, in that they produce less biased estimates of the reliability parameter of the general factor, are omega limit and omega hierarchical.
| S-EPMC8263896 | biostudies-literature
Project description:The interactions among macroprices with leads and lags play a significant role in explaining the behavior of an aggregate price index. Thus, to understand inflation and deflation, it is essential to explore the mechanism according to which these macroprices interact with each other. On the basis of a new method, we show that, irrespective of the sources of shocks, a robust flow of changes occurs in domestic prices from upstream to downstream. Moreover, we demonstrate that macroprices change in clusters, and we identify these clusters. Firms are not symmetric. Overall, our analysis suggests that the inertia arising from input/output linkages in production explains the behavior of aggregate prices.
| S-EPMC7018021 | biostudies-literature
Project description:BackgroundTo explore the feasibility of corrected slack angle acquired from two-dimensional shear wave elastography (2D-SWE) for quantitating the spasticity of medial gastrocnemius (MG) in stroke patients.MethodsConsecutive stroke patients with spastic MG and matched healthy controls were recruited. Intra- and interobserver reliability of 2D-SWE measurement were evaluated, and the correlation between corrected slack angle and modified Ashworth scale (MAS) score was examined. The corrected slack angle before and after botulinum toxin A (BoNT-A) injection was compared and its diagnostic performance in classifying the severity of spasticity were assessed with receiver operating characteristic (ROC) curve analysis.ResultsThe intra- (0.791 95% CI 0.432-0.932) and interobserver (0.751 95% CI 0.382-0.916) reliability of slack angle acquired with 2D-SWE were good. Significant correlation was found between corrected slack angle and MAS score (R = - 0.849, p < 0.001). The corrected slack angle increased after BoNT-A injection. The cutoff value of MAS ≥ 3 had the highest sensitivity (100%) and specificity (93.33%). The positive predictive value (PPV) for classification of MAS ≥ 1+ and the negative predictive value (NPV) for classification of MAS ≥ 3 were greater than 90%.Conclusion2D-SWE was a reliable method to quantitate the post-stroke spasticity. The corrected slack angle had advantage in classifying the severity of spasticity, especially in early identification of mild spasticity and confirmation of severe spasticity.
| S-EPMC8817514 | biostudies-literature
Project description:The prevailing maximum likelihood estimators for inferring power law models from rank-frequency data are biased. The source of this bias is an inappropriate likelihood function. The correct likelihood function is derived and shown to be computationally intractable. A more computationally efficient method of approximate Bayesian computation (ABC) is explored. This method is shown to have less bias for data generated from idealised rank-frequency Zipfian distributions. However, the existing estimators and the ABC estimator described here assume that words are drawn from a simple probability distribution, while language is a much more complex process. We show that this false assumption leads to continued biases when applying any of these methods to natural language to estimate Zipf exponents. We recommend that researchers be aware of the bias when investigating power laws in rank-frequency data.
| S-EPMC8397718 | biostudies-literature
Project description:Tree reconstruction methods are often judged by their accuracy, measured by how close they get to the true tree. Yet, most reconstruction methods like maximum likelihood (ML) do not explicitly maximize this accuracy. To address this problem, we propose a Bayesian solution. Given tree samples, we propose finding the tree estimate that is closest on average to the samples. This "median" tree is known as the Bayes estimator (BE). The BE literally maximizes posterior expected accuracy, measured in terms of closeness (distance) to the true tree. We discuss a unified framework of BE trees, focusing especially on tree distances that are expressible as squared euclidean distances. Notable examples include Robinson-Foulds (RF) distance, quartet distance, and squared path difference. Using both simulated and real data, we show that BEs can be estimated in practice by hill-climbing. In our simulation, we find that BEs tend to be closer to the true tree, compared with ML and neighbor joining. In particular, the BE under squared path difference tends to perform well in terms of both path difference and RF distances.
| S-EPMC3114872 | biostudies-literature
Project description:Basement membrane (BM) is a thin layer of extracellular matrix that surrounds most animal tissues, serving as a physical barrier while allowing nutrient exchange. Although they have important roles in tissue structural integrity, physical properties of BMs remain largely uncharacterized, which limits our understanding of their mechanical functions. Here, we perform pressure-controlled inflation and deflation to directly measure the nonlinear mechanics of BMs in situ. We show that the BMs behave as a permeable, hyperelastic material whose mechanical properties and permeability can be measured in a model-independent manner. Furthermore, we find that BMs exhibit a remarkable nonlinear stiffening behavior, in contrast to the reconstituted Matrigel. This nonlinear stiffening behavior helps the BMs to avoid the snap-through instability (or structural softening) widely observed during the inflation of most elastomeric balloons and thus maintain sufficient confining stress to the enclosed tissues during their growth.
| S-EPMC7980462 | biostudies-literature
Project description: Not available
| S-EPMC10533858 | biostudies-literature
Project description:BackgroundIn COPD patients, hyperinflation impairs cardiac function. We examined whether lung deflation improves oxygen pulse, a surrogate marker of stroke volume.MethodsIn 129 NETT patients with cardiopulmonary exercise testing (CPET) and arterial blood gases (ABG substudy), hyperinflation was assessed with residual volume to total lung capacity ratio (RV/TLC), and cardiac function with oxygen pulse (O(2) pulse=VO(2)/HR) at baseline and 6 months. Medical and surgical patients were divided into "deflators" and "non-deflators" based on change in RV/TLC from baseline (?RV/TLC). We defined deflation as the ?RV/TLC experienced by 75% of surgical patients. We examined changes in O(2) pulse at peak and similar (iso-work) exercise. Findings were validated in 718 patients who underwent CPET without ABGs.ResultsIn the ABG substudy, surgical and medical deflators improved their RV/TLC and peak O(2) pulse (median ?RV/TLC -18.0% vs. -9.3%, p=0.0003; median ?O(2) pulse 13.6% vs. 1.8%, p=0.12). Surgical deflators also improved iso-work O(2) pulse (0.53 mL/beat, p=0.04 at 20 W). In the validation cohort, surgical deflators experienced a greater improvement in peak O(2) pulse than medical deflators (mean 18.9% vs. 1.1%). In surgical deflators improvements in O(2) pulse at rest and during unloaded pedaling (0.32 mL/beat, p<0.0001 and 0.47 mL/beat, p<0.0001, respectively) corresponded with significant reductions in HR and improvements in VO(2). On multivariate analysis, deflators were 88% more likely than non-deflators to have an improvement in O(2) pulse (OR 1.88, 95% CI 1.30-2.72, p=0.0008).ConclusionIn COPD, decreased hyperinflation through lung volume reduction is associated with improved O(2) pulse.
| S-EPMC3233645 | biostudies-literature