Dataset Information

Restricted sample variance reduces generalizability.

ABSTRACT: One factor that affects the reliability of observed scores is restriction of range on the construct measured for a particular group of study participants. This study illustrates how researchers can use generalizability theory to evaluate the impact of restriction of range in particular sample characteristics on the generalizability of test scores and to estimate how changes in measurement design could improve the generalizability of the test scores. An observer-rated measure of child self-regulation (Response to Challenge Scale; Lakes, 2011) is used to examine scores for 198 children (Grades K through 5) within the generalizability theory (GT) framework. The generalizability of ratings within relatively developmentally homogeneous samples is examined and illustrates the effect of reduced variance among ratees on generalizability. Forecasts for g coefficients of various D study designs demonstrate how higher generalizability could be achieved by increasing the number of raters or items. In summary, the research presented illustrates the importance of and procedures for evaluating the generalizability of a set of scores in a particular research context.

SUBMITTER: Lakes KD

PROVIDER: S-EPMC3657319 | biostudies-literature | 2013 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Restricted sample variance reduces generalizability.

Lakes Kimberley D KD

Psychological assessment 20121203 2

One factor that affects the reliability of observed scores is restriction of range on the construct measured for a particular group of study participants. This study illustrates how researchers can use generalizability theory to evaluate the impact of restriction of range in particular sample characteristics on the generalizability of test scores and to estimate how changes in measurement design could improve the generalizability of the test scores. An observer-rated measure of child self-regula ...[more]

PMID: 23205627

Similar Datasets

Project description:BackgroundThe ratings of physician-patient communication are an important indicator of the quality of health care delivery and provide guidance for many important decisions in the health care setting and in health research. But there is no gold standard to assess physician-patient communication. Thus, depending on the specific measurement condition, multiple sources of variance may contribute to the total score variance of ratings of physician-patient communication. This may systematically impair the validity of conclusions drawn from rating data.ObjectiveTo examine the extent to which different measurement conditions and rater perspectives, respectively contribute to the variance of physician-patient communication ratings.MethodsVariance components of ratings of physician-patient communication gained from 32 general practitioners and 252 patients from 25 family practices in Germany were analyzed using generalizability theory. The communication dimensions "shared decision making", "effective and open communication" and "satisfaction" were considered.ResultsPhysician-patient communication ratings most substantially reflect unique rater-perspective and communication dimension combinations (32.7% interaction effect). The ratings also represented unique physician and rater-perspective combinations (16.3% interaction effect). However, physicians' communication behavior and the observed communication dimensions revealed only a low extent of score variance (1% physician effect; 3.7% communication dimension effect). Approximately half of the variance remained unexplained (46.2% three-way interaction, confounded with error).ConclusionThe ratings of physician-patient communication minimally reflect physician communication skills in general. Instead, these ratings exhibit primarily differences among physicians and patients in their tendency to perceive shared decision making and effective and open communication and to be satisfied with communication, regardless of the communication behavior of physicians. Rater training and assessing low inferential ratings of physician-patient communication dimensions should be considered when subjective aspects of rater perspectives are not of interest.

Project description:Machine learning techniques, specifically gradient-enhanced Kriging (GEK), have been implemented for molecular geometry optimization. GEK-based optimization has many advantages compared to conventional-step-restricted second-order truncated expansion-molecular optimization methods. In particular, the surrogate model given by GEK can have multiple stationary points, will smoothly converge to the exact model as the number of sample points increases, and contains an explicit expression for the expected error of the model function at an arbitrary point. Machine learning is, however, associated with abundance of data, contrary to the situation desired for efficient geometry optimizations. In this paper, we demonstrate how the GEK procedure can be utilized in a fashion such that in the presence of few data points, the surrogate surface will in a robust way guide the optimization to a minimum of a potential energy surface. In this respect, the GEK procedure will be used to mimic the behavior of a conventional second-order scheme but retaining the flexibility of the superior machine learning approach. Moreover, the expected error will be used in the optimizations to facilitate restricted-variance optimizations. A procedure which relates the eigenvalues of the approximate guessed Hessian with the individual characteristic lengths, used in the GEK model, reduces the number of empirical parameters to optimize to two: the value of the trend function and the maximum allowed variance. These parameters are determined using the extended Baker (e-Baker) and part of the Baker transition-state (Baker-TS) test suites as a training set. The so-created optimization procedure is tested using the e-Baker, full Baker-TS, and S22 test suites, at the density functional theory and second-order Møller-Plesset levels of approximation. The results show that the new method is generally of similar or better performance than a state-of-the-art conventional method, even for cases where no significant improvement was expected.

Project description:BACKGROUND: Supervised learning for classification of cancer employs a set of design examples to learn how to discriminate between tumors. In practice it is crucial to confirm that the classifier is robust with good generalization performance to new examples, or at least that it performs better than random guessing. A suggested alternative is to obtain a confidence interval of the error rate using repeated design and test sets selected from available examples. However, it is known that even in the ideal situation of repeated designs and tests with completely novel samples in each cycle, a small test set size leads to a large bias in the estimate of the true variance between design sets. Therefore different methods for small sample performance estimation such as a recently proposed procedure called Repeated Random Sampling (RSS) is also expected to result in heavily biased estimates, which in turn translates into biased confidence intervals. Here we explore such biases and develop a refined algorithm called Repeated Independent Design and Test (RIDT). RESULTS: Our simulations reveal that repeated designs and tests based on resampling in a fixed bag of samples yield a biased variance estimate. We also demonstrate that it is possible to obtain an improved variance estimate by means of a procedure that explicitly models how this bias depends on the number of samples used for testing. For the special case of repeated designs and tests using new samples for each design and test, we present an exact analytical expression for how the expected value of the bias decreases with the size of the test set. CONCLUSION: We show that via modeling and subsequent reduction of the small sample bias, it is possible to obtain an improved estimate of the variance of classifier performance between design sets. However, the uncertainty of the variance estimate is large in the simulations performed indicating that the method in its present form cannot be directly applied to small data sets.

Dataset Information

Restricted sample variance reduces generalizability.

Publications

Restricted sample variance reduces generalizability.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets