Dataset Information

The Effects of Vignette Scoring on Reliability and Validity of Self-Reports.

ABSTRACT: The research presented in this article combines mathematical derivations and empirical results to investigate effects of the nonparametric anchoring vignette approach proposed by King, Murray, Salomon, and Tandon on the reliability and validity of rating data. The anchoring vignette approach aims to correct rating data for response styles to improve comparability across individuals and groups. Vignettes are used to adjust self-assessment responses on the respondent level but entail significant assumptions: They are supposed to be invariant across respondents, and the responses to vignette prompts are supposed to be without error and strictly ordered. This article shows that these assumptions are not always met and that the anchoring vignette approach leads to higher Cronbach's alpha values and increased correlations among adjusted variables regardless of whether the assumptions of the approach are met or violated. Results suggest that the underlying assumptions and effects of the anchoring vignette approach should be carefully examined as the increased correlations and reliability estimates can be observed even for response variables that are independent random draws and uncorrelated with any other variable.

SUBMITTER: von Davier M

PROVIDER: S-EPMC5978608 | biostudies-literature | 2018 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

The Effects of Vignette Scoring on Reliability and Validity of Self-Reports.

von Davier Matthias M Shin Hyo-Jeong HJ Khorramdel Lale L Stankov Lazar L

Applied psychological measurement 20170927 4

The research presented in this article combines mathematical derivations and empirical results to investigate effects of the nonparametric anchoring vignette approach proposed by King, Murray, Salomon, and Tandon on the reliability and validity of rating data. The anchoring vignette approach aims to correct rating data for response styles to improve comparability across individuals and groups. Vignettes are used to adjust self-assessment responses on the respondent level but entail significant a ...[more]

PMID: 29881126

Similar Datasets

Project description:Background:With the rapid increase in ageing population, China is confronted with the daunting challenge of a growing number of patients with neurocognitive disorders (NCDs). This trend makes the maintenance of self-health and early intervention essential, highlighting the need for a tool that assesses self-efficacy of older adults in maintaining brain health or cognitive function. Aim:This study aimed to design the Brain Health Self-Efficacy Scale (BHSES) to measure elderly individuals' attitudes to NCDs, motivations and future plans for controlling risks. The psychometric properties of BHSES have been validated. Methods:Based on the current literature and relevant models, a 19-item scale was created during the first stage. A total of 660 older adults in the Yinhang community of Shanghai were included. The statistical approaches of item analysis, exploratory factor analysis (EFA), confirmatory factor analysis (CFA), criterion-related validity and reliability test were used to evaluate the quality of BHSES. In addition, the Geriatric Depression Scale (GDS) and the Self-Rating Anxiety Scale (SAS) were used as criteria to test the criterion-related validity. Results:To test item differentiation, the study adopted item analysis and excluded item 8. Selecting a random half of the sample for EFA, the BHSES was refined to 16 items, which were categorised into the following three dimensions: 'memory belief efficacy', 'self-care efficacy' and 'future planning efficacy'. These were highly consistent with the hypothesis model. Its cumulative variance contribution rate reached 61.14%, with factor loads of all items above 0.5. The three-factor model was confirmed by the remaining data through CFA. All fit indices reached the acceptable level (?2=3.045, Goodness of Fit Index=0.898, adjusted Goodness of Fit Index=0.863, Comparative Fit Index=0.916, Incremental Fit Index=0.917, Tucker-Lewis Index=0.900, root mean square error of approximation=0.079 and root mean residual=0.068). The GDS and SAS scores revealed significant correlations with the BHSES score, indicating a high criterion-related validity. The overall Cronbach's ? coefficient was 0.793, with the ? coefficients' distribution of subdimensions ranging from 0.748 to 0.883. Conclusions:The 16-item, self-compiled BHSES is a reliable and valid measurement. It could help identify older adults with potential risks for developing NCDs or with high suspicion of cognitive impairment onset in recent periods and also offer insight into tracking brain health self-efficacy in association with cognition status.

Project description:BackgroundFew assessment instruments have examined the nutrition and physical activity environments in child care, and none are self-administered. Given the emerging focus on child care settings as a target for intervention, a valid and reliable measure of the nutrition and physical activity environment is needed.MethodsTo measure inter-rater reliability, 59 child care center directors and 109 staff completed the self-assessment concurrently, but independently. Three weeks later, a repeat self-assessment was completed by a sub-sample of 38 directors to assess test-retest reliability. To assess criterion validity, a researcher-administered environmental assessment was conducted at 69 centers and was compared to a self-assessment completed by the director. A weighted kappa test statistic and percent agreement were calculated to assess agreement for each question on the self-assessment.ResultsFor inter-rater reliability, kappa statistics ranged from 0.20 to 1.00 across all questions. Test-retest reliability of the self-assessment yielded kappa statistics that ranged from 0.07 to 1.00. The inter-quartile kappa statistic ranges for inter-rater and test-retest reliability were 0.45 to 0.63 and 0.27 to 0.45, respectively. When percent agreement was calculated, questions ranged from 52.6% to 100% for inter-rater reliability and 34.3% to 100% for test-retest reliability. Kappa statistics for validity ranged from -0.01 to 0.79, with an inter-quartile range of 0.08 to 0.34. Percent agreement for validity ranged from 12.9% to 93.7%.ConclusionThis study provides estimates of criterion validity, inter-rater reliability and test-retest reliability for an environmental nutrition and physical activity self-assessment instrument for child care. Results indicate that the self-assessment is a stable and reasonably accurate instrument for use with child care interventions. We therefore recommend the Nutrition and Physical Activity Self-Assessment for Child Care (NAP SACC) instrument to researchers and practitioners interested in conducting healthy weight intervention in child care. However, a more robust, less subjective measure would be more appropriate for researchers seeking an outcome measure to assess intervention impact.

Dataset Information

The Effects of Vignette Scoring on Reliability and Validity of Self-Reports.

Publications

The Effects of Vignette Scoring on Reliability and Validity of Self-Reports.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure