Dataset Information

Does assessment type matter? A measurement invariance analysis of online and paper and pencil assessment of the Community Assessment of Psychic Experiences (CAPE).

ABSTRACT:

Background

The psychometric properties of an online test are not necessarily identical to its paper and pencil original. The aim of this study is to test whether the factor structure of the Community Assessment of Psychic Experiences (CAPE) is measurement invariant with respect to online vs. paper and pencil assessment.

Method

The factor structure of CAPE items assessed by paper and pencil (N = 796) was compared with the factor structure of CAPE items assessed by the Internet (N = 21,590) using formal tests for Measurement Invariance (MI). The effect size was calculated by estimating the Signed Item Difference in the Sample (SIDS) index and the Signed Test Difference in the Sample (STDS) for a hypothetical subject who scores 2 standard deviations above average on the latent dimensions.

Results

The more restricted Metric Invariance model showed a significantly worse fit compared to the less restricted Configural Invariance model (χ(2)(23) = 152.75, p<0.001). However, the SIDS indices appear to be small, with an average of -0.11. A STDS of -4.80 indicates that Internet sample members who score 2 standard deviations above average would be expected to score 4.80 points lower on the CAPE total scale (ranging from 42 to 114 points) than would members of the Paper sample with the same latent trait score.

Conclusions

Our findings did not support measurement invariance with respect to assessment method. Because of the small effect sizes, the measurement differences between the online assessed CAPE and its paper and pencil original can be neglected without major consequences for research purposes. However, a person with a high vulnerability for psychotic symptoms would score 4.80 points lower on the total scale if the CAPE is assessed online compared to paper and pencil assessment. Therefore, for clinical purposes, one should be cautious with online assessment of the CAPE.

SUBMITTER: Vleeschouwer M

PROVIDER: S-EPMC3898946 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:ObjectiveThe Community Assessment of Psychic Experiences has been widely translated and commonly used as a measure for psychotic experiences and psychosis proneness in clinical and research environments worldwide. This study aimed to establish the psychometric properties (reliability and validity) and factor structure of a Korean version of the Community Assessment of Psychic Experiences (K-CAPE) in the general population.MethodsA total of 1,467 healthy participants completed K-CAPE and other psychiatric symptom-related scales (Paranoia scale, Patient Health Questionnaire-9, Dissociative Experiences Scale-II, and Oxford-Liverpool Inventory of Feelings and Experiences) via online survey. K-CAPE's internal reliability was analyzed using Cronbach's alpha coefficient. Confirmatory factor analysis (CFA) was performed to investigate whether the original three-factor model (positive, negative, and depressive) and other hypothesized multidimensional models (including positive and negative subfactors) were suitable for our data. Exploratory factor analysis (EFA) was conducted to explore better alternative factor solutions with a follow-up CFA. To assess convergent and discriminant validity, we examined correlations between K-CAPE subscales with other established measures of psychiatric symptoms.ResultsK-CAPE showed good internal consistency in all original three subscales (all greater than α=0.827). The CFA demonstrated that the multidimensional models exhibited relatively better quality than the original three-dimensional model. Although the model fit indices did not reach their respective optimal thresholds, they were within an acceptable range. Results from the EFA indicated 3-5 factor solutions. In 3-factor solution, "negative-avolition" items were founded to be loaded more consistently with depressive items than with the negative dimension. In 4-factor solution, positive items were divided into two subfactors: "positive-bizarre experiences" and "positive-delusional thoughts," while negative symptoms were separated into two distinct subfactors in 5-factor solution: "negative-avolition (expressive)," and "negative-social (experiential)." The correlation coefficients between K-CAPE subscales and corresponding measurements were significant (p<0.001), confirming the convergent and discriminant validity.ConclusionOur study provides evidence to support the reliability and validity of the K-CAPE and its use as a measure of psychotic symptoms in the Korean population. Although alternative factor structures did not improve the model fit, our EFA findings implicate the use of subfactors to investigate more specific domains of positive and negative symptoms. Given the heterogeneous nature of psychotic symptoms, this may be useful in capturing their different underlying mechanisms.

Project description:BACKGROUND:As eye tracking-based assessment of cognition becomes more widely used in older adults, particularly those at risk for dementia, reliable and scalable methods to collect high-quality data are required. Eye tracking-based cognitive tests that utilize device-embedded cameras have the potential to reach large numbers of people as a screening tool for preclinical cognitive decline. However, to fully validate this approach, more empirical evidence about the comparability of eyetracking-based paradigms to existing cognitive batteries is needed. OBJECTIVE:Using a population of clinically normal older adults, we examined the relationship between a 30-minute Visual Paired Comparison (VPC) recognition memory task and cognitive composite indices sensitive to a subtle decline in domains associated with Alzheimer disease. Additionally, the scoring accuracy between software used with a commercial grade eye tracking camera at 60 frames per second (FPS) and a manually scored procedure used with a laptop-embedded web camera (3 FPS) on the VPC task was compared, as well as the relationship between VPC task performance and domain-specific cognitive function. METHODS:A group of 49 clinically normal older adults completed a 30-min VPC recognition memory task with simultaneous recording of eye movements by a commercial-grade eye-tracking camera and a laptop-embedded camera. Relationships between webcam VPC performance and the Preclinical Alzheimer Cognitive Composite (PACC) and National Institutes of Health Toolbox Cognitive Battery (NIHTB-CB) were examined. Inter-rater reliability for manually scored tests was analyzed using Krippendorff's kappa formula, and we used Spearman's Rho correlations to investigate the relationship between VPC performance scores with both cameras. We also examined the relationship between VPC performance with the device-embedded camera and domain-specific cognitive performance. RESULTS:Modest relationships were seen between mean VPC novelty preference and the PACC (r=.39, P=.007) and NIHTB-CB (r=.35, P=.03) composite scores, and additional individual neurocognitive task scores including letter fluency (r=.33, P=.02), category fluency (r=.36, P=.01), and Trail Making Test A (-.40, P=.006). Robust relationships were observed between the 60 FPS eye tracker and 3 FPS webcam on both trial-level VPC novelty preference (r=.82, P<.001) and overall mean VPC novelty preference (r=.92 P<.001). Inter-rater agreement of manually scored web camera data was high (kappa=.84). CONCLUSIONS:In a sample of clinically normal older adults, performance on a 30-minute VPC task correlated modestly with computerized and paper-pencil based cognitive composites that serve as preclinical Alzheimer disease cognitive indices. The strength of these relationships did not differ between camera devices. We suggest that using a device-embedded camera is a reliable and valid way to assess performance on VPC tasks accurately and that these tasks correlate with existing cognitive composites.