Dataset Information

Agreement between Computerized and Human Assessment of Performance on the Ruff Figural Fluency Test.

ABSTRACT: The Ruff Figural Fluency Test (RFFT) is a sensitive test for nonverbal fluency suitable for all age groups. However, assessment of performance on the RFFT is time-consuming and may be affected by interrater differences. Therefore, we developed computer software specifically designed to analyze performance on the RFFT by automated pattern recognition. The aim of this study was to compare assessment by the new software with conventional assessment by human raters. The software was developed using data from the Lifelines Cohort Study and validated in an independent cohort of the Prevention of Renal and Vascular End Stage Disease (PREVEND) study. The total study population included 1,761 persons: 54% men; mean age (SD), 58 (10) years. All RFFT protocols were assessed by the new software and two independent human raters (criterion standard). The mean number of unique designs (SD) was 81 (29) and the median number of perseverative errors (interquartile range) was 9 (4 to 16). The intraclass correlation coefficient (ICC) between the computerized and human assessment was 0.994 (95%CI, 0.988 to 0.996; p<0.001) and 0.991 (95%CI, 0.990 to 0.991; p<0.001) for the number of unique designs and perseverative errors, respectively. The mean difference (SD) between the computerized and human assessment was -1.42 (2.78) and +0.02 (1.94) points for the number of unique designs and perseverative errors, respectively. This was comparable to the agreement between two independent human assessments: ICC, 0.995 (0.994 to 0.995; p<0.001) and 0.985 (0.982 to 0.988; p<0.001), and mean difference (SD), -0.44 (2.98) and +0.56 (2.36) points for the number of unique designs and perseverative errors, respectively. We conclude that the agreement between the computerized and human assessment was very high and comparable to the agreement between two independent human assessments. Therefore, the software is an accurate tool for the assessment of performance on the RFFT.

SUBMITTER: Elderson MF

PROVIDER: S-EPMC5035016 | biostudies-literature | 2016

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Agreement between Computerized and Human Assessment of Performance on the Ruff Figural Fluency Test.

Elderson Martin F MF Pham Sander S van Eersel Marlise E A ME Wolffenbuttel Bruce H R BH Kok Johan J Gansevoort Ron T RT Tucha Oliver O van der Klauw Melanie M MM Slaets Joris P J JP Izaks Gerbrand J GJ

PloS one 20160923 9

The Ruff Figural Fluency Test (RFFT) is a sensitive test for nonverbal fluency suitable for all age groups. However, assessment of performance on the RFFT is time-consuming and may be affected by interrater differences. Therefore, we developed computer software specifically designed to analyze performance on the RFFT by automated pattern recognition. The aim of this study was to compare assessment by the new software with conventional assessment by human raters. The software was developed using ...[more]

PMID: 27661083

Similar Datasets

Project description:BackgroundThe Ruff Figural Fluency Test (RFFT) is a valid but time-consuming and labour-intensive cognitive paper-and-pencil test. A digital RFFT was developed that can be conducted independently using an iPad and Apple Pencil and RFFT scores are computed automatically. We investigated the validity and reliability of this digital RFFT.MethodsWe randomly allocated participants to the digital or paper-and-pencil RFFT. After the first test, the other test was performed immediately (cross-over). Participants were invited for a second digital RFFT 1 week later. For the digital RFFT, an (automatic) algorithm and two independent raters (criterion standard) assessed the number of unique designs (UD) and perseverative errors (PE). These raters also assessed the paper-and-pencil RFFT. We used Intraclass correlation coefficients (ICC), sensitivity, specificity, %-agreement, Kappa, and Bland-Altman plots.ResultsWe included 94 participants (mean (SD) age 39.9 (14.8), 73.4% follow-up). Mean (SD) UD and median (IQR) PE of the digital RFFT were 84.2 (26.0) and 4 (2-7.3), respectively. Agreement between manual and automatic scoring of the digital RFFT was high for UD (ICC = 0.99, 95% CI 0.98, 0.99, sensitivity = 0.98; specificity = 0.96) and PE (ICC = 0.99, 95% CI 0.98, 0.99; sensitivity = 0.90, specificity = 1.00), indicating excellent criterion validity. Small but significant differences in UD were found between the automatic and manual scoring (mean difference: - 1.12, 95% CI - 1.92, - 0.33). Digital and paper-and-pencil RFFT had moderate agreement for UD (ICC = 0.73, 95% CI 0.34, 0.87) and poor agreement for PE (ICC = 0.47, 95% CI 0.30, 0.62). Participants had fewer UD on the digital than paper-and-pencil RFFT (mean difference: - 7.09, 95% CI - 11.80, - 2.38). The number of UD on the digital RFFT was associated with higher education (Spearman's r = 0.43, p < 0.001), and younger age (Pearson's r = - 0.36, p < 0.001), showing its ability to discriminate between different age categories and levels of education. Test-retest reliability was moderate (ICC = 0.74, 95% CI 0.61, 0.83).ConclusionsThe automatic scoring of the digital RFFT has good criterion and convergent validity. There was low agreement between the digital RFFT and paper-and-pencil RFFT and moderate test-retest reliability, which can be explained by learning effects. The digital RFFT is a valid and reliable instrument to measure executive cognitive function among the general population and is a feasible alternative to the paper-and-pencil RFFT in large-scale studies. However, its scores cannot be used interchangeably with the paper-and-pencil RFFT scores.

Project description:BackgroundThe Ruff Figural Fluency Test (RFFT; a pencil and paper test) and the CogState (a computerized cognitive test battery) are well-validated and suitable tests to evaluate cognitive functioning in large observational studies at the population level. The LifeLines Cohort Study includes the RFFT as baseline measurement and incorporated the CogState as replacement for the RFFT at follow-up. It is unknown how these two tests relate to each other. Therefore, the aim of this study is to examine the correlation between the RFFT and the CogState and the impact of demographic characteristics on this association.MethodsA subcohort of the LifeLines Cohort Study, a large population based cohort study, participated in this study. Correlations between the RFFT and six subtasks of the CogState were examined. Subgroup analyses were performed to investigate the influence of age, education, and gender on the results. With sensitivity analyses we investigated the influence of computer experience and (physical) impairments.ResultsA total of 509 participants (mean age (SD): 53 years (14.6); range 18-87 years) participated in this study. All correlations between the RFFT and the CogState were statistically significant (except for the correlation between the RFFT error ratio and the CogState One Back Task), ranging from -0.39 to 0.28. Stratifying the analyses for age, education, and gender did not substantially affect our conclusions. Sensitivity analyses showed no substantial influence of level of computer experience or (physical) impairments.ConclusionsCorrelations found in the present study were only weak to moderate, indicating that cognitive functioning measured by the RFFT does not measure the same components of cognitive functioning as six subtasks of the CogState. Computerized testing such as the CogState may be very well suited for large cohort studies to assess cognitive functioning in the general population and to identify cognitive changes as early as possible, as it is a less time- and labor intensive tool.

Project description:The purpose of this study was to construct a computerized adaptive test (CAT) for measuring self-care performance (the CAT-SC) in children with developmental disabilities (DD) aged from 6 months to 12 years in a content-inclusive, precise, and efficient fashion.The study was divided into 3 phases: (1) item bank development, (2) item testing, and (3) a simulation study to determine the stopping rules for the administration of the CAT-SC. A total of 215 caregivers of children with DD were interviewed with the 73-item CAT-SC item bank. An item response theory model was adopted for examining the construct validity to estimate item parameters after investigation of the unidimensionality, equality of slope parameters, item fitness, and differential item functioning (DIF). In the last phase, the reliability and concurrent validity of the CAT-SC were evaluated.The final CAT-SC item bank contained 56 items. The stopping rules suggested were (a) reliability coefficient greater than 0.9 or (b) 14 items administered. The results of simulation also showed that 85% of the estimated self-care performance scores would reach a reliability higher than 0.9 with a mean test length of 8.5 items, and the mean reliability for the rest was 0.86. Administering the CAT-SC could reduce the number of items administered by 75% to 84%. In addition, self-care performances estimated by the CAT-SC and the full item bank were very similar to each other (Pearson r = 0.98).The newly developed CAT-SC can efficiently measure self-care performance in children with DD whose performances are comparable to those of TD children aged from 6 months to 12 years as precisely as the whole item bank. The item bank of the CAT-SC has good reliability and a unidimensional self-care construct, and the CAT can estimate self-care performance with less than 25% of the items in the item bank. Therefore, the CAT-SC could be useful for measuring self-care performance in children with DD in clinical and research settings.

Dataset Information

Agreement between Computerized and Human Assessment of Performance on the Ruff Figural Fluency Test.

Publications

Agreement between Computerized and Human Assessment of Performance on the Ruff Figural Fluency Test.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets