Dataset Information

Development and Validation of a Computerized Adaptive Assessment Tool for Discrimination and Measurement of Psychotic Symptoms.

ABSTRACT:

Objective

Time constraints limit the use of measurement-based approaches in research and routine clinical management of psychosis. Computerized adaptive testing (CAT) can reduce administration time, thus increasing measurement efficiency. This study aimed to develop and test the capacity of the CAT-Psychosis battery, both self-administered and rater-administered, to measure the severity of psychotic symptoms and discriminate psychosis from healthy controls.

Methods

An item bank was developed and calibrated. Two raters administered CAT-Psychosis for inter-rater reliability (IRR). Subjects rated themselves and were retested within 7 days for test-retest reliability. The Brief Psychiatric Rating Scale (BPRS) was administered for convergent validity and chart diagnosis, and the Structured Clinical Interview (SCID) was used to test psychosis discriminant validity.

Results

Development and calibration study included 649 psychotic patients. Simulations revealed a correlation of r = .92 with the total 73-item bank score, using an average of 12 items. Validation study included 160 additional patients and 40 healthy controls. CAT-Psychosis showed convergent validity (clinician: r = 0.690; 95% confidence interval [95% CI]: 0.610-0.757; self-report: r = .690; 95% CI: 0.609-0.756), IRR (intraclass correlation coefficient [ICC] = 0.733; 95% CI: 0.611-0.828), and test-retest reliability (clinician ICC = 0.862; 95% CI: 0.767-0.922; self-report ICC = 0.815; 95%CI: 0.741-0.871). CAT-Psychosis could discriminate psychosis from healthy controls (clinician: area under the receiver operating characteristic curve [AUC] = 0.965, 95% CI: 0.945-0.984; self-report AUC = 0.850, 95% CI: 0.807-0.894). The median length of the clinician-administered assessment was 5 minutes (interquartile range [IQR]: 3:23-8:29 min) and 1 minute, 20 seconds (IQR: 0:57-2:09 min) for the self-report.

Conclusion

CAT-Psychosis can quickly and reliably assess the severity of psychosis and discriminate psychotic patients from healthy controls, creating an opportunity for frequent remote assessment and patient/population-level follow-up.

SUBMITTER: Guinart D

PROVIDER: S-EPMC8084426 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:ImportanceVeterans from recent and past conflicts have high rates of posttraumatic stress disorder (PTSD). Adaptive testing strategies can increase accuracy of diagnostic screening and symptom severity measurement while decreasing patient and clinician burden.ObjectiveTo develop and validate a computerized adaptive diagnostic (CAD) screener and computerized adaptive test (CAT) for PTSD symptom severity.Design, setting, and participantsA diagnostic study of measure development and validation was conducted at a Veterans Health Administration facility. A total of 713 US military veterans were included. The study was conducted from April 25, 2017, to November 10, 2019.Main outcomes and measuresThe participants completed a PTSD-symptom questionnaire from the item bank and provided responses on the PTSD Checklist for Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) (PCL-5). A subsample of 304 participants were interviewed using the Clinician-Administered Scale for PTSD for DSM-5.ResultsOf the 713 participants, 585 were men; mean (SD) age was 52.8 (15.0) years. The CAD-PTSD reproduced the Clinician-Administered Scale for PTSD for DSM-5 PTSD diagnosis with high sensitivity and specificity as evidenced by an area under the curve of 0.91 (95% CI, 0.87-0.95). The CAT-PTSD demonstrated convergent validity with the PCL-5 (r = 0.88) and also tracked PTSD diagnosis (area under the curve = 0.85; 95% CI, 0.79-0.89). The CAT-PTSD reproduced the final 203-item bank score with a correlation of r = 0.95 with a mean of only 10 adaptively administered items, a 95% reduction in patient burden.Conclusions and relevanceUsing a maximum of only 6 items, the CAD-PTSD developed in this study was shown to have excellent diagnostic screening accuracy. Similarly, using a mean of 10 items, the CAT-PTSD provided valid severity ratings with excellent convergent validity with an extant scale containing twice the number of items. The 10-item CAT-PTSD also outperformed the 20-item PCL-5 in terms of diagnostic accuracy. The results suggest that scalable, valid, and rapid PTSD diagnostic screening and severity measurement are possible.

Project description:BACKGROUND:Technological advances in healthcare have enabled patients to participate in digital self-assessment, with reported benefits of enhanced healthcare efficiency and self-efficacy. This report describes the design and validation of a patient-administered preanaesthesia health assessment digital application for gathering medical history relevant to preanaesthesia assessment. Effective preoperative evaluation allows for timely optimization of medical conditions and reduces case cancellations on day of surgery. METHODS:Using an iterative mixed-methods approach of literature review, surveys and panel consensus, the study sought to develop and validate a digitized preanaesthesia health assessment questionnaire in terms of face and criterion validity. A total of 228 patients were enrolled at the preoperative evaluation clinic of a tertiary women's hospital. Inclusion criteria include: age ?? 21?years, scheduled for same-day-admission surgery, literacy in English and willingness to use a digital device. Patient perception of the digitized application was also evaluated using the QQ10 questionnaire. Reliability of health assessment questionnaire was evaluated by comparing the percentage agreement of patient responses with nurse assessment. RESULTS:Moderate to good criterion validity was obtained in 81.1 and 83.8% of questions for the paper and digital questionnaires respectively. Of total 3626 response-pairs obtained, there were 3405 (93.4%) concordant and 221 (6.1%) discrepant response-pairs for the digital questionnaire. Discrepant response-pairs, such as ""no/yes" and "unsure/yes", constitute only 3.7% of total response-pairs. Patient acceptability of the digitized assessment was high, with QQ10 value and burden scores of 76 and 30%, respectively. CONCLUSIONS:Self-administration of digitized preanaesthesia health assessment is acceptable to patients and reliable in eliciting medical history. Further iteration should focus on improving reliability of the digital tool, adapting it for use in other languages and incorporating clinical decision tools.

Project description:ImportanceThe rate of suicide among adolescents is rising in the US, yet many adolescents at risk are unidentified and receive no mental health services.ObjectiveTo develop and independently validate a novel computerized adaptive screen for suicidal youth (CASSY) for use as a universal screen for suicide risk in medical emergency departments (EDs).Design, setting, and participantsStudy 1 of this prognostic study prospectively enrolled adolescent patients at 13 geographically diverse US EDs in the Pediatric Emergency Care Applied Research Network. They completed a baseline suicide risk survey and participated in 3-month telephone follow-ups. Using 3 fixed Ask Suicide-Screening Questions items as anchors and additional items that varied in number and content across individuals, we derived algorithms for the CASSY. In study 2, data were collected from patients at 14 Pediatric Emergency Care Applied Research Network EDs and 1 Indian Health Service hospital. Algorithms were independently validated in a prospective cohort of adolescent patients who also participated in 3-month telephone follow-ups. Adolescents aged 12 to 17 years were consecutively approached during randomly assigned shifts.ExposuresPresentation at an ED.Main outcome and measureA suicide attempt between ED visit and 3-month follow-up, measured via patient and/or parent report.ResultsThe study 1 CASSY derivation sample included 2075 adolescents (1307 female adolescents [63.0%]; mean [SD] age, 15.1 [1.61] years) with 3-month follow-ups (72.9% retention [2075 adolescents]). The study 2 validation sample included 2754 adolescents (1711 female adolescents [62.1%]; mean [SD] age, 15.0 [1.65] years), with 3-month follow-ups (69.5% retention [2754 adolescents]). The CASSY algorithms had excellent predictive accuracy for suicide attempt (area under the curve, 0.89 [95% CI, 0.85-0.91]) in study 1. The mean number of adaptively administered items was 11 (range, 5-21). At a specificity of 80%, the CASSY had a sensitivity of 83%. It also demonstrated excellent accuracy in the study 2 validation sample (area under the curve, 0.87 [95% CI, 0.85-0.89]). In this study, the CASSY had a sensitivity of 82.4% for prediction of a suicide attempt at the 80% specificity cutoff established in study 1.Conclusions and relevanceIn this study, the adaptive and personalized CASSY demonstrated excellent suicide attempt risk recognition, which has the potential to facilitate linkage to services.

Project description:Background and aimsThe focus of this paper is on the improvement of substance use disorder (SUD) screening and measurement. Using a multi-dimensional item response theory model, the bifactor model, we provide a psychometric harmonization between SUD, depression, anxiety, trauma, social isolation, functional impairment and risk-taking behavior symptom domains, providing a more balanced view of SUD. The aims are to (1) develop the item-bank, (2) calibrate the item-bank using a bifactor model that includes a primary dimension and symptom-specific subdomains, (3) administer using computerized adaptive testing (CAT) and (4) validate the CAT-SUD in Spanish and English in the United States and Spain.DesignItem bank construction, item calibration phase, CAT-SUD validation phase.SettingPrimary care, community clinics, emergency departments and patient-to-patient referrals in Spain (Barcelona and Madrid) and the United States (Boston and Los Angeles).Participants/casesCalibration phase: the CAT-SUD was developed via simulation from complete item responses in 513 participants. Validation phase: 297 participants received the Composite International Diagnostic Interview (CIDI) and the CAT-SUD.MeasurementsA total of 252 items from five subdomains: (1) SUD, (2) psychological disorders, (3) risky behavior, (4) functional impairment and (5) social support. CAT-SUD scale scores and CIDI SUD diagnosis.FindingsCalibration: the bifactor model provided excellent fit to the multi-dimensional item bank; 168 items had high loadings (> 0.4 with the majority > 0.6) on the primary SUD dimension. Using an average of 11 items (four to 26), which represents a 94% reduction in respondent burden (average administration time of approximately 2 minutes), we found a correlation of 0.91 with the 168-item scale (precision of 5 points on a 100-point scale).Validationstrong agreement was found between the primary CAT-SUD dimension estimate and the results of a structured clinical interview. There was a 20-fold increase in the likelihood of a CIDI SUD diagnosis across the range of the CAT-SUD (AUC = 0.85).ConclusionsWe have developed a new approach for the screening and measurement of SUD and related severity based on multi-dimensional item response theory. The bifactor model harmonized information from mental health, trauma, social support and traditional SUD items to provide a more complete characterization of SUD. The CAT-SUD is highly predictive of a current SUD diagnosis based on a structured clinical interview, and may be predictive of the development of SUD in the future.

Project description:CONTEXT Unlike other areas of medicine, psychiatry is almost entirely dependent on patient report to assess the presence and severity of disease; therefore, it is particularly crucial that we find both more accurate and efficient means of obtaining that report. OBJECTIVE To develop a computerized adaptive test (CAT) for depression, called the Computerized Adaptive Test-Depression Inventory (CAT-DI), that decreases patient and clinician burden and increases measurement precision. DESIGN Case-control study. SETTING A psychiatric clinic and community mental health center. PARTICIPANTS A total of 1614 individuals with and without minor and major depression were recruited for study. MAIN OUTCOME MEASURES The focus of this study was the development of the CAT-DI. The 24-item Hamilton Rating Scale for Depression, Patient Health Questionnaire 9, and the Center for Epidemiologic Studies Depression Scale were used to study the convergent validity of the new measure, and the Structured Clinical Interview for DSM-IV was used to obtain diagnostic classifications of minor and major depressive disorder. RESULTS A mean of 12 items per study participant was required to achieve a 0.3 SE in the depression severity estimate and maintain a correlation of r = 0.95 with the total 389-item test score. Using empirically derived thresholds based on a mixture of normal distributions, we found a sensitivity of 0.92 and a specificity of 0.88 for the classification of major depressive disorder in a sample consisting of depressed patients and healthy controls. Correlations on the order of r = 0.8 were found with the other clinician and self-rating scale scores. The CAT-DI provided excellent discrimination throughout the entire depressive severity continuum (minor and major depression), whereas the traditional scales did so primarily at the extremes (eg, major depression). CONCLUSIONS Traditional measurement fixes the number of items administered and allows measurement uncertainty to vary. In contrast, a CAT fixes measurement uncertainty and allows the number of items to vary. The result is a significant reduction in the number of items needed to measure depression and increased precision of measurement.