Evidence of Construct Validity of Computer-Based Tests for Clinical Reasoning: Instrument Validation Study.
Ontology highlight
ABSTRACT: Background
Clinical reasoning (CR) is a fundamental skill for all medical students. In our medical education system, however, there are shortcomings in the conventional methods of teaching CR. New technology is needed to enhance our CR teaching, especially as we are facing an influx of new health trainees. China Medical University (CMU), in response to this need, has developed a computer-based CR training system (CMU-CBCRT).Objective
We aimed to find evidence of construct validity of the CMU-CBCRT.Methods
We recruited 385 students from fifth year undergraduates to postgraduate year (PGY) 3 to complete the test on CMU-CBCRT. The known-groups technique was used to evaluate the construct validity of the CBCRT by comparing the test scores among 4 training levels (fifth year MD, PGY-1, PGY-2, and PGY-3).Results
We found that test scores increased with years of training. Significant differences were found in the test scores on information collection, diagnosis, and treatment and total scores among different training years of participants. However, significant results were not found for treatment errors.Conclusions
We provided evidence of construct validity of the CMU-CBCRT, which could determine the CR skills of medical students at varying early stage in their careers.
SUBMITTER: Zuo T
PROVIDER: S-EPMC8663660 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Background and aim: 'Reasoning 4 Change' (R4C) is a newly developed instrument, including four domains (D1-D4), to assess clinical practitioners' and students' clinical reasoning with a focus on clients' behaviour change in a physiotherapy context. To establish its use in education and research, its psychometric properties needed to be evaluated. The aim of the study was to generate criterion scores and evaluate the reliability and construct validity of a web-based version of the R4C instrument. Methods: Fourteen physiotherapy experts and 39 final-year physiotherapy students completed the R4C instrument and the Pain Attitudes and Beliefs Scale for Physiotherapists (PABS-PT). Twelve experts and 17 students completed the R4C instrument on a second occasion. The R4C instrument was evaluated with regard to: internal consistency (five subscales of D1); test-retest reliability (D1-D4); inter-rater reliability (D2-D4); and construct validity in terms of convergent validity (D1.4, D2, D4). Criterion scores were generated based on the experts' responses to identify the scores of qualified practitioners' clinical reasoning abilities. Results: For the expert and student samples, the analyses demonstrated satisfactory internal consistency (? range: 0.67-0.91), satisfactory test-retest reliability (ICC range: 0.46-0.94) except for D3 for the experts and D4 for the students. The inter-rater reliability demonstrated excellent agreement within the expert group (ICC range: 0.94-1.0). The correlations between the R4C instrument and PABS-PT (r range: 0.06-0.76) supported acceptable construct validity. Conclusions: The web-based R4C instrument shows satisfactory psychometric properties and could be useful in education and research. The use of the instrument may contribute to a deeper understanding of physiotherapists' and students' clinical reasoning, valuable for curriculum development and improvements of competencies in clinical reasoning related to clients' behavioural change.
| S-EPMC6141557 | biostudies-literature
Project description:PurposeDeveloping medical students' clinical reasoning requires a structured longitudinal curriculum with frequent targeted assessment and feedback. Performance-based assessments, which have the strongest validity evidence, are currently not feasible for this purpose because they are time-intensive to score. This study explored the potential of using machine learning technologies to score one such assessment-the diagnostic justification essay.MethodFrom May to September 2018, machine scoring algorithms were trained to score a sample of 700 diagnostic justification essays written by 414 third-year medical students from the Southern Illinois University School of Medicine classes of 2012-2017. The algorithms applied semantically based natural language processing metrics (e.g., coherence, readability) to assess essay quality on 4 criteria (differential diagnosis, recognition and use of findings, workup, and thought process); the scores for these criteria were summed to create overall scores. Three sources of validity evidence (response process, internal structure, and association with other variables) were examined.ResultsMachine scores correlated more strongly with faculty ratings than faculty ratings did with each other (machine: .28-.53, faculty: .13-.33) and were less case-specific. Machine scores and faculty ratings were similarly correlated with medical knowledge, clinical cognition, and prior diagnostic justification. Machine scores were more strongly associated with clinical communication than were faculty ratings (.43 vs .31).ConclusionsMachine learning technologies may be useful for assessing medical students' long-form written clinical reasoning. Semantically based machine scoring may capture the communicative aspects of clinical reasoning better than faculty ratings, offering the potential for automated assessment that generalizes to the workplace. These results underscore the potential of machine scoring to capture an aspect of clinical reasoning performance that is difficult to assess with traditional analytic scoring methods. Additional research should investigate machine scoring generalizability and examine its acceptability to trainees and educators.
| S-EPMC8243833 | biostudies-literature
Project description:BackgroundCurrently, there are no outcome measures assessing the ability of people with non-specific low back pain to self-manage their illness. Inspired by the 'Patient Enablement Instrument', we developed the Patient Enablement Instrument for Back Pain (PEI-BP). The aim of this study was to describe the development of the Patient Enablement Instrument for Back Pain (PEI-BP) and investigate content validity, construct validity, internal consistency, test-retest reliability, measurement error, responsiveness and floor and ceiling effects.MethodsThe PEI-BP consists of 6 items that are rated on a 0-10 Numeric Rating Scale. Measurement properties were evaluated using the COSMIN taxonomy and were based on three cohorts from primary care with low back pain: The content validity cohort (N = 14) which participated in semi-structured interviews, the GLA:D® Back cohort (N = 272) and the test-retest cohort (N = 37) which both completed self-reported questionnaires. For construct validity and responsiveness, enablement was compared to disability (Oswestry Disability Index), back pain beliefs (Brief Illness Perception Questionnaire), fear avoidance (Fear-Avoidance Beliefs Questionnaire-physical activity), mental health (SF-36), educational level and number of previous episodes of low back pain.ResultsThe PEI-BP was found to have acceptable content validity, construct validity, reliability (internal consistency, test-retest reliability and measurement error) and responsiveness. The Smallest Detectable Change was 10.1 points illustrating that a patient would have to change more than 1/6 of the scale range for it to be a true change. A skewed distribution towards the high scores were found at baseline indicating a potentially problematic ceiling effect in the current population.ConclusionsThe PEI-BP can be considered a valid and reliable tool to measure enablement on people seeking care for non-specific LBP. Further testing of the PEI-BP in populations with more severe LBP is recommended.Trial registrationNot applicable.
| S-EPMC8033700 | biostudies-literature
Project description:BACKGROUND:Relative to their application with adults there is currently little information about the application of preference-based health-related quality of life (HRQL) instruments among populations of young people. The Child Health Utility 9D (CHU9D) is a paediatric-specific generic preference-based HRQL instrument, recently translated and linguistically validated into Danish (CHU9D-DK). The purpose of this study was to investigate the construct validity of the CHU9D-DK in a sample of Danish high school students. METHODS:All students attending a Danish High School were invited to participate in a web-based survey in January 2018 (N?=?272). The survey included the CHU9D-DK, the young adult version of the Pediatric Quality of Life Inventory™ 4.0 Generic Core Scales (PedsQL), self-reported health status, presence/absence of disability/chronic diseases, life satisfaction, and socio-economic questions. CHU9D-DK utility scores were generated by employing the two scoring algorithms developed from adults in the UK and adolescents in Australia, respectively. Internal consistency, reliability and construct validity of the CHU9D-DK instrument were investigated. RESULTS:Two hundred and twenty-eight (84%) students consented to participate and completed the survey. The mean?±?(standard deviation) values of the CHU9D-DK utilities were 0.84 (0.11) when the UK adult algorithm was applied and 0.70 (0.22), when the Australian adolescent algorithm was applied. The mean PedsQL score was 82.32 (13.14). The CHU9D-DK showed good internal consistency reliability (Cronbach's alpha?=?0.803). Higher levels of health status and life satisfaction were significantly associated with higher CHU9D-DK utility scores regardless of which scoring algorithm was applied (p-values <?0.001). Students living with a disability/chronic disease exhibited significantly lower utility scores relative to their healthy peers (p-values <?0.05). Higher socio-economic status (approximated by financial situation and frequency of family vacations) was also associated with higher utility scores (p-values <?0.005). CONCLUSION:The CHU9D-DK demonstrated good psychometric performance overall and shows potential as a valid and reliable instrument for assessing the HRQL of Danish young people. TRIAL REGISTRATION:ClinicalTrials.gov identifier: NCT03391999, Registered 15 October 2017.
| S-EPMC6929354 | biostudies-literature
Project description:The purpose of this study was to validate the construct and reliability of an instrument to assess the work environment as a single tool based on quality of life (QL), quality of work life (QWL), and organizational climate (OC). The methodology tested the construct validity through Exploratory Factor Analysis (EFA) and reliability through Cronbach's alpha. The EFA returned a Kaiser-Meyer-Olkin (KMO) value of 0.917; which demonstrated that the data were adequate for the factor analysis; and a significant Bartlett's test of sphericity (χ² = 7465.349; Df = 1225; p ≤ 0.000). After the EFA; the varimax rotation method was employed for a factor through commonality analysis; reducing the 14 initial factors to 10. Only question 30 presented commonality lower than 0.5; and the other questions returned values higher than 0.5 in the commonality analysis. Regarding the reliability of the instrument; all of the questions presented reliability as the values varied between 0.953 and 0.956. Thus; the instrument demonstrated construct validity and reliability.
| S-EPMC7600521 | biostudies-literature
Project description:Variability in sensitivity to the acute effects of alcohol is an important risk factor for the development of alcohol use disorder (AUD). The most commonly used retrospective self-report measure of sensitivity, the Self-Rating of the Effects of Alcohol (SRE) form, queries a limited number of alcohol effects and relies on respondents' ability to recall experiences that might have occurred in the distant past. Here, we investigated the construct validity of an alternative measure that queries a larger number of alcohol effects, the Alcohol Sensitivity Questionnaire (ASQ), and compared it to the SRE in predicting momentary subjective responses to an acute dose of alcohol.Healthy young adults (N = 423) completed the SRE and the ASQ and then were randomly assigned to consume either alcohol or a placebo beverage (between-subjects manipulation). Stimulation and sedation (Biphasic Alcohol Effects Scale) and subjective intoxication were measured multiple times after drinking.Hierarchical linear models showed that the ASQ reliably predicted each of these outcomes following alcohol but not placebo consumption, provided unique prediction beyond that associated with differences in recent alcohol involvement, and was preferred over the SRE (in terms of model fit) in direct model comparisons of stimulation and sedation.The ASQ compared favorably with the better-known SRE in predicting increased stimulation and reduced sedation following an acute alcohol challenge. The ASQ appears to be a valid self-report measure of alcohol sensitivity and therefore holds promise for identifying individuals at-risk for AUD and related problems.
| S-EPMC4820365 | biostudies-literature
Project description:BackgroundIt is not known if construct-related validity (progression of scores with different levels of training) and generalizability of Objective Structured Clinical Examination (OSCE) scenarios previously used with non-US graduating anesthesiology residents translate to a US training program.ObjectiveWe assessed for progression of scores with training for a validated high-stakes simulation-based anesthesiology examination.MethodsFifty US anesthesiology residents in postgraduate years (PGYs) 2 to 4 were evaluated in operating room, trauma, and resuscitation scenarios developed for and used in a high-stakes Israeli Anesthesiology Board examination, requiring a score of 70% on the checklist for passing (including all critical items).ResultsThe OSCE error rate was lower for PGY-4 than PGY-2 residents in each field, and for most scenarios within each field. The critical item error rate was significantly lower for PGY-4 than PGY-3 residents in operating room scenarios, and for PGY-4 than PGY-2 residents in resuscitation scenarios. The final pass rate was significantly higher for PGY-3 and PGY-4 than PGY-2 residents in operating room scenarios, and also was significantly higher for PGY-4 than PGY-2 residents overall. PGY-4 residents had a better error rate, total scenarios score, general evaluation score, critical items error rate, and final pass rate than PGY-2 residents.ConclusionsThe comparable error rates, performance grades, and pass rates for US PGY-4 and non-US (Israeli) graduating (PGY-4 equivalent) residents, and the progression of scores among US residents with training level, demonstrate the construct-related validity and generalizability of these high-stakes OSCE scenarios.
| S-EPMC4535213 | biostudies-literature
Project description:The screening of dementia in non-Westerners has so far relied on translations and adaptations of reputed instruments. Other efforts focused on developing culturally appropriate tests or tests in touch with new developments in the field. This study presents the rationale behind the construction of a new dementia screening test: the Dementia Screening Battery-100 (DSB-100).The DSB-100 was administered to 46 demented individuals and 159 healthy matched controls. All demented participants met the DSM-IV criteria for dementia. The healthy controls showed no cognitive impairment and were independent in activities of daily living. The DSB-100 was administered as part of a larger neuropsychological assessment to collect additional indices on the severity of patients' dementia, depression, and frontal dysfunctions. The same information was used for comparisons with DSB-100 scores.Multiple regression analysis suggested that age and education, but not the variable sex, are essential in predicting cognitive performance. Construct validation yielded 4 factors, namely attention-visuospatial factors, memory, language, and executive functions. The results showed that the DSB-100 has a high interrater reliability and an acceptable overall internal homogeneity.These results validate the DSB-100 and suggest its appropriateness for dementia screening in Tunisian elderly and possibly elderly people from other cultures with modifications to some subscales.
| S-EPMC5498961 | biostudies-other
Project description:BackgroundAlthough there is emerging data regarding the psychometric properties of the Child Health Utility-9D instrument, more evidence is required with respect to its validity for use in different country settings. The aim of this study was to examine the construct validity of the CHU-9D-CHN instrument in Chinese children.MethodsBaseline Health-Related Quality of Life (HRQoL) and demographic data were collected from children recruited to the CHIRPY DRAGON obesity prevention intervention randomised controlled trial in China. HRQoL was measured using the Chinese version of the CHU-9D instrument (CHU-9D-CHN) and the PedsQL instrument. CHU-9D-CHN utility scores were generated using two scoring algorithms [UK and Chinese tariffs]. Discriminant validity, known-group validity and convergent validity were evaluated using non-parametric test for trend, Kruskal-Wallis test and Spearman correlation coefficient analysis respectively.ResultsData was available for 1,539 children (mean age 6 years). The CHU-9D-CHN was sensitive to known group differences determined by the median PedsQL total score. Furthermore, the mean CHU-9D-CHN utility values decreased linearly with increasing levels of severity on each dimension of the PedsQL for emotional and social functioning domains. They decreased monotonically with increasing levels of severity on each dimension of the PedsQL for physical and school functioning domains (p < 0.001). Contrary to studies conducted in Western countries, and although not statistically significant, we found an indication that HRQoL, using both the CHU-9D-CHN and the PedsQL, was higher in children whose parents had lower levels of education, compared to those whose parents were university educated. The correlation between the CHU-9D-CHN utility values using UK and Chinese tariffs, and PedsQL total scores showed a statistically significant moderate positive correlation (Spearman's rho = 0.5221, p < 0.001 and Spearman's rho = 0.5316, p < 0.001), respectively. However, each CHU-9D-CHN dimension was either weakly, or very weakly correlated with each of the predetermined PedsQL domain functioning scores.ConclusionsOverall, the findings provide some support for the construct validity of the CHU-9D-CHN within a Chinese population aged 6-7 years. However, some uncertainty remains. We recommend future studies continue to test the validity of the CHU-9D in different country settings.Trial registrationISRCTN Identifier ISRCTN11867516, Registered on 19/08/2015.
| S-EPMC8394170 | biostudies-literature
Project description:ObjectivesTo identify the factors associated with medical students' clinical reasoning (CR) use and evidence-based medicine (EBM) use in the clinical setting.MethodsOur cross-sectional study surveyed 44 final-year medical students at an emerging academic medical center in Singapore. We queried the students' EBM and CR value and experiences in the classroom and clinical settings. We compared this to their perceptions of supervisors' value and experiences using t-tests. We developed measures of teaching culture and practice culture by combining relevant questions into summary scores. Multivariate linear regression models were applied to identify factors associated with the students' CR and EBM clinical use.ResultsEighty-nine percent of students responded (n=39). Students reported valuing CR (p=0.03) and EBM (p=0.001) more than their supervisors, but practiced these skills similarly (p=0.83; p=0.82). Clinical practice culture and classroom CR experience were independently associated with students' CR clinical use (p=0.05; p=0.04), and classroom EBM experience was independently associated with students' EBM clinical use (p=0.03). Clinical teaching culture was not associated with students' CR and EBM clinical use.ConclusionsOur study found that medical students' classroom experience and the clinical practice culture influenced their CR and EBM use. The clinical teaching culture did not. These findings suggest that in order to increase student CR and EBM use, in addition to providing classroom experience, medical educators may need to change the hospital culture by encouraging supervisors to use these skills in their clinical practice.
| S-EPMC4646359 | biostudies-literature