Dataset Information

Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: meta-regression of six UK longitudinal studies.

ABSTRACT:

Background

Measures used for medical student selection should predict future performance during training. A problem for any selection study is that predictor-outcome correlations are known only in those who have been selected, whereas selectors need to know how measures would predict in the entire pool of applicants. That problem of interpretation can be solved by calculating construct-level predictive validity, an estimate of true predictor-outcome correlation across the range of applicant abilities.

Methods

Construct-level predictive validities were calculated in six cohort studies of medical student selection and training (student entry, 1972 to 2009) for a range of predictors, including A-levels, General Certificates of Secondary Education (GCSEs)/O-levels, and aptitude tests (AH5 and UK Clinical Aptitude Test (UKCAT)). Outcomes included undergraduate basic medical science and finals assessments, as well as postgraduate measures of Membership of the Royal Colleges of Physicians of the United Kingdom (MRCP(UK)) performance and entry in the Specialist Register. Construct-level predictive validity was calculated with the method of Hunter, Schmidt and Le (2006), adapted to correct for right-censorship of examination results due to grade inflation.

Results

Meta-regression analyzed 57 separate predictor-outcome correlations (POCs) and construct-level predictive validities (CLPVs). Mean CLPVs are substantially higher (.450) than mean POCs (.171). Mean CLPVs for first-year examinations, were high for A-levels (.809; CI: .501 to .935), and lower for GCSEs/O-levels (.332; CI: .024 to .583) and UKCAT (mean = .245; CI: .207 to .276). A-levels had higher CLPVs for all undergraduate and postgraduate assessments than did GCSEs/O-levels and intellectual aptitude tests. CLPVs of educational attainment measures decline somewhat during training, but continue to predict postgraduate performance. Intellectual aptitude tests have lower CLPVs than A-levels or GCSEs/O-levels.

Conclusions

Educational attainment has strong CLPVs for undergraduate and postgraduate performance, accounting for perhaps 65% of true variance in first year performance. Such CLPVs justify the use of educational attainment measure in selection, but also raise a key theoretical question concerning the remaining 35% of variance (and measurement error, range restriction and right-censorship have been taken into account). Just as in astrophysics, 'dark matter' and 'dark energy' are posited to balance various theoretical equations, so medical student selection must also have its 'dark variance', whose nature is not yet properly characterized, but explains a third of the variation in performance during training. Some variance probably relates to factors which are unpredictable at selection, such as illness or other life events, but some is probably also associated with factors such as personality, motivation or study skills.

SUBMITTER: McManus IC

PROVIDER: S-EPMC3827328 | biostudies-literature | 2013 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: meta-regression of six UK longitudinal studies.

McManus I C IC Dewberry Chris C Nicholson Sandra S Dowell Jonathan S JS Woolf Katherine K Potts Henry W W HW

BMC medicine 20131114

<h4>Background</h4>Measures used for medical student selection should predict future performance during training. A problem for any selection study is that predictor-outcome correlations are known only in those who have been selected, whereas selectors need to know how measures would predict in the entire pool of applicants. That problem of interpretation can be solved by calculating construct-level predictive validity, an estimate of true predictor-outcome correlation across the range of applic ...[more]

PMID: 24229353

Similar Datasets

Project description:BackgroundStudent engagement is student investment of time and energy in academic and non-academic experiences that include learning, teaching, research, governance, and community activities. Although previous studies provided some evidence of measuring student engagement in PBL tutorials, there are no existing quantitative studies in which cognitive, behavioral, and emotional engagement of students in PBL tutorials is measured. Therefore, this study aims to develop and examine the construct validity of a questionnaire for measuring cognitive, behavioral, and emotional engagement of students in PBL tutorials.MethodsA 15-item questionnaire was developed guided by a previously published conceptual framework of student engagement. Focus group discussion (n = 12) with medical education experts was then conducted and the questionnaire was piloted with medical students. The questionnaire was then distributed to year 2 and 3 medical students (n = 176) in problem-based tutorial groups at the end of an integrated course, where PBL is the main strategy of learning. The validity of the internal structure of the questionnaire was tested by confirmatory factor analysis using structural equation modeling assuming five different models. Predictive validity evidence of the questionnaire was studied by examining the correlations between students' engagement and academic achievement.ResultsConfirmatory factor analysis indicates a good fit between the measurement and structural model of an 11-item questionnaire composed of a three-factor structure: behavioral engagement (3 items), emotional engagement (4 items), and cognitive engagement (4 items). Models in which the three latent factors were considered semi-independent provided the best fit. The construct reliabilities of behavioral, cognitive, and emotional factors were 0.82, 0.82, and 0.76, respectively. We failed however to find significant relationships between academic achievement and engagement.ConclusionsWe found a strong evidence to support the construct validity of a three-factor structure of student engagement in PBL tutorial questionnaire. Further studies are required to test the validity of this instrument in other educational settings. The predictive validity is another area needing further scrutiny.

Project description:BackgroundIndividuals with ADHD are at increased risk for poor occupational outcomes. Educational attainment and psychiatric comorbidity may be important contributing factors for these outcomes, but the role of these factors is not well characterized. This study aimed to investigate the associations between ADHD and occupational outcomes, and to examine the influence of educational attainment, comorbid developmental disorders and intellectual disability on these associations.MethodsWe linked the Swedish population graduating from compulsory school 1998-2008 (N = 1.2 millions) to population-wide register-based data on clinical psychiatric diagnoses and medications, objective annual measures of educational, and occupational outcomes. Individuals were followed for between 6 to 16 years after graduation.ResultsIndividuals with ADHD had annually on average 17 percent lower income, ratio = 0.83 (95% CI 0.83-0.84), 12.19 (11.89-12.49) more days of unemployment, and a higher likelihood of receiving disability pension, odds-ratio = 19.0 (18.4-19.6), compared to controls. Comorbid diagnoses of intellectual disability and developmental disorder explained most of the association between ADHD and disability pension, while lifetime educational attainment partially explained associations between ADHD and all occupational outcomes. Analyses of occupational trajectories found that income was lower and unemployment elevated relative to controls with the same educational attainment. Higher educational attainment correlated with higher income similarly among individuals with ADHD and controls after accounting for individual background factors.ConclusionsThe occupational burden associated with ADHD is substantial. Comorbid developmental disorders, intellectual disability and educational difficulties (e.g., failing grades) from childhood to adulthood are important factors to consider when designing interventions to improve occupational outcomes in individuals with ADHD.

Dataset Information

Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: meta-regression of six UK longitudinal studies.

Background

Methods

Results

Conclusions

Publications

Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: meta-regression of six UK longitudinal studies.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets