Dataset Information

Testing the validity of value-added measures of educational progress with genetic data.

ABSTRACT: Value-added measures of educational progress have been used by education researchers and policy-makers to assess the performance of teachers and schools, contributing to performance-related pay and position in school league tables. They are designed to control for all underlying differences between pupils and should therefore provide unbiased measures of school and teacher influence on pupil progress, however, their effectiveness has been questioned. We exploit genetic data from a UK birth cohort to investigate how successfully value-added measures control for genetic differences between pupils. We use raw value-added, contextual value-added (which additionally controls for background characteristics) and teacher-reported value-added measures built from data at ages 11, 14 and 16. Sample sizes for analyses range from 4,600 to 6,518. Our findings demonstrate that genetic differences between pupils explain little variation in raw value-added measures but explain up to 20% of the variation in contextual value-added measures (95% CI = 6.06% to 35.71%). Value-added measures built from teacher-rated ability have a greater proportion of variance explained by genetic differences between pupils, with 36.3% of their cross-sectional variation being statistically accounted for by genetics (95% CI = 22.8% to 49.8%). By contrast, a far greater proportion of variance is explained by genetic differences for raw test scores at each age of at least 47.3% (95% CI: 35.9 to 58.7). These findings provide evidence that value-added measures of educational progress can be influenced by genetic differences between pupils, and therefore may provide a biased measure of school and teacher performance. We include a glossary of genetic terms for educational researchers interested in the use of genetic data in educational research.

SUBMITTER: Morris TT

PROVIDER: S-EPMC6448053 | biostudies-literature | 2018 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Testing the validity of value-added measures of educational progress with genetic data.

Morris Tim T TT Davies Neil M NM Dorling Danny D Richmond Rebecca C RC Smith George Davey GD

British educational research journal 20180909 5

Value-added measures of educational progress have been used by education researchers and policy-makers to assess the performance of teachers and schools, contributing to performance-related pay and position in school league tables. They are designed to control for all underlying differences between pupils and should therefore provide unbiased measures of school and teacher influence on pupil progress, however, their effectiveness has been questioned. We exploit genetic data from a UK birth cohor ...[more]

PMID: 30983649

Similar Datasets

Project description:There is no consensus on which statistical model estimates school value-added (VA) most accurately. To date, the two most common statistical models used for the calculation of VA scores are two classical methods: linear regression and multilevel models. These models have the advantage of being relatively transparent and thus understandable for most researchers and practitioners. However, these statistical models are bound to certain assumptions (e.g., linearity) that might limit their prediction accuracy. Machine learning methods, which have yielded spectacular results in numerous fields, may be a valuable alternative to these classical models. Although big data is not new in general, it is relatively new in the realm of social sciences and education. New types of data require new data analytical approaches. Such techniques have already evolved in fields with a long tradition in crunching big data (e.g., gene technology). The objective of the present paper is to competently apply these "imported" techniques to education data, more precisely VA scores, and assess when and how they can extend or replace the classical psychometrics toolbox. The different models include linear and non-linear methods and extend classical models with the most commonly used machine learning methods (i.e., random forest, neural networks, support vector machines, and boosting). We used representative data of 3,026 students in 153 schools who took part in the standardized achievement tests of the Luxembourg School Monitoring Program in grades 1 and 3. Multilevel models outperformed classical linear and polynomial regressions, as well as different machine learning models. However, it could be observed that across all schools, school VA scores from different model types correlated highly. Yet, the percentage of disagreements as compared to multilevel models was not trivial and real-life implications for individual schools may still be dramatic depending on the model type used. Implications of these results and possible ethical concerns regarding the use of machine learning methods for decision-making in education are discussed.

Project description:BackgroundIn present-day life-insurance medical underwriting practice the risk assessment starts with a standard health declaration (SHD). Indication for additional medical screening depends predominantly on age and amount of insured capital. From a medical perspective it is questionable whether there is an association between the level of insured capital and medical risk in terms of mortality. The aim of the study is to examine the prognostic value of parameters from the health declaration and application form on extra mortality based on results from additional medical testing.MethodsA history register-based cohort study was conducted including about 15.000 application files accepted between 2007 and 2010. Blood pressure, lipids, cotinine and glucose levels were used as dependent variables in logistic regression models. Resampling validation was applied using 250 bootstrap samples to calculate area under the curves (AUC's). The AUC was used to discriminate between persons with and without at least 25% extra mortality.ResultsBMI and the overall assessment of the health declaration by an insurance physician or medical underwriter showed the strongest discrimination in multivariable analysis. Including all variables at minimum cut-off levels resulted in an AUC of 0.710 while by using a model with BMI, the assessment of the health declaration and gender, the AUC was 0.708. Including all variables at maximum cut-off levels lead to an AUC of 0.743 while a model with BMI, the assessment of the health declaration and age resulted in an AUC of 0.741.ConclusionsThe outcome of this study shows that BMI and the overall assessment of the health declaration were the dominant variables to discriminate between applicants for life-insurance with and without at least 25 percent extra mortality. The variable insured capital set by insurers as factor for additional medical testing could not be established in this study population. The indication for additional medical testing at underwriting life-insurance can possibly be done on limited variables instead of the obligatory medical testing based on age and the amount of insured capital.

Project description:Objective(i) To develop the Pharmacy Value-Added Services Questionnaire (PVASQ) using emerging themes generated from interviews. (ii) To establish reliability and validity of questionnaire instrument.MethodsUsing an extended Theory of Planned Behavior as the theoretical model, face-to-face interviews generated salient beliefs of pharmacy value-added services. The PVASQ was constructed initially in English incorporating important themes and later translated into the Malay language with forward and backward translation. Intention (INT) to adopt pharmacy value-added services is predicted by attitudes (ATT), subjective norms (SN), perceived behavioral control (PBC), knowledge and expectations. Using a 7-point Likert-type scale and a dichotomous scale, test-retest reliability (N=25) was assessed by administrating the questionnaire instrument twice at an interval of one week apart. Internal consistency was measured by Cronbach's alpha and construct validity between two administrations was assessed using the kappa statistic and the intraclass correlation coefficient (ICC). Confirmatory Factor Analysis, CFA (N=410) was conducted to assess construct validity of the PVASQ.ResultsThe kappa coefficients indicate a moderate to almost perfect strength of agreement between test and retest. The ICC for all scales tested for intra-rater (test-retest) reliability was good. The overall Cronbach' s alpha (N=25) is 0.912 and 0.908 for the two time points. The result of CFA (N=410) showed most items loaded strongly and correctly into corresponding factors. Only one item was eliminated.ConclusionsThis study is the first to develop and establish the reliability and validity of the Pharmacy Value-Added Services Questionnaire instrument using the Theory of Planned Behavior as the theoretical model. The translated Malay language version of PVASQ is reliable and valid to predict Malaysian patients' intention to adopt pharmacy value-added services to collect partial medicine supply.

Project description:BackgroundPharmacovigilance consists in monitoring and preventing the occurrence of adverse drug reactions (ADR). This activity requires the collection and analysis of data from the patient record or any other sources to find clues of a causality link between the drug and the ADR. This can be time-consuming because often patient data are heterogeneous and scattered in several files. To facilitate this task, we developed a timeline prototype to gather and classify patient data according to their chronology. Here, we evaluated its usability and quantified its contribution to routine pharmacovigilance using real ADR cases.MethodsThe timeline prototype was assessed using the biomedical data warehouse eHOP (from entrepôt de données biomédicales de l'HOPital) of the Rennes University Hospital Centre. First, the prototype usability was tested by six experts of the Regional Pharmacovigilance Centre of Rennes. Their experience was assessed with the MORAE software and a System and Usability Scale (SUS) questionnaire. Then, to quantify the timeline contribution to pharmacovigilance routine practice, three of them were asked to investigate possible ADR cases with the "Usual method" (analysis of electronic health record data with the DxCare software) or the "Timeline method". The time to complete the task and the data quality in their reports (using the vigiGrade Completeness score) were recorded and compared between methods.ResultsAll participants completed their tasks. The usability could be considered almost excellent with an average SUS score of 82.5/100. The time to complete the assessment was comparable between methods (P = 0.38) as well as the average vigiGrade Completeness of the data collected with the two methods (P = 0.49).ConclusionsThe results showed a good general level of usability for the timeline prototype. Conversely, no difference in terms of the time spent on each ADR case and data quality was found compared with the usual method. However, this absence of difference between the timeline and the usual tools that have been in use for several years suggests a potential use in pharmacovigilance especially because the testers asked to continue using the timeline after the evaluation.

Project description:BackgroundModels that predict postoperative complications often ignore important intraoperative events and physiological changes. This study tested the hypothesis that accuracy, discrimination, and precision in predicting postoperative complications would improve when using both preoperative and intraoperative data input data compared with preoperative data alone.MethodsThis retrospective cohort analysis included 43,943 adults undergoing 52,529 inpatient surgeries at a single institution during a 5-y period. Random forest machine learning models in the validated MySurgeryRisk platform made patient-level predictions for seven postoperative complications and mortality occurring during hospital admission using electronic health record data and patient neighborhood characteristics. For each outcome, one model trained with preoperative data alone; one model trained with both preoperative and intraoperative data. Models were compared by accuracy, discrimination (expressed as area under the receiver operating characteristic curve), precision (expressed as area under the precision-recall curve), and reclassification indices.ResultsMachine learning models incorporating both preoperative and intraoperative data had greater accuracy, discrimination, and precision than models using preoperative data alone for predicting all seven postoperative complications (intensive care unit length of stay >48 h, mechanical ventilation >48 h, neurologic complications including delirium, cardiovascular complications, acute kidney injury, venous thromboembolism, and wound complications), and in-hospital mortality (accuracy: 88% versus 77%; area under the receiver operating characteristic curve: 0.93 versus 0.87; area under the precision-recall curve: 0.21 versus 0.15). Overall reclassification improvement was 2.4%-10.0% for complications and 11.2% for in-hospital mortality.ConclusionsIncorporating both preoperative and intraoperative data significantly increased the accuracy, discrimination, and precision of machine learning models predicting postoperative complications and mortality.

Dataset Information

Testing the validity of value-added measures of educational progress with genetic data.

Publications

Testing the validity of value-added measures of educational progress with genetic data.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets