Dataset Information

Assessing methods for generalizing experimental impact estimates to target populations.

ABSTRACT: Randomized experiments are considered the gold standard for causal inference, as they can provide unbiased estimates of treatment effects for the experimental participants. However, researchers and policymakers are often interested in using a specific experiment to inform decisions about other target populations. In education research, increasing attention is being paid to the potential lack of generalizability of randomized experiments, as the experimental participants may be unrepresentative of the target population of interest. This paper examines whether generalization may be assisted by statistical methods that adjust for observed differences between the experimental participants and members of a target population. The methods examined include approaches that reweight the experimental data so that participants more closely resemble the target population and methods that utilize models of the outcome. Two simulation studies and one empirical analysis investigate and compare the methods' performance. One simulation uses purely simulated data while the other utilizes data from an evaluation of a school-based dropout prevention program. Our simulations suggest that machine learning methods outperform regression-based methods when the required structural (ignorability) assumptions are satisfied. When these assumptions are violated, all of the methods examined perform poorly. Our empirical analysis uses data from a multi-site experiment to assess how well results from a given site predict impacts in other sites. Using a variety of extrapolation methods, predicted effects for each site are compared to actual benchmarks. Flexible modeling approaches perform best, although linear regression is not far behind. Taken together, these results suggest that flexible modeling techniques can aid generalization while underscoring the fact that even state-of-the-art statistical techniques still rely on strong assumptions.

SUBMITTER: Kern HL

PROVIDER: S-EPMC5030077 | biostudies-literature | 2016

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Assessing methods for generalizing experimental impact estimates to target populations.

Kern Holger L HL Stuart Elizabeth A EA Hill Jennifer J Green Donald P DP

Journal of research on educational effectiveness 20160114 1

Randomized experiments are considered the gold standard for causal inference, as they can provide unbiased estimates of treatment effects for the experimental participants. However, researchers and policymakers are often interested in using a specific experiment to inform decisions about other target populations. In education research, increasing attention is being paid to the potential lack of generalizability of randomized experiments, as the experimental participants may be unrepresentative o ...[more]

PMID: 27668031

Similar Datasets

Project description:Recent years have seen increasing interest in and attention to evidence-based practices, where the "evidence" generally comes from well-conducted randomized trials. However, while those trials yield accurate estimates of the effect of the intervention for the participants in the trial (known as "internal validity"), they do not always yield relevant information about the effects in a particular target population (known as "external validity"). This may be due to a lack of specification of a target population when designing the trial, difficulties recruiting a sample that is representative of a prespecified target population, or to interest in considering a target population somewhat different from the population directly targeted by the trial. This paper first provides an overview of existing design and analysis methods for assessing and enhancing the ability of a randomized trial to estimate treatment effects in a target population. It then provides a case study using one particular method, which weights the subjects in a randomized trial to match the population on a set of observed characteristics. The case study uses data from a randomized trial of school-wide positive behavioral interventions and supports (PBIS); our interest is in generalizing the results to the state of Maryland. In the case of PBIS, after weighting, estimated effects in the target population were similar to those observed in the randomized trial. The paper illustrates that statistical methods can be used to assess and enhance the external validity of randomized trials, making the results more applicable to policy and clinical questions. However, there are also many open research questions; future research should focus on questions of treatment effect heterogeneity and further developing these methods for enhancing external validity. Researchers should think carefully about the external validity of randomized trials and be cautious about extrapolating results to specific populations unless they are confident of the similarity between the trial sample and that target population.

Project description:BackgroundGenome annotation is a crucial component of RNA-seq data analysis. Much effort has been devoted to producing an accurate and rational annotation of the human genome. An annotated genome provides a comprehensive catalogue of genomic functional elements. Currently, at least six human genome annotations are publicly available, including AceView Genes, Ensembl Genes, H-InvDB Genes, RefSeq Genes, UCSC Known Genes, and Vega Genes. Characteristics of these annotations differ because of variations in annotation strategies and information sources. When performing RNA-seq data analysis, researchers need to choose a genome annotation. However, the effect of genome annotation choice on downstream RNA-seq expression estimates is still unclear. This study (1) investigates the effect of different genome annotations on RNA-seq quantification and (2) provides guidelines for choosing a genome annotation based on research focus.ResultsWe define the complexity of human genome annotations in terms of the number of genes, isoforms, and exons. This definition facilitates an investigation of potential relationships between complexity and variations in RNA-seq quantification. We apply several evaluation metrics to demonstrate the impact of genome annotation choice on RNA-seq expression estimates. In the mapping stage, the least complex genome annotation, RefSeq Genes, appears to have the highest percentage of uniquely mapped short sequence reads. In the quantification stage, RefSeq Genes results in the most stable expression estimates in terms of the average coefficient of variation over all genes. Stable expression estimates in the quantification stage translate to accurate statistics for detecting differentially expressed genes. We observe that RefSeq Genes produces the most accurate fold-change measures with respect to a ground truth of RT-qPCR gene expression estimates.ConclusionsBased on the observed variations in the mapping, quantification, and differential expression calling stages, we demonstrate that the selection of human genome annotation results in different gene expression estimates. When conducting research that emphasizes reproducible and robust gene expression estimates, a less complex genome annotation may be preferred. However, simpler genome annotations may limit opportunities for identifying or characterizing novel transcriptional or regulatory mechanisms. When conducting research that aims to be more exploratory, a more complex genome annotation may be preferred.

Project description:Camera traps are used increasingly to estimate population density for elusive and difficult to observe species. A standard practice for mammalian surveys is to place cameras on roads, trails, and paths to maximize detections and/or increase efficiency in the field. However, for many species it is unclear whether track-based camera surveys provide reliable estimates of population density.Understanding how the spatial arrangement of camera traps affects population density estimates is of key interest to contemporary conservationists and managers given the rapid increase in camera-based wildlife surveys.We evaluated the effect of camera-trap placement, using several survey designs, on density estimates of a widespread mesopredator, the red fox Vulpes vulpes, over a two-year period in a semi-arid conservation reserve in south-eastern Australia. Further, we used the certainty in the identity and whereabouts of individuals (via GPS collars) to assess how resighting rates of marked foxes affect density estimates using maximum likelihood spatially explicit mark-resight methods.Fox detection rates were much higher at cameras placed on tracks compared with off-track cameras, yet in the majority of sessions, camera placement had relatively little effect on point estimates of density. However, for each survey design, the precision of density estimates varied considerably across sessions, influenced heavily by the absolute number of marked foxes detected, the number of times marked foxes was resighted, and the number of detection events of unmarked foxes.Our research demonstrates that the precision of population density estimates using spatially explicit mark-resight models is sensitive to resighting rates of identifiable individuals. Nonetheless, camera surveys based either on- or off-track can provide reliable estimates of population density using spatially explicit mark-resight models. This underscores the importance of incorporating information on the spatial behavior of the subject species when planning camera-trap surveys.

Project description:BackgroundA common means of vaccination coverage measurement is the administrative method, done by dividing the aggregated number of doses administered over a set period (numerator) by the target population (denominator). To assess the quality of national target populations, we defined nine potential denominator data inconsistencies or flags that would warrant further exploration and examination of data reported by Member States to the World Health Organization (WHO) and UNICEF between 2000 and 2016.Methods and findingsWe used the denominator reported to calculate national coverage for BCG, a tuberculosis vaccine, and for the third dose of diphtheria-tetanus-pertussis-containing (DTP3) vaccines, usually live births (LB) and surviving infants (SI), respectively. Out of 2,565 possible reporting events (data points for countries using administrative coverage with the vaccine in the schedule and year) for BCG and 2,939 possible reporting events for DTP3, 194 and 274 reporting events were missing, respectively. Reported coverage exceeding 100% was seen in 11% of all reporting events for BCG and in 6% for DTP3. Of all year-to-year percent differences in reported denominators, 12% and 11% exceeded 10% for reported LB and SI, respectively. The implied infant mortality rate, based on the country's reported LB and SI, would be negative in 9% of all reporting events i.e., the country reported more SI than LB for the same year. Overall, reported LB and SI tended to be lower than the UN Population Division 2017 estimates, which would lead to overestimation of coverage, but this difference seems to be decreasing over time. Other inconsistencies were identified using the nine proposed criteria.ConclusionsApplying a set of criteria to assess reported target populations used to estimate administrative vaccination coverage can flag potential quality issues related to the national denominators and may be useful to help monitor ongoing efforts to improve the quality of vaccination coverage estimates.

Dataset Information

Assessing methods for generalizing experimental impact estimates to target populations.

Publications

Assessing methods for generalizing experimental impact estimates to target populations.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets