How best to quantify replication success? A simulation study on the comparison of replication success metrics.
Ontology highlight
ABSTRACT:
SUBMITTER: Muradchanian J
PROVIDER: S-EPMC8131945 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:BackgroundEvidence-based conservation is urgently needed to identify, apply and promote effective interventions for mitigation of threats and recovery of the natural environment. Estimation of intervention effectiveness is subject to robust study design and statistical analysis, and much progress is documented in these fields. In contrast, little is understood about the accuracy and biases (underestimation and overestimation) of different effectiveness metrics and how they are affected by sample size.MethodsIn this study, a dataset (n = 500 cases) consisting of random, positive, integer numbers was simulated to produce frequency input data for the 2 × 2 contingency table. For each case, three metrics of the relative risk, odds ratio and the magnitude of change were calculated, their disparity was estimated and the characteristics of treatment (with intervention) and control (without intervention) samples significantly affecting this disparity were studied by means of linear regression. Further, four case studies from different conservation interventions are provided to support the results.ResultsThe study has shown that the relative risk and the magnitude of change produce identical estimates of intervention effectiveness only when treatment and control samples are equal, or when the number of target outcomes (e.g., number of livestock killed by predators) in treatment sample reaches zero. In other situations, the magnitude of change gives overestimates or underestimates, depending on relationships between treatment and control sample sizes. The table summarizing the conditions of equalities and biases between these two metrics is provided. These conditions are valid for both reduction-aimed interventions reducing negative target outcomes (e.g., livestock protection to reduce livestock losses to predators) and for addition-aimed interventions increasing positive target outcomes (e.g., establishment of protected areas to increase species presence). No significant effects on the odds ratio were found.ConclusionResearchers should set equal treatment and control sample sizes so that to produce identical estimates of intervention effectiveness by the relative risk and the magnitude of change. Otherwise, these estimates are biased if produced by the magnitude of change and the relative risk should be used instead. As setting equal treatment and control samples can be impractical, I encourage researchers and practitioners to use the relative risk in estimation of intervention effectiveness. This will not take additional efforts as both metrics are calculated from the same contingency table. Treatment and control sample sizes, along with their sub-samples affected and not by an intervention, should be explicitly reported by researchers to allow independent evaluation of intervention effectiveness. This approach can help obtain more accurate information on intervention effectiveness for making better decisions in conservation actions.
| S-EPMC7519717 | biostudies-literature
Project description:To increase the probability of success in drug discovery, the concept of drug-like properties was introduced. Efficiency metrics that normalize potency against these properties could help reach drug-like space more efficiently. Potential reasons for the inefficient use of metrics and suboptimal decision making are discussed.
| S-EPMC4027587 | biostudies-other
Project description:ObjectivesThe Joint United Nations Programme on HIV/AIDS (UNAIDS) 90-90-90 and other cross-sectional metrics can lead to potentially counterintuitive conclusions when used to evaluate health systems' performance. This study demonstrates how time and population dynamics impact UNAIDS 90-90-90 metrics in comparison with a longitudinal analogue.DesignA simplified simulation representing a hypothetical population was used to estimate and compare inference from UNAIDS 90-90-90 metrics and longitudinal metrics based on Kaplan-Meier-estimated 2-year probability of transition between stages.MethodsWe simulated a large cohort over 15 years. Everyone started out at risk for HIV, and then transitioned through the HIV care continuum based on fixed daily probabilities of acquiring HIV, learning status, entering care, initiating antiretroviral therapy (ART), and becoming virally suppressed, or dying. We varied the probability of ART initiation over three five-year periods (low, high, and low). We repeated the simulation with an increased probability of death.ResultsThe cross-sectional probability of being on ART among persons who were diagnosed responded relatively slowly to changes in the rate of ART initiation. Increases in ART initiation rates caused apparent declines in the cross-sectional probability of being virally suppressed among persons who had initiated ART, despite no changes in the rate of viral suppression. In some cases, higher mortality resulted in the cross-sectional metrics implying improved healthcare system performance. The longitudinal continuum was robust to these issues.ConclusionThe UNAIDS 90-90-90 care continuum may lead to incorrect inference when used to evaluate health systems performance. We recommend that evaluation of HIV care delivery include longitudinal care continuum metrics wherever possible.
| S-EPMC7253182 | biostudies-literature
Project description:Simulated oral microbiome with artificial duplicates
| PRJEB26333 | ENA
Project description:Simulated oral microbiome with human contamination.
| PRJEB25791 | ENA
Project description:The recent replication crisis has caused several scientific disciplines to self-reflect on the frequency with which they replicate previously published studies and to assess their success in such endeavours. The rate of replication, however, has yet to be assessed for ecology and evolution. Here, I survey the open-access ecology and evolution literature to determine how often ecologists and evolutionary biologists replicate, or at least claim to replicate, previously published studies. I found that approximately 0.023% of ecology and evolution studies are described by their authors as replications. Two of the 11 original-replication study pairs provided sufficient statistical detail for three effects so as to permit a formal analysis of replication success. Replicating authors correctly concluded that they replicated an original effect in two cases; in the third case, my analysis suggests that the finding by the replicating authors was consistent with the original finding, contrary the conclusion of "replication failure" by the authors.
| S-EPMC6743472 | biostudies-literature
Project description:Recently, fMRI researchers have begun to realize that the brain's intrinsic network patterns may undergo substantial changes during a single resting state (RS) scan. However, despite the growing interest in brain dynamics, metrics that can quantify the variability of network patterns are still quite limited. Here, we first introduce various quantification metrics based on the extension of co-activation pattern (CAP) analysis, a recently proposed point-process analysis that tracks state alternations at each individual time frame and relies on very few assumptions; then apply these proposed metrics to quantify changes of brain dynamics during a sustained 2-back working memory (WM) task compared to rest. We focus on the functional connectivity of two prominent RS networks, the default-mode network (DMN) and executive control network (ECN). We first demonstrate less variability of global Pearson correlations with respect to the two chosen networks using a sliding-window approach during WM task compared to rest; then we show that the macroscopic decrease in variations in correlations during a WM task is also well characterized by the combined effect of a reduced number of dominant CAPs, increased spatial consistency across CAPs, and increased fractional contributions of a few dominant CAPs. These CAP metrics may provide alternative and more straightforward quantitative means of characterizing brain network dynamics than time-windowed correlation analyses.
| S-EPMC4386757 | biostudies-literature
Project description:Quantifying white matter damage in vivo is becoming increasingly important for investigating the effects of neuroprotective and repair strategies in multiple sclerosis (MS). While various approaches are available, the relationship between MRI-based metrics of white matter microstructure in the disease, that is, to what extent the metrics provide complementary versus redundant information, remains largely unexplored. We obtained four microstructural metrics from 123 MS patients: fractional anisotropy (FA), radial diffusivity (RD), myelin water fraction (MWF), and magnetisation transfer ratio (MTR). Coregistration of maps of these four indices allowed quantification of microstructural damage through voxel-wise damage scores relative to healthy tissue, as assessed in a group of 27 controls. We considered three white matter tissue-states, which were expected to vary in microstructural damage: normal appearing white matter (NAWM), T2-weighted hyperintense lesional tissue without T1-weighted hypointensity (T2L), and T1-weighted hypointense lesional tissue with corresponding T2-weighted hyperintensity (T1L). All MRI indices suggested significant damage in all three tissue-states, the greatest damage being in T1L. The correlations between indices ranged from r = 0.18 to r = 0.87. MWF was most sensitive when differentiating T2L from NAWM, while MTR was most sensitive when differentiating T1L from NAWM and from T2L. Combining the four metrics into one, through a principal component analysis, did not yield a measure more sensitive to damage than any single measure. Our findings suggest that the metrics are (at least partially) correlated with each other, but sensitive to the different aspects of pathology. Leveraging these differences could be beneficial in clinical trials testing the effects of therapeutic interventions.
| S-EPMC6563497 | biostudies-literature
Project description:The advancement of synthetic biology requires the ability to create new DNA sequences to produce unique behaviors in biological systems. Automation is increasingly employed to carry out well-established assembly methods of DNA fragments in a multiplexed, high-throughput fashion, allowing many different configurations to be tested simultaneously. However, metrics are required to determine when automation is warranted based on factors such as assembly methodology, protocol details, and number of samples. The goal of our synthetic biology automation work is to develop and test protocols, hardware, and software to investigate and optimize DNA assembly through quantifiable metrics. We performed a parameter analysis of DNA assembly to develop a standardized, highly efficient, and reproducible MoClo protocol, suitable to be used both manually and with liquid-handling robots. We created a key DNA assembly metric (Q-metric) to characterize a given automation method's advantages over conventional manual manipulations with regard to researchers' highest-priority parameters: output, cost, and time. A software tool called Puppeteer was developed to formally capture these metrics, help define the assembly design, and provide human and robotic liquid-handling instructions. Altogether, we contribute to a growing foundation of standardizing practices, metrics, and protocols for automating DNA assembly.
| S-EPMC6819997 | biostudies-literature
Project description:Nanopore sequencing ground study, extreme environment study and International Space Station (ISS) study
| PRJEB41406 | ENA