Dataset Information

A comparison of common metrics used to quantify the effectiveness of conservation interventions.

ABSTRACT:

Background

Evidence-based conservation is urgently needed to identify, apply and promote effective interventions for mitigation of threats and recovery of the natural environment. Estimation of intervention effectiveness is subject to robust study design and statistical analysis, and much progress is documented in these fields. In contrast, little is understood about the accuracy and biases (underestimation and overestimation) of different effectiveness metrics and how they are affected by sample size.

Methods

In this study, a dataset (n = 500 cases) consisting of random, positive, integer numbers was simulated to produce frequency input data for the 2 × 2 contingency table. For each case, three metrics of the relative risk, odds ratio and the magnitude of change were calculated, their disparity was estimated and the characteristics of treatment (with intervention) and control (without intervention) samples significantly affecting this disparity were studied by means of linear regression. Further, four case studies from different conservation interventions are provided to support the results.

Results

The study has shown that the relative risk and the magnitude of change produce identical estimates of intervention effectiveness only when treatment and control samples are equal, or when the number of target outcomes (e.g., number of livestock killed by predators) in treatment sample reaches zero. In other situations, the magnitude of change gives overestimates or underestimates, depending on relationships between treatment and control sample sizes. The table summarizing the conditions of equalities and biases between these two metrics is provided. These conditions are valid for both reduction-aimed interventions reducing negative target outcomes (e.g., livestock protection to reduce livestock losses to predators) and for addition-aimed interventions increasing positive target outcomes (e.g., establishment of protected areas to increase species presence). No significant effects on the odds ratio were found.

Conclusion

Researchers should set equal treatment and control sample sizes so that to produce identical estimates of intervention effectiveness by the relative risk and the magnitude of change. Otherwise, these estimates are biased if produced by the magnitude of change and the relative risk should be used instead. As setting equal treatment and control samples can be impractical, I encourage researchers and practitioners to use the relative risk in estimation of intervention effectiveness. This will not take additional efforts as both metrics are calculated from the same contingency table. Treatment and control sample sizes, along with their sub-samples affected and not by an intervention, should be explicitly reported by researchers to allow independent evaluation of intervention effectiveness. This approach can help obtain more accurate information on intervention effectiveness for making better decisions in conservation actions.

SUBMITTER: Khorozyan I

PROVIDER: S-EPMC7519717 | biostudies-literature | 2020

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A comparison of common metrics used to quantify the effectiveness of conservation interventions.

Khorozyan Igor I

PeerJ 20200923

<h4>Background</h4>Evidence-based conservation is urgently needed to identify, apply and promote effective interventions for mitigation of threats and recovery of the natural environment. Estimation of intervention effectiveness is subject to robust study design and statistical analysis, and much progress is documented in these fields. In contrast, little is understood about the accuracy and biases (underestimation and overestimation) of different effectiveness metrics and how they are affected ...[more]

PMID: 33024627

Similar Datasets

Project description:Increasing energy and housing demands are impacting wildlife populations throughout western North America. Greater sage-grouse (Centrocercus urophasianus), a species known for its sensitivity to landscape-scale disturbance, inhabits the same low elevation sage-steppe in which much of this development is occurring. Wyoming has committed to maintain sage-grouse populations through conservation easements and policy changes that conserves high bird abundance "core" habitat and encourages development in less sensitive landscapes. In this study, we built new predictive models of oil and gas, wind, and residential development and applied build-out scenarios to simulate future development and measure the efficacy of conservation actions for maintaining sage-grouse populations. Our approach predicts sage-grouse population losses averted through conservation action and quantifies return on investment for different conservation strategies. We estimate that without conservation, sage-grouse populations in Wyoming will decrease under our long-term scenario by 14-29% (95% CI: 4-46%). However, a conservation strategy that includes the "core area" policy and $250 million in targeted easements could reduce these losses to 9-15% (95% CI: 3-32%), cutting anticipated losses by roughly half statewide and nearly two-thirds within sage-grouse core breeding areas. Core area policy is the single most important component, and targeted easements are complementary to the overall strategy. There is considerable uncertainty around the magnitude of our estimates; however, the relative benefit of different conservation scenarios remains comparable because potential biases and assumptions are consistently applied regardless of the strategy. There is early evidence based on a 40% reduction in leased hectares inside core areas that Wyoming policy is reducing potential for future fragmentation inside core areas. Our framework using build-out scenarios to anticipate species declines provides estimates that could be used by decision makers to determine if expected population losses warrant ESA listing.

Project description:As human pressures on the environment continue to spread and intensify, effective conservation interventions are direly needed to prevent threats, reduce conflicts, and recover populations and landscapes in a liaison between science and conservation. It is practically important to discriminate between true and false (or misperceived) effectiveness of interventions as false perceptions may shape a wrong conservation agenda and lead to inappropriate decisions and management actions. This study used the false positive risk (FPR) to estimate the rates of misperceived effectiveness of electric fences (overstated if reported as effective but actually ineffective based on FPR; understated otherwise), explain their causes and propose recommendations on how to improve the representation of true effectiveness. Electric fences are widely applied to reduce damage to fenced assets, such as livestock and beehives, or increase survival of fenced populations. The analysis of 109 cases from 50 publications has shown that the effectiveness of electric fences was overstated in at least one-third of cases, from 31.8% at FPR = 0.2 (20% risk) to 51.1% at FPR = 0.05 (5% risk, true effectiveness). In contrast, understatement reduced from 23.8% to 9.5% at these thresholds of FPR. This means that truly effective applications of electric fences were only 48.9% of all cases reported as effective, but truly ineffective cases were 90.5%, implying that the effectiveness of electric fences was heavily overstated. The main reasons of this bias were the lack of statistical testing or improper reporting of test results (63.3% of cases) and interpretation of marginally significant results (p < 0.05, p < 0.1 and p around 0.05) as indicators of effectiveness (10.1%). In conclusion, FPR is an important tool for estimating true effectiveness of conservation interventions and its application is highly recommended to disentangle true and false effectiveness for planning appropriate conservation actions. Researchers are encouraged to calculate FPR, publish its constituent statistics (especially treatment and control sample sizes) and explicitly provide test results with p values. It is suggested to call the effectiveness "true" if FPR < 0.05, "suggestive" if 0.05 ≤ FPR < 0.2 and "false" if FPR ≥ 0.2.

Dataset Information

A comparison of common metrics used to quantify the effectiveness of conservation interventions.

Background

Methods

Results

Conclusion

Publications

A comparison of common metrics used to quantify the effectiveness of conservation interventions.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets