Dataset Information

Novel comparison of evaluation metrics for gene ontology classifiers reveals drastic performance differences.

ABSTRACT: Automated protein annotation using the Gene Ontology (GO) plays an important role in the biosciences. Evaluation has always been considered central to developing novel annotation methods, but little attention has been paid to the evaluation metrics themselves. Evaluation metrics define how well an annotation method performs and allows for them to be ranked against one another. Unfortunately, most of these metrics were adopted from the machine learning literature without establishing whether they were appropriate for GO annotations. We propose a novel approach for comparing GO evaluation metrics called Artificial Dilution Series (ADS). Our approach uses existing annotation data to generate a series of annotation sets with different levels of correctness (referred to as their signal level). We calculate the evaluation metric being tested for each annotation set in the series, allowing us to identify whether it can separate different signal levels. Finally, we contrast these results with several false positive annotation sets, which are designed to expose systematic weaknesses in GO assessment. We compared 37 evaluation metrics for GO annotation using ADS and identified drastic differences between metrics. We show that some metrics struggle to differentiate between different signal levels, while others give erroneously high scores to the false positive data sets. Based on our findings, we provide guidelines on which evaluation metrics perform well with the Gene Ontology and propose improvements to several well-known evaluation metrics. In general, we argue that evaluation metrics should be tested for their performance and we provide software for this purpose (https://bitbucket.org/plyusnin/ads/). ADS is applicable to other areas of science where the evaluation of prediction results is non-trivial.

SUBMITTER: Plyusnin I

PROVIDER: S-EPMC6855565 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundSpurred by the Coronavirus infectious disease 2019 pandemic, aerosol containment devices (ACDs) were developed to capture infectious respiratory aerosols generated by patients at their source. Prior reviews indicated that such devices had low evidence of effectiveness, but did not address how ACDs should be evaluated, how well they should perform, nor have clearly defined performance standards. Towards developing design criteria for ACDs, two questions were posed: 1) What characteristics have guided the design of ACDs? 2) How have these characteristics been evaluated?MethodsA scoping review was performed consistent with PRISMA guidelines. Data were extracted with respect to general study information, intended use of the device, device design characteristics and evaluation.ResultsFifty-four articles were included. Evaluation was most commonly performed with respect to device aerosol containment (n = 31, 61%), with only 5 (9%), 3 (6%) and 8 (15%) formally assessing providing experience, patient experience and procedure impact, respectively. Nearly all of the studies that explored provider experience and procedure impact studied intubation. Few studies provided a priori performance criteria for any evaluation metric, or referenced any external guidelines by which to bench mark performance.ConclusionWith respect to aerosol containment, ACDs should reduce exposure among HCP with the device compared with the absence of the device, and provide ≥90% reduction in respirable aerosols, equivalent in performance to N95 filtering facepiece respirators, if the goal is to reduce reliance on personal protective equipment. The ACD should not increase awkward or uncomfortable postures, or adversely impact biomechanics of the procedure itself as this could have implications for procedure outcomes. A variety of standardized instruments exist to assess the experience of patients and healthcare personnel. Integration of ACDs into routine clinical practice requires rigorous studies of aerosol containment and the user experience.

Project description:The performance of a newly developed, high resolution, microangiographic fluoroscope (MAF) (35 μm pixel pitch and 300 μm thick CsI phosphor) was evaluated using a generalized linear system analysis and compared with that of a standard amorphous Si thin film transistor flat panel detector (FPD) (194 μm pixel pitch and 600 μm thick CsI phosphor). The linear system metrics such as modulation transfer function (MTF), noise power spectrum, and detection quantum efficiency (DQE) are commonly used to gauge the intrinsic detector performance in the detector plane. However, these linear system metrics do not provide information about the image receptor performance in a real system since they do not include the effects of other parameters such as focal spot distribution, scatter radiation, and geometric unsharpness, which may compromise detector performance characteristics. Use of generalized linear system metrics [generalized modulation transfer function (GMTF), generalized normalized noise power spectrum (GNNPS), and generalized detection quantum efficiency (GDQE)] that include these effects gives a more meaningful, complete, and appropriate evaluation of detector performance as part of the imaging system.A uniform head equivalent phantom was used to simulate realistic clinical parameters and x-ray spectra. The detector MTFs were measured using the slanted edge method and the focal spot MTFs were measured using a pinhole assembly. The scatter MTF was simulated and the scatter fraction was measured for a head-equivalent phantom. The generalized system metrics were calculated for different combinations of three choices of focal spots and three different magnifications with two different air-gaps. The performance of the MAF was also illustrated using stent images obtained with different focal spots under similar conditions.Results for the generalized metrics provide a quantitative description of the performance of the imaging system for both detectors. This generalized analysis demonstrated that both detectors have similar imaging capabilities at lower spatial frequencies, but that the MAF has superior performance over the FPD at higher frequencies even when considering focal spot blurring and scatter.This generalized performance analysis demonstrates the significance of focal spot size, magnification, and scatter on the system performance metrics (GMTF, GNNPS, and GDQE). Although the ideal detector performance characteristics of the MAF are not fully realized due to these other system factors, it still retains an advantage in DQE at high spatial frequencies over the FPD. Similar studies based on the generalized linear system metrics can serve as an efficient tool to evaluate total system capabilities under different realistic conditions to enable optimal design for specific imaging tasks.

Dataset Information

Novel comparison of evaluation metrics for gene ontology classifiers reveals drastic performance differences.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets