Dataset Information

Maximizing human effort for analyzing scientific images: A case study using digitized herbarium sheets.

ABSTRACT: Premise:Digitization and imaging of herbarium specimens provides essential historical phenotypic and phenological information about plants. However, the full use of these resources requires high-quality human annotations for downstream use. Here we provide guidance on the design and implementation of image annotation projects for botanical research. Methods and Results:We used a novel gold-standard data set to test the accuracy of human phenological annotations of herbarium specimen images in two settings: structured, in-person sessions and an online, community-science platform. We examined how different factors influenced annotation accuracy and found that botanical expertise, academic career level, and time spent on annotations had little effect on accuracy. Rather, key factors included traits and taxa being scored, the annotation setting, and the individual scorer. In-person annotations were significantly more accurate than online annotations, but both generated relatively high-quality outputs. Gathering multiple, independent annotations for each image improved overall accuracy. Conclusions:Our results provide a best-practices basis for using human effort to annotate images of plants. We show that scalable community science mechanisms can produce high-quality data, but care must be taken to choose tractable taxa and phenophases and to provide informative training material.

SUBMITTER: Brenskelle L

PROVIDER: S-EPMC7328657 | biostudies-literature | 2020 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Maximizing human effort for analyzing scientific images: A case study using digitized herbarium sheets.

Brenskelle Laura L Guralnick Rob P RP Denslow Michael M Stucky Brian J BJ

Applications in plant sciences 20200601 6

<h4>Premise</h4>Digitization and imaging of herbarium specimens provides essential historical phenotypic and phenological information about plants. However, the full use of these resources requires high-quality human annotations for downstream use. Here we provide guidance on the design and implementation of image annotation projects for botanical research.<h4>Methods and results</h4>We used a novel gold-standard data set to test the accuracy of human phenological annotations of herbarium specim ...[more]

PMID: 32626612

Similar Datasets

Project description:Phenology-the timing of life-history events-is a key trait for understanding responses of organisms to climate. The digitization and online mobilization of herbarium specimens is rapidly advancing our understanding of plant phenological response to climate and climatic change. The current practice of manually harvesting data from individual specimens, however, greatly restricts our ability to scale-up data collection. Recent investigations have demonstrated that machine-learning approaches can facilitate this effort. However, present attempts have focused largely on simplistic binary coding of reproductive phenology (e.g., presence/absence of flowers). Here, we use crowd-sourced phenological data of buds, flowers, and fruits from >3,000 specimens of six common wildflower species of the eastern United States (Anemone canadensis L., A. hepatica L., A. quinquefolia L., Trillium erectum L., T. grandiflorum (Michx.) Salisb., and T. undulatum Wild.) to train models using Mask R-CNN to segment and count phenological features. A single global model was able to automate the binary coding of each of the three reproductive stages with >87% accuracy. We also successfully estimated the relative abundance of each reproductive structure on a specimen with ≥90% accuracy. Precise counting of features was also successful, but accuracy varied with phenological stage and taxon. Specifically, counting flowers was significantly less accurate than buds or fruits likely due to their morphological variability on pressed specimens. Moreover, our Mask R-CNN model provided more reliable data than non-expert crowd-sourcers but not botanical experts, highlighting the importance of high-quality human training data. Finally, we also demonstrated the transferability of our model to automated phenophase detection and counting of the three Trillium species, which have large and conspicuously-shaped reproductive organs. These results highlight the promise of our two-phase crowd-sourcing and machine-learning pipeline to segment and count reproductive features of herbarium specimens, thus providing high-quality data with which to investigate plant responses to ongoing climatic change.

Project description:The digitization of herbaria and their online access will greatly facilitate access to plant collections around the world. This will improve the efficiency of taxonomy and help reduce inequalities between scientists. The Botanic Garden Meise, Belgium, is currently digitizing 1.2 million specimens including label data. In this paper we describe the user requirements analysis conducted for a new herbarium web portal. The aim was to identify the required functionality, but also to assist in the prioritization of software development and data acquisition. The Garden conducted the analysis in cooperation with Clockwork, the digital engagement agency of Ordina. Using a series of interactive interviews, potential users were consulted from universities, research institutions, science-policy initiatives and the Botanic Garden Meise. Although digital herbarium data have many potential stakeholders, we focused on the needs of taxonomists, ecologists and historians, who are currently the primary users of the Meise herbarium data portal. The three categories of user have similar needs, all wanted as much specimen data as possible, and for those data, to be interlinked with other digital resources within and outside the Garden. Many users wanted an interactive system that they could comment on, or correct online, particularly if such corrections and annotations could be used to rank the reliability of data. Many requirements depend on the quality of the digitized data associated with each specimen. The essential data fields are the taxonomic name; geographic location; country; collection date; collector name and collection number. Also all researchers valued linkage between biodiversity literature and specimens. Nevertheless, to verify digitized data the researchers still want access to high quality images, even if fully transcribed label information is provided. The only major point of disagreement is the level of access users should have and what they should be allowed to do with the data and images. Not all of the user requirements are feasible given the current technical and regulatory landscape, however, the potential of these suggestions is discussed. Currently, there is no off-the-shelf solution to satisfy all these user requirements, but the intention of this paper is to guide other herbaria who are prioritising their investment in digitization and online web functionality.

Project description:Using the genus Casearia, we assessed the status of nested singletons: individual specimens corresponding to accepted species but in molecular trees appearing nested within clades of closely related species. Normally, such cases would be left undecided, while on the other hand, timely taxonomic decisions are required. We argue that morphological, chorological, and ecological data can be informative to illuminate patterns of speciation. Their use can provide a first step in testing taxon concepts at species level. We focused on five cases of nested singletons in trees of the genus Casearia. We employed PCA and cluster analysis to assess phenotypic differentiation. Using geocoordinates, we calculated niche space differentiation based on 19 bioclim variables, by means of PCA and niche equivalency and similarity tests and generated dot maps. We found that the singletons were morphologically distinctive in two of the five cases (Casearia selloana and C. manausensis), relatively distinctive in two other cases (C. zizyphoides and C. mariquitensis), and partially overlapping in the last case (C. grandiflora). For two cases (C. mariquitensis and C. selloana), ecological niche space was broadly overlapping, in two cases it was found broadly nested (C. grandiflora and C. zizyphoides), and in one case narrowly nested (C. manausensis), but in no case niche differentiation was observed. Niche overlap, similarity and equivalency showed corresponding patterns. Given these data, one would interpret C. selloana and C. manausensis as presumably well-distinguished taxa, their narrow distribution ranges suggesting recently emerging lineages. The other three cases are not clearcut. Morphological data would suggest particularly C. grandiflora conspecific with C. arborea, but differences in the distribution are intriguing. Our approach would reject the notion of potential synonymy based on nested phylogenetic placement for at least two of the five cases. The other case also shows no complete lack of differentiation which would support synonymy.

Project description:Importance:Decades of effort have been devoted to establishing an automated microscopic diagnosis of malaria, but there are challenges in achieving expert-level performance in real-world clinical settings because publicly available annotated data for benchmark and validation are required. Objective:To assess an expert-level malaria detection algorithm using a publicly available benchmark image data set. Design, Setting, and Participants:In this diagnostic study, clinically validated malaria image data sets, the Taiwan Images for Malaria Eradication (TIME), were created by digitizing thin blood smears acquired from patients with malaria selected from the biobank of the Taiwan Centers for Disease Control from January 1, 2003, to December 31, 2018. These smear images were annotated by 4 clinical laboratory scientists who worked in medical centers in Taiwan and trained for malaria microscopic diagnosis at the national reference laboratory of the Taiwan Centers for Disease Control. With TIME, a convolutional neural network-based object detection algorithm was developed for identification of malaria-infected red blood cells. A diagnostic challenge using another independent data set within TIME was performed to compare the algorithm performance against that of human experts as clinical validation. Main Outcomes and Measures:Performance on detecting Plasmodium falciparum-infected blood cells was measured by average precision, and performance on detecting P falciparum infection at the image level was measured using sensitivity, specificity, and area under the receiver operating characteristic curve (AUC). Results:The TIME data sets contained 8145 images of 36 blood smears from patients with suspected malaria (30 P falciparum-positive and 6 P falciparum-negative smears) that had reliable annotations. For clinical validation, the average precision was 0.885 for detecting P falciparum-infected blood cells and 0.838 for ring form. For detecting P falciparum infection on blood smear images, the algorithm had expert-level performance (sensitivity, 0.995; specificity, 0.900; AUC, 0.997 [95% CI, 0.993-0.999]), especially in detecting ring form (sensitivity, 0.968; specificity, 0.960; AUC, 0.995 [95% CI, 0.990-0.998]) compared with experienced microscopists (mean sensitivity, 0.995 [95% CI, 0.993-0.998]; mean specificity, 0.955 [95% CI, 0.885-1.000]). Conclusions and Relevance:The findings suggest that a clinically validated expert-level malaria detection algorithm can be developed by using reliable data sets.

Dataset Information

Maximizing human effort for analyzing scientific images: A case study using digitized herbarium sheets.

Publications

Maximizing human effort for analyzing scientific images: A case study using digitized herbarium sheets.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets