Dataset Information

Objective and automated protocols for the evaluation of biomedical search engines using No Title Evaluation protocols.

ABSTRACT:

Background

The evaluation of information retrieval techniques has traditionally relied on human judges to determine which documents are relevant to a query and which are not. This protocol is used in the Text Retrieval Evaluation Conference (TREC), organized annually for the past 15 years, to support the unbiased evaluation of novel information retrieval approaches. The TREC Genomics Track has recently been introduced to measure the performance of information retrieval for biomedical applications.

Results

We describe two protocols for evaluating biomedical information retrieval techniques without human relevance judgments. We call these protocols No Title Evaluation (NT Evaluation). The first protocol measures performance for focused searches, where only one relevant document exists for each query. The second protocol measures performance for queries expected to have potentially many relevant documents per query (high-recall searches). Both protocols take advantage of the clear separation of titles and abstracts found in Medline. We compare the performance obtained with these evaluation protocols to results obtained by reusing the relevance judgments produced in the 2004 and 2005 TREC Genomics Track and observe significant correlations between performance rankings generated by our approach and TREC. Spearman's correlation coefficients in the range of 0.79-0.92 are observed comparing bpref measured with NT Evaluation or with TREC evaluations. For comparison, coefficients in the range 0.86-0.94 can be observed when evaluating the same set of methods with data from two independent TREC Genomics Track evaluations. We discuss the advantages of NT Evaluation over the TRels and the data fusion evaluation protocols introduced recently.

Conclusion

Our results suggest that the NT Evaluation protocols described here could be used to optimize some search engine parameters before human evaluation. Further research is needed to determine if NT Evaluation or variants of these protocols can fully substitute for human evaluations.

SUBMITTER: Campagne F

PROVIDER: S-EPMC2292696 | biostudies-literature | 2008 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Objective and automated protocols for the evaluation of biomedical search engines using No Title Evaluation protocols.

Campagne Fabien F

BMC bioinformatics 20080229

<h4>Background</h4>The evaluation of information retrieval techniques has traditionally relied on human judges to determine which documents are relevant to a query and which are not. This protocol is used in the Text Retrieval Evaluation Conference (TREC), organized annually for the past 15 years, to support the unbiased evaluation of novel information retrieval approaches. The TREC Genomics Track has recently been introduced to measure the performance of information retrieval for biomedical app ...[more]

PMID: 18312673

Similar Datasets

Project description:The goal of this work is to develop an automatic system for the evaluation of the gustatory sensitivity of patients using an electrophysiological recording of the response of bud cells to taste stimuli. In particular, the study aims to evaluate the effectiveness and limitations of supervised classifiers in the discrimination between subjects belonging to the three 6-n-propylthiouracil (PROP) taster categories (supertasters, medium tasters, and non-tasters), exploiting features extracted from electrophysiological recordings of the tongue. Thirty-nine subjects (equally divided into the three PROP status classes by standard non-objective scaling methods) underwent a non-invasive, differential, biopotential recording of their tongues during stimulation with PROP by using a custom-made, flexible, silver electrode. Two different classifiers were trained to recognize up to seven different features extracted from the recorded depolarization signal. The classification results indicate that the identified set of features allows to distinguish between PROP tasters and non-tasters (average accuracy of 80% ± 18% and up to 94% ± 15% when only supertasters and non-tasters are considered), but medium tasters were difficult to identify. However, these apparent classification errors are related to uncertainty in the labeling procedures, which are based on non-objective tests, in which the subjects provided borderline evaluations. Thus, using the proposed method, it is possible, for the first time, to automatically achieve objective PROP taster status identification with high accuracy. The simplicity of the recording technique allows for easy reproduction of the experimental setting; thus the technique can be used in future studies to evaluate other gustatory stimuli. The proposed approach represents the first objective and automatic method to directly measure human gustatory responses and a milestone for physiological taste studies, with applications ranging from basic science to food tasting evaluations.

Dataset Information

Objective and automated protocols for the evaluation of biomedical search engines using No Title Evaluation protocols.

Background

Results

Conclusion

Publications

Objective and automated protocols for the evaluation of biomedical search engines using No Title Evaluation protocols.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets