Dataset Information

Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enough.

ABSTRACT:

Background

Recent research indicates a high recall in Google Scholar searches for systematic reviews. These reports raised high expectations of Google Scholar as a unified and easy to use search interface. However, studies on the coverage of Google Scholar rarely used the search interface in a realistic approach but instead merely checked for the existence of gold standard references. In addition, the severe limitations of the Google Search interface must be taken into consideration when comparing with professional literature retrieval tools.The objectives of this work are to measure the relative recall and precision of searches with Google Scholar under conditions which are derived from structured search procedures conventional in scientific literature retrieval; and to provide an overview of current advantages and disadvantages of the Google Scholar search interface in scientific literature retrieval.

Methods

General and MEDLINE-specific search strategies were retrieved from 14 Cochrane systematic reviews. Cochrane systematic review search strategies were translated to Google Scholar search expression as good as possible under consideration of the original search semantics. The references of the included studies from the Cochrane reviews were checked for their inclusion in the result sets of the Google Scholar searches. Relative recall and precision were calculated.

Results

We investigated Cochrane reviews with a number of included references between 11 and 70 with a total of 396 references. The Google Scholar searches resulted in sets between 4,320 and 67,800 and a total of 291,190 hits. The relative recall of the Google Scholar searches had a minimum of 76.2% and a maximum of 100% (7 searches). The precision of the Google Scholar searches had a minimum of 0.05% and a maximum of 0.92%. The overall relative recall for all searches was 92.9%, the overall precision was 0.13%.

Conclusion

The reported relative recall must be interpreted with care. It is a quality indicator of Google Scholar confined to an experimental setting which is unavailable in systematic retrieval due to the severe limitations of the Google Scholar search interface. Currently, Google Scholar does not provide necessary elements for systematic scientific literature retrieval such as tools for incremental query optimization, export of a large number of references, a visual search builder or a history function. Google Scholar is not ready as a professional searching tool for tasks where structured retrieval methodology is necessary.

SUBMITTER: Boeker M

PROVIDER: S-EPMC3840556 | biostudies-literature | 2013 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enough.

Boeker Martin M Vach Werner W Motschall Edith E

BMC medical research methodology 20131026

<h4>Background</h4>Recent research indicates a high recall in Google Scholar searches for systematic reviews. These reports raised high expectations of Google Scholar as a unified and easy to use search interface. However, studies on the coverage of Google Scholar rarely used the search interface in a realistic approach but instead merely checked for the existence of gold standard references. In addition, the severe limitations of the Google Search interface must be taken into consideration when ...[more]

PMID: 24160679

Similar Datasets

Project description:BACKGROUND:The usefulness of Google Scholar (GS) as a bibliographic database for biomedical systematic review (SR) searching is a subject of current interest and debate in research circles. Recent research has suggested GS might even be used alone in SR searching. This assertion is challenged here by testing whether GS can locate all studies included in 21 previously published SRs. Second, it examines the recall of GS, taking into account the maximum number of items that can be viewed, and tests whether more complete searches created by an information specialist will improve recall compared to the searches used in the 21 published SRs. METHODS:The authors identified 21 biomedical SRs that had used GS and PubMed as information sources and reported their use of identical, reproducible search strategies in both databases. These search strategies were rerun in GS and PubMed, and analyzed as to their coverage and recall. Efforts were made to improve searches that underperformed in each database. RESULTS:GS' overall coverage was higher than PubMed (98% versus 91%) and overall recall is higher in GS: 80% of the references included in the 21 SRs were returned by the original searches in GS versus 68% in PubMed. Only 72% of the included references could be used as they were listed among the first 1,000 hits (the maximum number shown). Practical precision (the number of included references retrieved in the first 1,000, divided by 1,000) was on average 1.9%, which is only slightly lower than in other published SRs. Improving searches with the lowest recall resulted in an increase in recall from 48% to 66% in GS and, in PubMed, from 60% to 85%. CONCLUSIONS:Although its coverage and precision are acceptable, GS, because of its incomplete recall, should not be used as a single source in SR searching. A specialized, curated medical database such as PubMed provides experienced searchers with tools and functionality that help improve recall, and numerous options in order to optimize precision. Searches for SRs should be performed by experienced searchers creating searches that maximize recall for as many databases as deemed necessary by the search expert.

Project description:PURPOSE:Epidemic and seasonal infectious conjunctivitis outbreaks can impact education, workforce, and economy adversely. Yet conjunctivitis typically is not a reportable disease, potentially delaying mitigating intervention. Our study objective was to determine if conjunctivitis epidemics could be identified using Google Trends search data. DESIGN:Search data for conjunctivitis-related and control search terms from 5 years and countries worldwide were obtained. Country and term were masked. Temporal scan statistics were applied to identify candidate epidemics. Candidates then were assessed for geotemporal concordance with an a priori defined collection of known reported conjunctivitis outbreaks, as a measure of sensitivity. PARTICIPANTS:Populations by country that searched Google's search engine using our study terms. MAIN OUTCOME MEASURES:Percent of known conjunctivitis outbreaks also found in the same country and period by our candidate epidemics, identified from conjunctivitis-related searches. RESULTS:We identified 135 candidate conjunctivitis epidemic periods from 77 countries. Compared with our a priori defined collection of known reported outbreaks, candidate conjunctivitis epidemics identified 18 of 26 (69% sensitivity) of the reported country-wide or island nationwide outbreaks, or both; 9 of 20 (45% sensitivity) of the reported region or district-wide outbreaks, or both; but far fewer nosocomial and reported smaller outbreaks. Similar overall and individual sensitivity, as well as specificity, were found on a country-level basis. We also found that 83% of our candidate epidemics had start dates before (of those, 20% were more than 12 weeks before) their concurrent reported outbreak's report issuance date. Permutation tests provided evidence that on average, conjunctivitis candidate epidemics occurred geotemporally closer to outbreak reports than chance alone suggests (P < 0.001) unlike control term candidates (P = 0.40). CONCLUSIONS:Conjunctivitis outbreaks can be detected using temporal scan analysis of Google search data alone, with more than 80% detected before an outbreak report's issuance date, some as early as the reported outbreak's start date. Future approaches using data from smaller regions, social media, and more search terms may improve sensitivity further and cross-validate detected candidates, allowing identification of candidate conjunctivitis epidemics from Internet search data potentially to complementarily benefit traditional reporting and detection systems to improve epidemic awareness.

Dataset Information

Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enough.

Background

Methods

Results

Conclusion

Publications

Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enough.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets