Dataset Information

A user-oriented web crawler for selectively acquiring online content in e-health research.

ABSTRACT:

Motivation

Life stories of diseased and healthy individuals are abundantly available on the Internet. Collecting and mining such online content can offer many valuable insights into patients' physical and emotional states throughout the pre-diagnosis, diagnosis, treatment and post-treatment stages of the disease compared with those of healthy subjects. However, such content is widely dispersed across the web. Using traditional query-based search engines to manually collect relevant materials is rather labor intensive and often incomplete due to resource constraints in terms of human query composition and result parsing efforts. The alternative option, blindly crawling the whole web, has proven inefficient and unaffordable for e-health researchers.

Results

We propose a user-oriented web crawler that adaptively acquires user-desired content on the Internet to meet the specific online data source acquisition needs of e-health researchers. Experimental results on two cancer-related case studies show that the new crawler can substantially accelerate the acquisition of highly relevant online content compared with the existing state-of-the-art adaptive web crawling technology. For the breast cancer case study using the full training set, the new method achieves a cumulative precision between 74.7 and 79.4% after 5 h of execution till the end of the 20-h long crawling session as compared with the cumulative precision between 32.8 and 37.0% using the peer method for the same time period. For the lung cancer case study using the full training set, the new method achieves a cumulative precision between 56.7 and 61.2% after 5 h of execution till the end of the 20-h long crawling session as compared with the cumulative precision between 29.3 and 32.4% using the peer method. Using the reduced training set in the breast cancer case study, the cumulative precision of our method is between 44.6 and 54.9%, whereas the cumulative precision of the peer method is between 24.3 and 26.3%; for the lung cancer case study using the reduced training set, the cumulative precisions of our method and the peer method are, respectively, between 35.7 and 46.7% versus between 24.1 and 29.6%. These numbers clearly show a consistently superior accuracy of our method in discovering and acquiring user-desired online content for e-health research.

Availability and implementation

The implementation of our user-oriented web crawler is freely available to non-commercial users via the following Web site: http://bsec.ornl.gov/AdaptiveCrawler.shtml. The Web site provides a step-by-step guide on how to execute the web crawler implementation. In addition, the Web site provides the two study datasets including manually labeled ground truth, initial seeds and the crawling results reported in this article.

SUBMITTER: Xu S

PROVIDER: S-EPMC3866553 | biostudies-literature | 2014 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A user-oriented web crawler for selectively acquiring online content in e-health research.

Xu Songhua S Yoon Hong-Jun HJ Tourassi Georgia G

Bioinformatics (Oxford, England) 20130929 1

<h4>Motivation</h4>Life stories of diseased and healthy individuals are abundantly available on the Internet. Collecting and mining such online content can offer many valuable insights into patients' physical and emotional states throughout the pre-diagnosis, diagnosis, treatment and post-treatment stages of the disease compared with those of healthy subjects. However, such content is widely dispersed across the web. Using traditional query-based search engines to manually collect relevant mater ...[more]

PMID: 24078710

Similar Datasets

Project description:BackgroundOnline health communities (OHCs) have increasingly gained traction with patients, caregivers, and supporters globally. Chinese OHCs are no exception. However, user-generated content (UGC) and the associated user behaviors in Chinese OHCs are largely underexplored and rarely analyzed systematically, forfeiting valuable opportunities for optimizing treatment design and care delivery with insights gained from OHCs.ObjectiveThis study aimed to reveal both the shared and distinct characteristics of 2 popular OHCs in China by systematically and comprehensively analyzing their UGC and the associated user behaviors.MethodsWe concentrated on studying the lung cancer forum (LCF) and breast cancer forum (BCF) on Mijian, and the diabetes consultation forum (DCF) on Sweet Home, because of the importance of the 3 diseases among Chinese patients and their prevalence on Chinese OHCs in general. Our analysis explored the key user activities, small-world effect, and scale-free characteristics of each social network. We examined the UGC of these forums comprehensively and adopted the weighted knowledge network technique to discover salient topics and latent relations among these topics on each forum. Finally, we discussed the public health implications of our analysis findings.ResultsOur analysis showed that the number of reads per thread on each forum followed gamma distribution (HL=0, HB=0, and HD=0); the number of replies on each forum followed exponential distribution (adjusted RL2=0.946, adjusted RB2=0.958, and adjusted RD2=0.971); and the number of threads a user is involved with (adjusted RL2=0.978, adjusted RB2=0.964, and adjusted RD2=0.970), the number of followers of a user (adjusted RL2=0.989, adjusted RB2=0.962, and adjusted RD2=0.990), and a user's degrees (adjusted RL2=0.997, adjusted RB2=0.994, and adjusted RD2=0.968) all followed power-law distribution. The study further revealed that users are generally more active during weekdays, as commonly witnessed in all 3 forums. In particular, the LCF and DCF exhibited high temporal similarity (ρ=0.927; P<.001) in terms of the relative thread posting frequencies during each hour of the day. Besides, the study showed that all 3 forums exhibited the small-world effect (mean σL=517.15, mean σB=275.23, and mean σD=525.18) and scale-free characteristics, while the global clustering coefficients were lower than those of counterpart international OHCs. The study also discovered several hot topics commonly shared among the 3 disease forums, such as disease treatment, disease examination, and diagnosis. In particular, the study found that after the outbreak of COVID-19, users on the LCF and BCF were much more likely to bring up COVID-19-related issues while discussing their medical issues.ConclusionsUGC and related online user behaviors in Chinese OHCs can be leveraged as important sources of information to gain insights regarding individual and population health conditions. Effective and timely mining and utilization of such content can continuously provide valuable firsthand clues for enhancing the situational awareness of health providers and policymakers.

Project description:BackgroundMen can be hard to reach with face-to-face health-related information, while increasingly, research shows that they are seeking health information from online sources. Recognizing this trend, there is merit in developing innovative online knowledge translation (KT) strategies capable of translating research on men's health into engaging health promotion materials. While the concept of KT has become a new mantra for researchers wishing to bridge the gap between research evidence and improved health outcomes, little is written about the process, necessary skills, and best practices by which researchers can develop online knowledge translation.ObjectiveOur aim was to illustrate some of the processes and challenges involved in, and potential value of, developing research knowledge online to promote men's health.MethodsWe present experiences of KT across two case studies of men's health. First, we describe a study that uses interactive Web apps to translate knowledge relating to Canadian men's depression. Through a range of mechanisms, study findings were repackaged with the explicit aim of raising awareness and reducing the stigma associated with men's depression and/or help-seeking. Second, we describe an educational resource for teenage men about unintended pregnancy, developed for delivery in the formal Relationship and Sexuality Education school curricula of Ireland, Northern Ireland (United Kingdom), and South Australia. The intervention is based around a Web-based interactive film drama entitled "If I Were Jack".ResultsFor each case study, we describe the KT process and strategies that aided development of credible and well-received online content focused on men's health promotion. In both case studies, the original research generated the inspiration for the interactive online content and the core development strategy was working with a multidisciplinary team to develop this material through arts-based approaches. In both cases also, there is an acknowledgment of the need for gender and culturally sensitive information. Both aimed to engage men by disrupting stereotypes about men, while simultaneously addressing men through authentic voices and faces. Finally, in both case studies we draw attention to the need to think beyond placement of content online to delivery to target audiences from the outset.ConclusionsThe case studies highlight some of the new skills required by academics in the emerging paradigm of translational research and contribute to the nascent literature on KT. Our approach to online KT was to go beyond dissemination and diffusion to actively repackage research knowledge through arts-based approaches (videos and film scripts) as health promotion tools, with optimal appeal, to target male audiences. Our findings highlight the importance of developing a multidisciplinary team to inform the design of content, the importance of adaptation to context, both in terms of the national implementation context and consideration of gender-specific needs, and an integrated implementation and evaluation framework in all KT work.

Project description:Introduction:User generated content (UGC) is a valuable but underutilized source of information about individuals who participate in online cessation interventions. This study represents a first effort to passively detect smoking status among members of an online cessation program using UGC. Methods:Secondary data analysis was performed on data from 826 participants in a web-based smoking cessation randomized trial that included an online community. Domain experts from the online community reviewed each post and comment written by participants and attempted to infer the author's smoking status at the time it was written. Inferences from UGC were validated by comparison with self-reported 30-day point prevalence abstinence (PPA). Following validation, the impact of this method was evaluated across all individuals and time points in the study period. Results:Of the 826 participants in the analytic sample, 719 had written at least one post from which content inference was possible. Among participants for whom unambiguous smoking status was inferred during the 30 days preceding their 3-month follow-up survey, concordance with self-report was almost perfect (kappa = 0.94). Posts indicating abstinence tended to be written shortly after enrollment (median = 14 days). Conclusions:Passive inference of smoking status from UGC in online cessation communities is possible and highly reliable for smokers who actively produce content. These results lay the groundwork for further development of observational research tools and intervention innovations. Implications:A proof-of-concept methodology for inferring smoking status from user generated content in online cessation communities is presented and validated. Content inference of smoking status makes a key cessation variable available for use in observational designs. This method provides a powerful tool for researchers interested in online cessation interventions and establishes a foundation for larger scale application via machine learning.

Dataset Information

A user-oriented web crawler for selectively acquiring online content in e-health research.

Motivation

Results

Availability and implementation

Publications

A user-oriented web crawler for selectively acquiring online content in e-health research.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets