Dataset Information

Text classification for assisting moderators in online health communities.

ABSTRACT:

Objectives

Patients increasingly visit online health communities to get help on managing health. The large scale of these online communities makes it impossible for the moderators to engage in all conversations; yet, some conversations need their expertise. Our work explores low-cost text classification methods to this new domain of determining whether a thread in an online health forum needs moderators' help.

Methods

We employed a binary classifier on WebMD's online diabetes community data. To train the classifier, we considered three feature types: (1) word unigram, (2) sentiment analysis features, and (3) thread length. We applied feature selection methods based on ?² statistics and under sampling to account for unbalanced data. We then performed a qualitative error analysis to investigate the appropriateness of the gold standard.

Results

Using sentiment analysis features, feature selection methods, and balanced training data increased the AUC value up to 0.75 and the F1-score up to 0.54 compared to the baseline of using word unigrams with no feature selection methods on unbalanced data (0.65 AUC and 0.40 F1-score). The error analysis uncovered additional reasons for why moderators respond to patients' posts.

Discussion

We showed how feature selection methods and balanced training data can improve the overall classification performance. We present implications of weighing precision versus recall for assisting moderators of online health communities. Our error analysis uncovered social, legal, and ethical issues around addressing community members' needs. We also note challenges in producing a gold standard, and discuss potential solutions for addressing these challenges.

Conclusion

Social media environments provide popular venues in which patients gain health-related information. Our work contributes to understanding scalable solutions for providing moderators' expertise in these large-scale, social media environments.

SUBMITTER: Huh J

PROVIDER: S-EPMC3874858 | biostudies-literature | 2013 Dec

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Text classification for assisting moderators in online health communities.

Huh Jina J Yetisgen-Yildiz Meliha M Pratt Wanda W

Journal of biomedical informatics 20130908 6

<h4>Objectives</h4>Patients increasingly visit online health communities to get help on managing health. The large scale of these online communities makes it impossible for the moderators to engage in all conversations; yet, some conversations need their expertise. Our work explores low-cost text classification methods to this new domain of determining whether a thread in an online health forum needs moderators' help.<h4>Methods</h4>We employed a binary classifier on WebMD's online diabetes comm ...[more]

PMID: 24025513

Similar Datasets

Project description:Online health community (OHC) moderators help facilitate conversations and provide information to members. However, the necessity of the moderator in helping members achieve goals by providing the support they need remains unclear, with some prior research suggesting that moderation is unnecessary or even harmful for close-knit OHCs. Similarly, members' perceptions of moderator roles are underexplored. Starting January of 2013, WebMD moderators stopped working for WebMD communities. This event provided an opportunity for us to study the perceived role of moderators in OHCs.We examine the OHC members' perception on OHC moderators by studying their reactions toward the departure of moderators in their communities. We also analyzed the relative posting activity on OHCs before and after the departure of moderators from the communities among all members and those who discussed moderators' departures.We applied a mixed-methods approach to study the posts of all 55 moderated WebMD communities by querying the terms relating to discussions surrounding moderators' disappearance from the WebMD community. We performed open and axial coding and affinity diagramming to thematically analyze patients' reactions to the disappeared moderators. The number of posts and poster groups (members and moderators) were analyzed over time to understand posting patterns around moderators' departure.Of 821 posts retrieved under 95 threads, a total of 166 open codes were generated. The codes were then grouped into 2 main themes with 6 total subthemes. First, patients attempted to understand why moderators had left and what could be done to fill the void left by the missing moderators. During these discussions, the posts revealed that patients believed that moderators played critical roles in the communities by making the communities vibrant and healthy, finding solutions, and giving medical information. Some patients felt personally attached with moderators, expressing they would cease their community participation. On the other hand, patients also indicated that moderators were not useful or sometimes even harmful for peer interactions. The overall communities' posting activity, which was already in decline, showed no significant difference before and after the moderators' departure. In fact, the overall posting activities of the communities were declining well before the moderators' departure. These declining posting activities might be the reason why WebMD removed the moderators.Compassionate moderators who provide medical expertise, control destructive member posts, and help answer questions can provide important support for patient engagement in OHCs. Moderators are in general received positively by community members and do not appear to interfere with peer interactions. Members are well aware of the possibility of misinformation spreading in OHCs. Further investigation into the attitudes of less vocal community members should be conducted.

Project description:Automated monitoring of websites that trade wildlife is increasingly necessary to inform conservation and biosecurity efforts. However, e-commerce and wildlife trading websites can contain a vast number of advertisements, an unknown proportion of which may be irrelevant to researchers and practitioners. Given that many wildlife-trade advertisements have an unstructured text format, automated identification of relevant listings has not traditionally been possible, nor attempted. Other scientific disciplines have solved similar problems using machine learning and natural language processing models, such as text classifiers. Here, we test the ability of a suite of text classifiers to extract relevant advertisements from wildlife trade occurring on the Internet. We collected data from an Australian classifieds website where people can post advertisements of their pet birds (n = 16.5k advertisements). We found that text classifiers can predict, with a high degree of accuracy, which listings are relevant (ROC AUC ≥ 0.98, F1 score ≥ 0.77). Furthermore, in an attempt to answer the question 'how much data is required to have an adequately performing model?', we conducted a sensitivity analysis by simulating decreases in sample sizes to measure the subsequent change in model performance. From our sensitivity analysis, we found that text classifiers required a minimum sample size of 33% (c. 5.5k listings) to accurately identify relevant listings (for our dataset), providing a reference point for future applications of this sort. Our results suggest that text classification is a viable tool that can be applied to the online trade of wildlife to reduce time dedicated to data cleaning. However, the success of text classifiers will vary depending on the advertisements and websites, and will therefore be context dependent. Further work to integrate other machine learning tools, such as image classification, may provide better predictive abilities in the context of streamlining data processing for wildlife trade related online data.

Project description:BACKGROUND:Patient education materials given to breast cancer survivors may not be a good fit for their information needs. Needs may change over time, be forgotten, or be misreported, for a variety of reasons. An automated content analysis of survivors' postings to online health forums can identify expressed information needs over a span of time and be repeated regularly at low cost. Identifying these unmet needs can guide improvements to existing education materials and the creation of new resources. OBJECTIVE:The primary goals of this project are to assess the unmet information needs of breast cancer survivors from their own perspectives and to identify gaps between information needs and current education materials. METHODS:This approach employs computational methods for content modeling and supervised text classification to data from online health forums to identify explicit and implicit requests for health-related information. Potential gaps between needs and education materials are identified using techniques from information retrieval. RESULTS:We provide a new taxonomy for the classification of sentences in online health forum data. 260 postings from two online health forums were selected, yielding 4179 sentences for coding. After annotation of data and training alternative one-versus-others classifiers, a random forest-based approach achieved F1 scores from 66% (Other, dataset2) to 90% (Medical, dataset1) on the primary information types. 136 expressions of need were used to generate queries to indexed education materials. Upon examination of the best two pages retrieved for each query, 12% (17/136) of queries were found to have relevant content by all coders, and 33% (45/136) were judged to have relevant content by at least one. CONCLUSIONS:Text from online health forums can be analyzed effectively using automated methods. Our analysis confirms that breast cancer survivors have many information needs that are not covered by the written documents they typically receive, as our results suggest that at most a third of breast cancer survivors' questions would be addressed by the materials currently provided to them.

Project description:BACKGROUND:Regional population management (PM) health initiatives require insight into experienced quality of care at the regional level. Unsolicited online provider ratings have shown potential for this use. This study explored the addition of comments accompanying unsolicited online ratings to regional analyses. OBJECTIVE:The goal was to create additional insight for each PM initiative as well as overall comparisons between these initiatives by attempting to determine the reasoning and rationale behind a rating. METHODS:The Dutch Zorgkaart database provided the unsolicited ratings from 2008 to 2017 for the analyses. All ratings included both quantitative ratings as well as qualitative text comments. Nine PM regions were used to aggregate ratings geographically. Sentiment analyses were performed by categorizing ratings into negative, neutral, and positive ratings. Per category, as well as per PM initiative, word frequencies (ie, unigrams and bigrams) were explored. Machine learning-naïve Bayes and random forest models-was applied to identify the most important predictors for rating overall sentiment and for identifying PM initiatives. RESULTS:A total of 449,263 unsolicited ratings were available in the Zorgkaart database: 303,930 positive ratings, 97,739 neutral ratings, and 47,592 negative ratings. Bigrams illustrated that feeling like not being "taken seriously" was the dominant bigram in negative ratings, while bigrams in positive ratings were mostly related to listening, explaining, and perceived knowledge. Comparing bigrams between PM initiatives showed a lot of overlap but several differences were identified. Machine learning was able to predict sentiments of comments but was unable to distinguish between specific PM initiatives. CONCLUSIONS:Adding information from text comments that accompany online ratings to regional evaluations provides insight for PM initiatives into the underlying reasons for ratings. Text comments provide useful overarching information for health care policy makers but due to a lot of overlap, they add little region-specific information. Specific outliers for some PM initiatives are insightful.

Project description:BACKGROUND:An online health community (OHC) is a novel sharing channel through which doctors share professional health care knowledge with patients. While doctors have the authority to protect their patients' privacy in OHCs, we have limited information on how doctors' privacy protection choices affect their professional health care knowledge sharing with patients. OBJECTIVE:We examined the relationship between privacy protection and professional health care knowledge sharing in OHCs. Specifically, we examined the effects of privacy protection settings in an OHC on doctors' interactive professional health care knowledge sharing and searching professional health care knowledge sharing (two dimensions of professional health care knowledge sharing). Moreover, we explored how such effects differ across different levels of disease stigma. METHODS:We collected the monthly panel data of 19,456 doctors from Good Doctor, one of the largest OHCs in China, from January 2008 to April 2016. A natural experimental empirical study with difference-in-difference analysis was conducted to test our hypotheses. The time fixed effect and the individual fixed effect were both considered to better identify the effects of a privacy protection setting on professional health care knowledge sharing. Additionally, a cross-sectional analysis was performed for a robust check. RESULTS:The results indicate that the privacy protection setting has a significant positive effect on interactive professional health care knowledge sharing (?=.123, P<.001). However, the privacy protection setting has a significant negative effect on searching professional health care knowledge sharing (?=-.225, P=.05). Moreover, we found that high disease stigma positively impacts the effect of privacy protection on interactive professional health care knowledge sharing (coefficients are in the same valence) and negatively impacts the effects of privacy protection on searching professional health care knowledge sharing (coefficients are in the reverse valence). CONCLUSIONS:Privacy protection has a bilateral effect on professional health care knowledge sharing (ie, a positive effect on interactive professional health care knowledge sharing and a negative effect on searching professional health care knowledge sharing). Such bilateral switches of professional health care knowledge sharing call for a balanced state in conjunction with practical implications. This research also identifies a moderate effect of disease stigma on privacy protection settings and professional health care knowledge sharing.

Project description:BackgroundThe role of online health communities (OHCs) in patient empowerment is growing and has been increasingly studied in recent years. Research has focused primarily on individualistic conception of patients' empowerment, with much less attention paid to the role of OHCs in the development of patients' collective empowerment. Although OHCs have immense potential for empowerment that goes beyond the individual, the concept and scale of collective empowerment in OHCs have not yet been developed or validated.ObjectiveThis study aimed to develop an instrument for measuring collective empowerment in online health communities (CE-OHC) and to test its quality by investigating its factorial structure, reliability, construct validity, and predictive validity.MethodsThe CE-OHC scale was developed according to a strict methodology for developing valid and reliable scales. An initial set of 20 items was first tested in the pilot study conducted in 2016 using a sample of 280 registered users of Slovenia's largest OHC. A refined version with 11 items was tested in the main study conducted in 2018 on a random sample of 30,000 registered users of the same OHC. The final sample comprised 784 users. Exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) were used to investigate the factorial structure, discriminant validity, and convergent validity of the scale. Cronbach alpha coefficient was used to determine the CE-OHC scale's internal consistency. To establish the predictive validity, ordinary least squares regression was performed to test the role of CE-OHC in users' civic participation.ResultsThe EFA resulted in a two-factor solution, and the two factors-knowledge of resources and resource mobilization for collective action-together explain 63.8% of the variance. The second-order CFA demonstrated a good fit to the data (root mean square error of approximation=0.07) and the scale had a good internal consistency (alpha=.86). Although evidence of the scale's convergent validity was partially provided, discriminant validity of the scale remained unconfirmed. Overall, CE-OHC was confirmed to be a predictor of users' civic participation, but the influence was somewhat weak and inconsistent across two subscales.ConclusionsThe proposed CE-OHC scale is a reliable and relatively valid instrument and serves as a good baseline to advance the measurement of collective empowerment in OHC contexts. This is the first scale developed for this purpose, and future research should focus on the development of a clear nomological network of the collective empowerment construct in relation to the OHC settings.

Dataset Information

Text classification for assisting moderators in online health communities.

Objectives

Methods

Results

Discussion

Conclusion

Publications

Text classification for assisting moderators in online health communities.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets