Assessing the reliability of automatic sentiment analysis tools on rating the sentiment of reviews of NHS dental practices in England.
Ontology highlight
ABSTRACT: Background
Online reviews may act as a rich source of data to assess the quality of dental practices. Assessing the content and sentiment of reviews on a large scale is time consuming and expensive. Automation of the process of assigning sentiment to big data samples of reviews may allow for reviews to be used as Patient Reported Experience Measures for primary care dentistry.Aim
To assess the reliability of three different online sentiment analysis tools (Amazon Comprehend DetectSentiment API (ACDAPI), Google and Monkeylearn) at assessing the sentiment of reviews of dental practices working on National Health Service contracts in the United Kingdom.Methods
A Python 3 script was used to mine 15800 reviews from 4803 unique dental practices on the NHS.uk websites between April 2018 -March 2019. A random sample of 270 reviews were rated by the three sentiment analysis tools. These reviews were rated by 3 blinded independent human reviewers and a pooled sentiment score was assigned. Kappa statistics and polychoric evalutaiton were used to assess the level of agreement. Disagreements between the automated and human reviewers were qualitatively assessed.Results
There was good agreement between the sentiment assigned to reviews by the human reviews and ACDAPI (k = 0.660). The Google (k = 0.706) and Monkeylearn (k = 0.728) showed slightly better agreement at the expense of usability on a massive dataset. There were 33 disagreements in rating between ACDAPI and human reviewers, of which n = 16 were due to syntax errors, n = 10 were due to misappropriation of the strength of conflicting emotions and n = 7 were due to a lack of overtly emotive language in the text.Conclusions
There is good agreement between the sentiment of an online review assigned by a group of humans and by cloud-based sentiment analysis. This may allow the use of automated sentiment analysis for quality assessment of dental service provision in the NHS.
SUBMITTER: Byrne M
PROVIDER: S-EPMC8673612 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.
| S-EPMC10248995 | biostudies-literature
Project description:Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy. A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work. Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field. Supplementary Information The online version contains supplementary material available at 10.1186/s12879-023-08304-x.
| S-EPMC10247272 | biostudies-literature
Project description:Customer reviews are valuable resources containing customer opinions and sentiments toward the product. The reviews are informative but can be quite lengthy or may contain repetitive information calling for opinion summarization systems that retain only the significant opinion information from the review. Abstractive summarization is a form of text summarization that generates a summary mimicking a human-written summary [1]. When pretrained language models are finetuned for abstractive review summarization, there usually occurs a problem known as the 'domain shift', because the source and target domains exhibit data from varying distributions [2]. This issue results in performance degradation of the model at the target end. This paper contributes a data package comprising of an annotated abstractive summarization dataset (annotated_abs_summ) of airline reviews having 500 reviews and abstractive summary pairs, a dataset (review_titles_data) consisting of 7079 reviews and review title pairs for review title generatioon or domain adaptive training [3] to address the domain shift problem for abstractive opinion summarization and, an annotated reviews dataset (annotated_sentiment) for rating-based sentiment classification. All datasets have been collected from the Skytrax Review Portal via web scraping using Python programming language. The datasets have several potential use cases. The abstractive summarization dataset can serve as a benchmark dataset for airline review summarization. The dataset for domain adaptive training can be used as a standalone dataset for review title generation. The dataset for sentiment analysis is multipurpose having columns like user rating and recommendation value, that can be used for statistical analysis like finding correlation between these data items as well as for other Natural Language Processing (NLP) tasks like predicting rating or recommendation value from the customer reviews. The datasets can be extended using various data augmentation techniques [4,5]. Moreover, the datasets are related and can be collectively used to develop a multi-task learning model [6] for better learning efficiency and improved performance.
| S-EPMC10504493 | biostudies-literature
Project description:BackgroundUnderstanding the collective sociotechnical experiences of the staff in dental practices with automatic washer disinfectors (AWDs) may help shape future strategies that encourage the transition towards best practice in dental instrument decontamination and reprocessing.ObjectivesTo find the emerging themes that reflect the experience of working with an AWD in dental practice. To compare the experience of practice owners to that of the dental nurses.MethodsA qualitative semi-structured interview-based methodology was applied using a convenience sample of dentists and dental nurses. Verbatim transcripts of audio recordings underwent thematic analysis.ResultsDental practice owners and dental nurses were interviewed. Four themes were common to both groups: impact on daily routine; mundane technology; the decontamination cycle; and safety. Three themes were unique to the dentists: impact on the business; professionalism; and external motivators for change.DiscussionAWDs are a mundane form of technology that dental nurses find simple to operate. The extended time it takes to reprocess instruments using an AWD means that dental nurses must adapt their daily working practices to accommodate this. Initial funding to purchase an AWD, especially where there is a professional expert leading a campaign championing their use, can be effective.
| S-EPMC6346326 | biostudies-literature
Project description:BackgroundMental health in general, and depression in particular, remain undertreated conditions. Mobile health (mHealth) apps offer tremendous potential to overcome the barriers to accessing mental health care and millions of depression apps have been installed and used. However, little is known about the effect of these apps on a potentially vulnerable user population and the emotional reactions that they generate, even though emotions are a key component of mental health. App reviews, spontaneously posted by the users on app stores, offer up-to-date insights into the experiences and emotions of this population and are increasingly decisive in influencing mHealth app adoption.ObjectiveThis study aims to investigate the emotional reactions of depression app users to different app features by systematically analyzing the sentiments expressed in app reviews.MethodsWe extracted 3261 user reviews of depression apps. The 61 corresponding apps were categorized by the features they offered (psychoeducation, medical assessment, therapeutic treatment, supportive resources, and entertainment). We then produced word clouds by features and analyzed the reviews using the Linguistic Inquiry Word Count 2015 (Pennebaker Conglomerates, Inc), a lexicon-based natural language analytical tool that analyzes the lexicons used and the valence of a text in 4 dimensions (authenticity, clout, analytic, and tone). We compared the language patterns associated with the different features of the underlying apps.ResultsThe analysis highlighted significant differences in the sentiments expressed for the different features offered. Psychoeducation apps exhibited more clout but less authenticity (ie, personal disclosure). Medical assessment apps stood out for the strong negative emotions and the relatively negative ratings that they generated. Therapeutic treatment app features generated more positive emotions, even though user feedback tended to be less authentic but more analytical (ie, more factual). Supportive resources (connecting users to physical services and people) and entertainment apps also generated fewer negative emotions and less anxiety.ConclusionsDevelopers should be careful in selecting the features they offer in their depression apps. Medical assessment features may be riskier as users receive potentially disturbing feedback on their condition and may react with strong negative emotions. In contrast, offering information, contacts, or even games may be safer starting points to engage people with depression at a distance. We highlight the necessity to differentiate how mHealth apps are assessed and vetted based on the features they offer. Methodologically, this study points to novel ways to investigate the impact of mHealth apps and app features on people with mental health issues. mHealth apps exist in a rapidly changing ecosystem that is driven by user satisfaction and adoption decisions. As such, user perceptions are essential and must be monitored to ensure adoption and avoid harm to a fragile population that may not benefit from traditional health care resources.
| S-EPMC8715360 | biostudies-literature
Project description:Sentiment lexicon plays a vital role in lexicon-based sentiment analysis. The lexicon-based method is often preferred because it leads to more explainable answers in comparison with many machine learning-based methods. But, semantic orientation of a word depends on its domain. Hence, a general-purpose sentiment lexicon may gives sub-optimal performance compare with a domain-specific lexicon. However, it is challenging to manually generate a domain-specific sentiment lexicon for each domain. Still, it is impractical to generate complete sentiment lexicon for a domain from a single corpus. To this end, we propose an approach to automatically generate a domain-specific sentiment lexicon using a vector model enriched by weights. Importantly, we propose an incremental approach for updating an existing lexicon to either the same domain or different domain (domain-adaptation). Finally, we discuss how to incorporate sentiment lexicons information in neural models (word embedding) for better performance.
| S-EPMC7148022 | biostudies-literature
Project description:BackgroundAs COVID-19 death rates have risen and health-care systems have experienced increased demand, national testing strategies have come under scrutiny. Utilising qualitative interview data from a larger COVID-19 study, this paper provides insights into influences on and the enactment of national COVID-19 testing strategies for health care workers (HCWs) in English NHS settings during wave one of the COVID-19 pandemic (March-August 2020). Through the findings we aim to inform learning about COVID-19 testing policies and practices; and to inform future pandemic diagnostic preparedness.MethodsA remote qualitative, semi-structured longitudinal interview method was employed with a purposive snowball sample of senior scientific advisors to the UK Government on COVID-19, and HCWs employed in NHS primary and secondary health care settings in England. Twenty-four interviews from 13 participants were selected from the larger project dataset using a key term search, as not all of the transcripts contained references to testing. Framework analysis was informed by the non-adoption, abandonment, scale-up, spread, and sustainability of patient-facing health and care technologies implementation framework (NASSS) and by normalisation process theory (NPT).ResultsOur account highlights tensions between the communication and implementation of national testing developments; scientific advisor and HCW perceptions about infectiousness; and uncertainties about the responsibility for testing and its implications at the local level.ConclusionsConsideration must be given to the implications of mass NHS staff testing, including the accuracy of information communicated to HCWs; how HCWs interpret, manage, and act on testing guidance; and the influence these have on health care organisations and services.
| S-EPMC8224254 | biostudies-literature
Project description:Introduction Advanced NHS restorative dentistry services are an important aspect of patient care. Managed clinical networks (MCNs) have been proposed as a future model of care.Aim To assess general dental practitioners' (GDPs') satisfaction with the current provision of advanced NHS restorative dentistry services and assess their views on MCNs.Methods A self-administered, online survey was distributed by the Northern Dental Practice Based Research Network and was shared on social media.Results In total, 108 responses were received from GDPs working in England; 55% in the North East. GDPs felt current services for periodontics, endodontics, tooth surface loss and temporomandibular disorders were the most important and were most unsatisfied with periodontics, endodontics and tooth surface loss. The most important barriers to current referral practice were previous referral rejections and the return of costly treatment plans. Opinions were favourable towards the proposed MCNs, with one-third of participants feeling they were already suitable to apply to deliver Level 2 services. There was a notable imbalance between sexes, with fewer female dentists (57%) registering interest in joining an MCN compared to their male counterparts (76%) and female dentists also identifying more barriers to join MCNs, including access to suitable training. Recent graduates were least likely to cite a lack of time as a barrier to engaging with MCNs.Conclusions GDPs are currently unsatisfied with advanced NHS restorative dentistry services but appear keen to engage with MCNs. The survey identified important insights which may help healthcare planners develop services.
| S-EPMC8896073 | biostudies-literature
Project description:Customer satisfaction and their positive sentiments are some of the various goals for successful companies. However, analyzing customer reviews to predict accurate sentiments have been proven to be challenging and time-consuming due to high volumes of collected data from various sources. Several researchers approach this with algorithms, methods, and models. These include machine learning and deep learning (DL) methods, unigram and skip-gram based algorithms, as well as the Artificial Neural Network (ANN) and bag-of-word (BOW) regression model. Studies and research have revealed incoherence in polarity, model overfitting and performance issues, as well as high cost in data processing. This experiment was conducted to solve these revealing issues, by building a high performance yet cost-effective model for predicting accurate sentiments from large datasets containing customer reviews. This model uses the fastText library from Facebook's AI research (FAIR) Lab, as well as the traditional Linear Support Vector Machine (LSVM) to classify text and word embedding. Comparisons of this model were also done with the author's a custom multi-layer Sentiment Analysis (SA) Bi-directional Long Short-Term Memory (SA-BLSTM) model. The proposed fastText model, based on results, obtains a higher accuracy of 90.71% as well as 20% in performance compared to LSVM and SA-BLSTM models.
| S-EPMC8725657 | biostudies-literature
Project description:Regional languages are being used more frequently in online platforms as a result of the expanding use of digital technology. Understanding user opinions on social media platforms, forums, blogs, and other digital platforms that employ Indian regional languages has become significant due to their role in various applications. Research on sentiment analysis of Indian regional language texts suffers due to the unavailability of available regional language datasets. The curated Malayalam Aspect Based Sentiment Analysis (MABSA) dataset is a labeled dataset for Aspect Based Sentiment Analysis (ABSA) on the Indian regional language Malayalam over the movie review domain. Malayalam movie reviews, an excellent source of text data for ABSA, are collected from an online survey using Google form and manually collecting reviews from three social media platforms: IMDb, Facebook, and YouTube. Nine target aspects were identified, and three annotators annotated the dataset based on the sentiment polarity of each aspect. A total of 4000 reviews were collected, and a total of 7507 aspects are identified in the reviews. Spearman's correlation and Fleiss Kappa Index are used to analyze the annotated dataset's correlation. It has been found that the high correlation between the annotators implies that the MABSA dataset is of gold standard.
| S-EPMC10415832 | biostudies-literature