Statistics-Based Outlier Detection and Correction Method for Amazon Customer Reviews.
Ontology highlight
ABSTRACT:
SUBMITTER: Chatterjee I
PROVIDER: S-EPMC8700267 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Customer satisfaction and their positive sentiments are some of the various goals for successful companies. However, analyzing customer reviews to predict accurate sentiments have been proven to be challenging and time-consuming due to high volumes of collected data from various sources. Several researchers approach this with algorithms, methods, and models. These include machine learning and deep learning (DL) methods, unigram and skip-gram based algorithms, as well as the Artificial Neural Network (ANN) and bag-of-word (BOW) regression model. Studies and research have revealed incoherence in polarity, model overfitting and performance issues, as well as high cost in data processing. This experiment was conducted to solve these revealing issues, by building a high performance yet cost-effective model for predicting accurate sentiments from large datasets containing customer reviews. This model uses the fastText library from Facebook's AI research (FAIR) Lab, as well as the traditional Linear Support Vector Machine (LSVM) to classify text and word embedding. Comparisons of this model were also done with the author's a custom multi-layer Sentiment Analysis (SA) Bi-directional Long Short-Term Memory (SA-BLSTM) model. The proposed fastText model, based on results, obtains a higher accuracy of 90.71% as well as 20% in performance compared to LSVM and SA-BLSTM models.
| S-EPMC8725657 | biostudies-literature
Project description:BackgroundOutliers, data points that significantly deviate from the norm, can have a substantial impact on statistical inference and provide valuable insights in data analysis. Multiple methods have been developed for outlier detection, however, almost all available approaches fail to consider the spatial dependence and heterogeneity in spatial data. Spatial data has diverse formats and semantics, requiring specialized outlier detection methodology to handle these unique properties. For now, there is limited research exists on robust spatial outlier detection methods designed specifically under the spatial error model (SEM) structure.MethodWe propose the Spatial-Θ-Iterative Procedure for Outlier Detection (Spatial-Θ-IPOD), which utilizes a mean-shift vector to identify outliers within the SEM. Our method enables an effective detection of spatial outliers while also providing robust coefficient estimates. To assess the performance of our approach, we conducted extensive simulations and applied it to a real-world empirical study using life expectancy data from multiple countries.ResultsSimulation results showed that the masking and JD (Joint Detection) indicators of our Spatial-Θ-IPOD method outperformed several commonly used methods, even in high-dimensional scenarios, demonstrating stable performance. Conversely, the Θ-IPOD method proved to be ineffective in detecting outliers when spatial correlation was present. Moreover, our model successfully provided reliable coefficient estimation alongside outlier detection. The proposed method consistently outperformed other models (both robust and non-robust) in most cases. In the empirical study, our proposed model successfully detected outliers and provided valuable insights in the modeling process.ConclusionsOur proposed Spatial-Θ-IPOD offers an effective solution for detecting spatial outliers for SEM while providing robust coefficient estimates. Notably, our approach showcases its relative superiority even in the presence of high leverage points. By successfully identifying outliers, our method enhances the overall understanding of the data and provides valuable insights for further analysis.
| S-EPMC11323683 | biostudies-literature
Project description:This document describes a set of customer feedback data concerning the Post Bank. We collected data from 16,659 feedback lines using the Beautiful Soup package from the authoritative site banki.ru is selected as the source of data for collection. The dataset is compiled to monitor the level of trust of bank customers in its banking service. The data presents text reviews for 2013 - 2019 and includes, with or without ratings. Scientists can predict feedback ratings with an empty value in the future. We added additional columns to the dataset with official comments of bank employees, as well as values for the fog-index by Gunning parameter, which is used for the readability of the text. The data can be useful for customer service managers to identify problems in customer service and solve these problems, to assess the dynamics of the appearance of positive and negative reviews of bank customers.
| S-EPMC7452671 | biostudies-literature
Project description:MotivationMicroarray experiments can be used to help study the role of chromosomal translocation in cancer development through cancer outlier detection. The aim is to identify genes that are up- or down-regulated in a subset of cancer samples in comparison to normal samples.ResultsWe propose a likelihood-based approach which targets detecting the change of point in mean expression intensity in the group of cancer samples. A desirable property of the proposed approach is the availability of theoretical significance-level results. Simulation studies showed that the performance of the proposed approach is appealing in terms of both detection power and false discovery rate. And the real data example also favored the likelihood-based approach in terms of the biological relevance of the results.AvailabilityR code to implement the proposed method in the statistical package R is available at: http://odin.mdacc.tmc.edu/~jhhu/cod-analysis/.
| S-EPMC2732272 | biostudies-literature
Project description:BackgroundReal-time PCR has recently become the technique of choice for absolute and relative nucleic acid quantification. The gold standard quantification method in real-time PCR assumes that the compared samples have similar PCR efficiency. However, many factors present in biological samples affect PCR kinetic, confounding quantification analysis. In this work we propose a new strategy to detect outlier samples, called SOD.ResultsRichards function was fitted on fluorescence readings to parameterize the amplification curves. There was not a significant correlation between calculated amplification parameters (plateau, slope and y-coordinate of the inflection point) and the Log of input DNA demonstrating that this approach can be used to achieve a "fingerprint" for each amplification curve. To identify the outlier runs, the calculated parameters of each unknown sample were compared to those of the standard samples. When a significant underestimation of starting DNA molecules was found, due to the presence of biological inhibitors such as tannic acid, IgG or quercitin, SOD efficiently marked these amplification profiles as outliers. SOD was subsequently compared with KOD, the current approach based on PCR efficiency estimation. The data obtained showed that SOD was more sensitive than KOD, whereas SOD and KOD were equally specific.ConclusionOur results demonstrated, for the first time, that outlier detection can be based on amplification shape instead of PCR efficiency. SOD represents an improvement in real-time PCR analysis because it decreases the variance of data thus increasing the reliability of quantification.
| S-EPMC2873533 | biostudies-literature
Project description:MotivationTraditional gene expression analysis techniques capture an average gene expression state across sample replicates. However, the average signal across replicates will not capture activated gene networks in different states across replicates. For example, if a particular gene expression network is activated within a subset or all sample replicates, yet the activation state across the sample replicates differs by the specific genes activated in each replicate, the activation of this network will be washed out by averaging across replicates. This situation is likely to occur in single cell gene expression experiments or in noisy experimental settings where a small sub-population of cells contributes to the gene expression signature of interest.Results and implementationIn this light, we developed a novel network-based approach which considers gene expression within each replicate across its entire gene expression profile, and identifies outliers across replicates. The power of this method is demonstrated by its ability to enrich for distant metastasis related genes derived from noisy expression data of CD44+CD24-/low tumor initiating cells.
| S-EPMC2800353 | biostudies-literature
Project description:BackgroundEpidemiologic and medical studies often rely on evaluators to obtain measurements of exposures or outcomes for study participants, and valid estimates of associations depends on the quality of data. Even though statistical methods have been proposed to adjust for measurement errors, they often rely on unverifiable assumptions and could lead to biased estimates if those assumptions are violated. Therefore, methods for detecting potential 'outlier' evaluators are needed to improve data quality during data collection stage.MethodsIn this paper, we propose a two-stage algorithm to detect 'outlier' evaluators whose evaluation results tend to be higher or lower than their counterparts. In the first stage, evaluators' effects are obtained by fitting a regression model. In the second stage, hypothesis tests are performed to detect 'outlier' evaluators, where we consider both the power of each hypothesis test and the false discovery rate (FDR) among all tests. We conduct an extensive simulation study to evaluate the proposed method, and illustrate the method by detecting potential 'outlier' audiologists in the data collection stage for the Audiology Assessment Arm of the Conservation of Hearing Study, an epidemiologic study for examining risk factors of hearing loss in the Nurses' Health Study II.ResultsOur simulation study shows that our method not only can detect true 'outlier' evaluators, but also is less likely to falsely reject true 'normal' evaluators.ConclusionsOur two-stage 'outlier' detection algorithm is a flexible approach that can effectively detect 'outlier' evaluators, and thus data quality can be improved during data collection stage.
| S-EPMC10391872 | biostudies-literature
Project description:BackgroundRecent cancer genomic studies have generated detailed molecular data on a large number of cancer patients. A key remaining problem in cancer genomics is the identification of driver genes.ResultsWe propose BetweenNet, a computational approach that integrates genomic data with a protein-protein interaction network to identify cancer driver genes. BetweenNet utilizes a measure based on betweenness centrality on patient specific networks to identify the so-called outlier genes that correspond to dysregulated genes for each patient. Setting up the relationship between the mutated genes and the outliers through a bipartite graph, it employs a random-walk process on the graph, which provides the final prioritization of the mutated genes. We compare BetweenNet against state-of-the art cancer gene prioritization methods on lung, breast, and pan-cancer datasets.ConclusionsOur evaluations show that BetweenNet is better at recovering known cancer genes based on multiple reference databases. Additionally, we show that the GO terms and the reference pathways enriched in BetweenNet ranked genes and those that are enriched in known cancer genes overlap significantly when compared to the overlaps achieved by the rankings of the alternative methods.
| S-EPMC7877041 | biostudies-literature
Project description:Decades of research strongly suggest that the genetic etiology of autism spectrum disorders (ASDs) is heterogeneous. However, most published studies focus on group differences between cases and controls. In contrast, we hypothesized that the heterogeneity of the disorder could be characterized by identifying pathways for which individuals are outliers rather than pathways representative of shared group differences of the ASD diagnosis.Two previously published blood gene expression data sets--the Translational Genetics Research Institute (TGen) dataset (70 cases and 60 unrelated controls) and the Simons Simplex Consortium (Simons) dataset (221 probands and 191 unaffected family members)--were analyzed. All individuals of each dataset were projected to biological pathways, and each sample's Mahalanobis distance from a pooled centroid was calculated to compare the number of case and control outliers for each pathway.Analysis of a set of blood gene expression profiles from 70 ASD and 60 unrelated controls revealed three pathways whose outliers were significantly overrepresented in the ASD cases: neuron development including axonogenesis and neurite development (29% of ASD, 3% of control), nitric oxide signaling (29%, 3%), and skeletal development (27%, 3%). Overall, 50% of cases and 8% of controls were outliers in one of these three pathways, which could not be identified using group comparison or gene-level outlier methods. In an independently collected data set consisting of 221 ASD and 191 unaffected family members, outliers in the neurogenesis pathway were heavily biased towards cases (20.8% of ASD, 12.0% of control). Interestingly, neurogenesis outliers were more common among unaffected family members (Simons) than unrelated controls (TGen), but the statistical significance of this effect was marginal (Chi squared P < 0.09).Unlike group difference approaches, our analysis identified the samples within the case and control groups that manifested each expression signal, and showed that outlier groups were distinct for each implicated pathway. Moreover, our results suggest that by seeking heterogeneity, pathway-based outlier analysis can reveal expression signals that are not apparent when considering only shared group differences.
| S-EPMC3849321 | biostudies-literature
Project description:Outliers are data points that significantly deviate from other data points in a data set because of different mechanisms or unusual processes. Outlier detection is one of the intensively studied research topics for identification of novelties, frauds, anomalies, deviations or exceptions in addition to its use for data cleansing in data science. In this study, we propose two novel outlier detection approaches using the typicality degrees which are the partitioning result of unsupervised possibilistic clustering algorithms. The proposed approaches are based on finding the atypical data points below a predefined threshold value, a possibilistic level for evaluating a point as an outlier. The experiments on the synthetic and real data sets showed that the proposed approaches can be successfully used to detect outliers without considering the structure and distribution of the features in multidimensional data sets.
| S-EPMC9575855 | biostudies-literature