Beyond Benford's Law: Distinguishing Noise from Chaos.
Ontology highlight
ABSTRACT:
SUBMITTER: Li Q
PROVIDER: S-EPMC4452586 | biostudies-literature | 2015
REPOSITORIES: biostudies-literature
Publications
Beyond Benford's Law: Distinguishing Noise from Chaos.
Li Qinglei Q Fu Zuntao Z Yuan Naiming N
PloS one 20150601 6
Determinism and randomness are two inherent aspects of all physical processes. Time series from chaotic systems share several features identical with those generated from stochastic processes, which makes them almost undistinguishable. In this paper, a new method based on Benford's law is designed in order to distinguish noise from chaos by only information from the first digit of considered series. By applying this method to discrete data, we confirm that chaotic data indeed can be distinguishe ...[more]
Similar Datasets
Project description:A recently proposed methodology called the Horizontal Visibility Graph (HVG) [Luque et al., Phys. Rev. E., 80, 046103 (2009)] that constitutes a geometrical simplification of the well known Visibility Graph algorithm [Lacasa et al., Proc. Natl. Sci. U.S.A. 105, 4972 (2008)], has been used to study the distinction between deterministic and stochastic components in time series [L. Lacasa and R. Toral, Phys. Rev. E., 82, 036120 (2010)]. Specifically, the authors propose that the node degree distribution of these processes follows an exponential functional of the form [Formula: see text], in which [Formula: see text] is the node degree and [Formula: see text] is a positive parameter able to distinguish between deterministic (chaotic) and stochastic (uncorrelated and correlated) dynamics. In this work, we investigate the characteristics of the node degree distributions constructed by using HVG, for time series corresponding to [Formula: see text] chaotic maps, 2 chaotic flows and [Formula: see text] different stochastic processes. We thoroughly study the methodology proposed by Lacasa and Toral finding several cases for which their hypothesis is not valid. We propose a methodology that uses the HVG together with Information Theory quantifiers. An extensive and careful analysis of the node degree distributions obtained by applying HVG allow us to conclude that the Fisher-Shannon information plane is a remarkable tool able to graphically represent the different nature, deterministic or stochastic, of the systems under study.
| S-EPMC4172653 | biostudies-literature
Project description:Benford's Law describes the finding that the distribution of leading (or leftmost) digits of innumerable datasets follows a well-defined logarithmic trend, rather than an intuitive uniformity. In practice this means that the most common leading digit is 1, with an expected frequency of 30.1%, and the least common is 9, with an expected frequency of 4.6%. Currently, the most common application of Benford's Law is in detecting number invention and tampering such as found in accounting-, tax-, and voter-fraud. We demonstrate that answers to end-of-chapter exercises in physics and chemistry textbooks conform to Benford's Law. Subsequently, we investigate whether this fact can be used to gain advantage over random guessing in multiple-choice tests, and find that while testbank answers in introductory physics closely conform to Benford's Law, the testbank is nonetheless secure against such a Benford's attack for banal reasons.
| S-EPMC4331362 | biostudies-literature
Project description:Integrity and trust in that integrity are fundamental to academic research. However, procedures for monitoring the trustworthiness of research, and for investigating cases where concern about possible data fraud have been raised are not well established. Here we suggest a practical approach for the investigation of work suspected of fraudulent data manipulation using Benford's Law. This should be of value to both individual peer-reviewers and academic institutions and journals. In this, we draw inspiration from well-established practices of financial auditing. We provide synthesis of the literature on tests of adherence to Benford's Law, culminating in advice of a single initial test for digits in each position of numerical strings within a dataset. We also recommend further tests which may prove useful in the event that specific hypotheses regarding the nature of data manipulation can be justified. Importantly, our advice differs from the most common current implementations of tests of Benford's Law. Furthermore, we apply the approach to previously-published data, highlighting the efficacy of these tests in detecting known irregularities. Finally, we discuss the results of these tests, with reference to their strengths and limitations.
| S-EPMC10088595 | biostudies-literature
Project description:Benford Law (BL) states that the occurrence of significant digits in many natural and human phenomena data sets are not uniformly scattered, as one could naively expect, but follow a logarithmic-type distribution. Here, we present a method that consists of the use of BL analysis over first and first-two digits, three statistical conformity tests - Z-statistics, Mean Absolute Deviation (MAD) and Chi-square (χ2) as well as the summation test which looks for excessively large numbers, having fraud detection as one of its application. We developed the method for fraud detection in the case of the Brazilian Bolsa Familia welfare program. In this case, we submitted four periods of Brazilian welfare program payments to the method with a dataset of 13,442,529 records. We provide a practical implementation of the method based on open-source R library released on a public repository. Furthermore, code implementation of the algorithm as well as datasets are freely available. Advantages of the algorithm are listed below: • The method was developed based on open source libraries • The technique is simple, rapid and ease of use • Easily applicable to other social welfare program auditing.
| S-EPMC8720889 | biostudies-literature
Project description:Working with a large temporal dataset spanning several decades often represents a challenging task, especially when the record is heterogeneous and incomplete. The use of statistical laws could potentially overcome these problems. Here we apply Benford's Law (also called the "First-Digit Law") to the traveled distances of tropical cyclones since 1842. The record of tropical cyclones has been extensively impacted by improvements in detection capabilities over the past decades. We have found that, while the first-digit distribution for the entire record follows Benford's Law prediction, specific changes such as satellite detection have had serious impacts on the dataset. The least-square misfit measure is used as a proxy to observe temporal variations, allowing us to assess data quality and homogeneity over the entire record, and at the same time over specific periods. Such information is crucial when running climatic models and Benford's Law could potentially be used to overcome and correct for data heterogeneity and/or to select the most appropriate part of the record for detailed studies.
| S-EPMC4496784 | biostudies-other
Project description:Typical responses of cortical neurons to identical sensory stimuli appear highly variable. It has thus been proposed that the cortex primarily uses a rate code. However, other studies have argued for spike-time coding under certain conditions. The potential role of spike-time coding is directly limited by the internally generated variability of cortical circuits, which remains largely unexplored. Here, we quantify this internally generated variability using a biophysical model of rat neocortical microcircuitry with biologically realistic noise sources. We find that stochastic neurotransmitter release is a critical component of internally generated variability, causing rapidly diverging, chaotic recurrent network dynamics. Surprisingly, the same nonlinear recurrent network dynamics can transiently overcome the chaos in response to weak feed-forward thalamocortical inputs, and support reliable spike times with millisecond precision. Our model shows that the noisy and chaotic network dynamics of recurrent cortical microcircuitry are compatible with stimulus-evoked, millisecond spike-time reliability, resolving a long-standing debate.
| S-EPMC6706377 | biostudies-literature
Project description:Benford's law states that the first digits of numbers in any natural dataset appear with defined frequencies. Pioneering, we use Benford distribution to analyse the geo-location of cities and their population in the majority of countries. We use distances in three dimensions: 1D between the population values, 2D between the cities, based on geo-coordinates of location, 3D between cities' location and population, which jointly reflects separation and mass of urban locations. We get four main findings. Firstly, we empirically show that mutual 3D socio-geo distances between cities and populations in most countries conform with Benford's law, and thus the urban geo-locations have natural spatial distribution. Secondly, we show empirically that the population of cities within countries follows the composition of gamma (1,1) distributions and that 1D distance between populations also conforms to Benford's law. Thirdly, we pioneer in replicating spatial natural distribution-we discover in simulation that a mixture of three pure point-patterns: clustered, ordered and random in proportions 15:3:2 makes the 2D spatial distribution Benford-like. Complex 3D Benford-like patterns can be built upon 2D (spatial) Benford distribution and gamma (1,1) distribution of cities' sizes. This finding enables generating 2D and 3D Benford distributions, which may replicate well the urban settlement. Fourth, we use historical settlement analysis to claim that the geo-location of cities and inhabitants worldwide followed the evolutionary process, resulting in natural Benford-like spatial distribution and to justify our statistical findings. Those results are very novel. This study develops new spatial distribution to simulate natural locations. It shows that evolutionary settlement patterns resulted in the natural location of cities, and historical distortions in urbanisation, even if persistent till now, are being evolutionary corrected.
| S-EPMC9584388 | biostudies-literature
Project description:We develop a Bayesian time-varying model that tracks periods at which conformance to Benford's Law is lower. Our methods are motivated by recent attempts to assess how the quality and homogeneity of large datasets may change over time by using the First-Digit Rule. We resort to a smooth multinomial logistic model which captures the dynamics governing the proportion of first digits, and apply the proposed model to global tropical cyclone tracks over the past two centuries. Our findings indicate that cumulative technological improvements may have only had a moderate influence on the homogeneity of the dataset, and hint that recent heterogeneity could be due to other drivers.
| S-EPMC6461246 | biostudies-literature
Project description:Benford's Law states that, in many real-world data sets, the frequency of numbers' first digits is predicted by the formula log(1 + (1/d)). Numbers beginning with a 1 occur roughly 30% of the time, and are six times more common than numbers beginning with a 9. We show that Benford's Law applies to the the frequency rank of words in English, German, French, Spanish, and Italian. We calculated the frequency rank of words in the Google Ngram Viewer corpora. Then, using the first significant digit of the frequency rank, we found the FSD distribution adhered to the expected Benford's Law distribution. Over a series of additional corpora from sources ranging from news to books to social media and across the languages studied, we consistently found adherence to Benford's Law. Furthermore, at the user-level on social media, we found Benford's Law holds for the vast majority of users' collected posts and significant deviations from Benford's Law tends to be a mark of spam bots.
| S-EPMC10501622 | biostudies-literature
Project description:BackgroundBreast cancer is the most frequent cancer in women all over the world, and it is one of the leading causes of cancer-related deaths in women. A pathologist's partiality for the last digit of a patient's name can lead to errors in the measurement of malignancies. This means that, rather than recording the exact measurement of a tumor, a pathologist might round it off to his preferred terminal digit.MethodsIt is a retrospective cross-sectional study in which data on primary tumor resection for 1000 breast cancer patients was obtained from KRL Hospital's patient directory from November 2016 to December 2020. The tumors were measured in cm to one decimal point along their longest dimension. Ki-67 markers were used to categorize the tumors into nine categories. Terminal digit preference was evaluated using Benford's law.ResultsThe recording of the Ki-67 index revealed evidence of pentameric preference. The numbers three, five, and six appeared more frequently in the histogram of the Ki-67 index distribution measured in percentage. The frequency of nine dropped dramatically. However, the influence of tumor size terminal digits on Ki-67 staining scores (low proliferative vs high proliferative) assessed using the Mann-Whitney U Test demonstrated that tumor size terminal digits had no significant effect on Ki-67 staining scores (p = 0.114).ConclusionThe Ki-67 index shows evidence of pentameric preference for digits three, five, and six. The frequency of nine has dropped dramatically. The influence of tumor size on terminal digits on staining scores (low proliferative vs. high proliferative) was assessed using the Mann-Whitney U Test.
| S-EPMC9289319 | biostudies-literature