Browse
Submit Data
Databases
API
Help

Dataset Information

0 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

Do neural nets learn statistical laws behind natural language?

ABSTRACT: The performance of deep learning in natural language processing has been spectacular, but the reasons for this success remain unclear because of the inherent complexity of deep learning. This paper provides empirical evidence of its effectiveness and of a limitation of neural networks for language engineering. Precisely, we demonstrate that a neural language model based on long short-term memory (LSTM) effectively reproduces Zipf's law and Heaps' law, two representative statistical properties underlying natural language. We discuss the quality of reproducibility and the emergence of Zipf's law and Heaps' law as training progresses. We also point out that the neural language model has a limitation in reproducing long-range correlation, another statistical property of natural language. This understanding could provide a direction for improving the architectures of neural networks.

SUBMITTER: Takahashi S

PROVIDER: S-EPMC5747447 | biostudies-literature | 2017

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Publications

Do neural nets learn statistical laws behind natural language?

Takahashi Shuntaro S Tanaka-Ishii Kumiko K

PloS one 20171229 12

The performance of deep learning in natural language processing has been spectacular, but the reasons for this success remain unclear because of the inherent complexity of deep learning. This paper provides empirical evidence of its effectiveness and of a limitation of neural networks for language engineering. Precisely, we demonstrate that a neural language model based on long short-term memory (LSTM) effectively reproduces Zipf's law and Heaps' law, two representative statistical properties un ...[more]

PMID: 29287076

Similar Datasets

Parsimonious neural networks learn interpretable physical laws.

Project description:Machine learning is playing an increasing role in the physical sciences and significant progress has been made towards embedding domain knowledge into models. Less explored is its use to discover interpretable physical laws from data. We propose parsimonious neural networks (PNNs) that combine neural networks with evolutionary optimization to find models that balance accuracy with parsimony. The power and versatility of the approach is demonstrated by developing models for classical mechanics and to predict the melting temperature of materials from fundamental properties. In the first example, the resulting PNNs are easily interpretable as Newton's second law, expressed as a non-trivial time integrator that exhibits time-reversibility and conserves energy, where the parsimony is critical to extract underlying symmetries from the data. In the second case, the PNNs not only find the celebrated Lindemann melting law, but also new relationships that outperform it in the pareto sense of parsimony vs. accuracy.

| S-EPMC8211802 | biostudies-literature

A Cognitive Neural Architecture Able to Learn and Communicate through Natural Language.

Project description:Communicative interactions involve a kind of procedural knowledge that is used by the human brain for processing verbal and nonverbal inputs and for language production. Although considerable work has been done on modeling human language abilities, it has been difficult to bring them together to a comprehensive tabula rasa system compatible with current knowledge of how verbal information is processed in the brain. This work presents a cognitive system, entirely based on a large-scale neural architecture, which was developed to shed light on the procedural knowledge involved in language elaboration. The main component of this system is the central executive, which is a supervising system that coordinates the other components of the working memory. In our model, the central executive is a neural network that takes as input the neural activation states of the short-term memory and yields as output mental actions, which control the flow of information among the working memory components through neural gating mechanisms. The proposed system is capable of learning to communicate through natural language starting from tabula rasa, without any a priori knowledge of the structure of phrases, meaning of words, role of the different classes of words, only by interacting with a human through a text-based interface, using an open-ended incremental learning process. It is able to learn nouns, verbs, adjectives, pronouns and other word classes, and to use them in expressive language. The model was validated on a corpus of 1587 input sentences, based on literature on early language assessment, at the level of about 4-years old child, and produced 521 output sentences, expressing a broad range of language processing functionalities.

| S-EPMC4641699 | biostudies-literature

Statistical inferences for polarity identification in natural language.

Project description:Information forms the basis for all human behavior, including the ubiquitous decision-making that people constantly perform in their every day lives. It is thus the mission of researchers to understand how humans process information to reach decisions. In order to facilitate this task, this work proposes LASSO regularization as a statistical tool to extract decisive words from textual content in order to study the reception of granular expressions in natural language. This differs from the usual use of the LASSO as a predictive model and, instead, yields highly interpretable statistical inferences between the occurrences of words and an outcome variable. Accordingly, the method suggests direct implications for the social sciences: it serves as a statistical procedure for generating domain-specific dictionaries as opposed to frequently employed heuristics. In addition, researchers can now identify text segments and word choices that are statistically decisive to authors or readers and, based on this knowledge, test hypotheses from behavioral research.

| S-EPMC6303018 | biostudies-literature

Individual differences in artificial and natural language statistical learning.

Project description:Statistical learning (SL) is considered a cornerstone of cognition. While decades of research have unveiled the remarkable breadth of structures that participants can learn from statistical patterns in experimental contexts, how this ability interfaces with real-world cognitive phenomena remains inconclusive. These mixed results may arise from the fact that SL is often treated as a general ability that operates uniformly across all domains, typically assuming that sensitivity to one kind of regularity implies equal sensitivity to others. In a preregistered study, we sought to clarify the link between SL and language by aligning the type of structure being processed in each task. We focused on the learning of trigram patterns using artificial and natural language statistics, to evaluate whether SL predicts sensitivity to comparable structures in natural speech. Adults were trained and tested on an artificial language incorporating statistically-defined syllable trigrams. We then evaluated their sensitivity to similar statistical structures in natural language using a multiword chunking task, which examines serial recall of high-frequency word trigrams-one of the building blocks of language. Participants' aptitude in learning artificial syllable trigrams positively correlated with their sensitivity to high-frequency word trigrams in natural language, suggesting that similar computations span learning across both tasks. Short-term SL taps into key aspects of long-term language acquisition when the statistical structures-and the computations used to process them-are comparable. Better aligning the specific statistical patterning across tasks may therefore provide an important steppingstone toward elucidating the relationship between SL and cognition at large.

| S-EPMC10311571 | biostudies-literature

Statistical Evolutionary Laws in Music Styles.

Project description:If a cultural feature is transmitted over generations and exposed to stochastic selection when spreading in a population, its evolution may be governed by statistical laws and be partly predictable, as in the case of genetic evolution. Music exhibits steady changes of styles over time, with new characteristics developing from traditions. Recent studies have found trends in the evolution of music styles, but little is known about their relations to the evolution theory. Here we analyze Western classical music data and find statistical evolutionary laws. For example, distributions of the frequencies of some rare musical events (e.g. dissonant intervals) exhibit steady increase in the mean and standard deviation as well as constancy of their ratio. We then study an evolutionary model where creators learn their data-generation models from past data and generate new data that will be socially selected by evaluators according to the content dissimilarity (novelty) and style conformity (typicality) with respect to the past data. The model reproduces the observed statistical laws and can make non-trivial predictions for the evolution of independent musical features. In addition, the same model with different parameterization can predict the evolution of Japanese enka music, which is developed in a different society and has a qualitatively different tendency of evolution. Our results suggest that the evolution of musical styles can partly be explained and predicted by the evolutionary model incorporating statistical learning, which can be important for other cultures and future music technologies.

| S-EPMC6831699 | biostudies-literature

What do patients learn about psychotropic medications on the web? A natural language processing study.

Project description:BackgroundLow rates of medication adherence remain a major challenge across psychiatry. In part, this likely reflects patient concerns about safety and adverse effects, accurate or otherwise. We therefore sought to characterize online information about common psychiatric medications in terms of positive and negative sentiment.MethodsWe applied a natural language processing tool to score the sentiment expressed in web search results for 51 psychotropic medications across 3 drug classes (antidepressants, antipsychotics, and mood stabilizers), as a means of seeing if articles referencing these medications were generally positive or generally negative in tone. We compared between medications of the same class, and across medication classes.ResultsAcross 12,733 web search results, significant within-class differences in positive (antidepressants: F(24,2682)?=?2.97, p < 0.001; antipsychotics: F(16,4029) = 3.25, p < 0.001; mood stabilizers: F(8,2371)?=?6.88, p < 0.001) and negative sentiment (antidepressants: F(24,6282)?=?11.17, p < 0.001; antipsychotics: F(16, 4029) ?=?12.13, p < 0.001; mood stabilizers: F(8, 2371)?=?13.28, p < 0.001) were identified. Among these were significantly greater negative sentiment for the antidepressants sertraline, duloxetine, venlafaxine, and paroxetine, and for the antipsychotics, quetiapine and risperidone. Conversely, lithium preparations and valproate exhibited less negative sentiment than other mood stabilizing medications.LimitationsWhile these results provide a novel means of comparing medications, the present analyses cannot be linked to individual patient consumption of this information, or to its influence on their future clinical interactions.ConclusionsOverall, a subset of psychotropic medications were associated with significantly more negative sentiment. Characterizing these differences may allow clinicians to anticipate patient willingness to initiate or continue medications.

| S-EPMC6921244 | biostudies-literature

Language models can learn complex molecular distributions.

Project description:Deep generative models of molecules have grown immensely in popularity, trained on relevant datasets, these models are used to search through chemical space. The downstream utility of generative models for the inverse design of novel functional compounds, depends on their ability to learn a training distribution of molecules. The most simple example is a language model that takes the form of a recurrent neural network and generates molecules using a string representation. Since their initial use, subsequent work has shown that language models are very capable, in particular, recent research has demonstrated their utility in the low data regime. In this work, we investigate the capacity of simple language models to learn more complex distributions of molecules. For this purpose, we introduce several challenging generative modeling tasks by compiling larger, more complex distributions of molecules and we evaluate the ability of language models on each task. The results demonstrate that language models are powerful generative models, capable of adeptly learning complex molecular distributions. Language models can accurately generate: distributions of the highest scoring penalized LogP molecules in ZINC15, multi-modal molecular distributions as well as the largest molecules in PubChem. The results highlight the limitations of some of the most popular and recent graph generative models- many of which cannot scale to these molecular distributions.

| S-EPMC9174447 | biostudies-literature

Visual statistical learning is related to natural language ability in adults: An ERP study.

Project description:Statistical learning (SL) is believed to enable language acquisition by allowing individuals to learn regularities within linguistic input. However, neural evidence supporting a direct relationship between SL and language ability is scarce. We investigated whether there are associations between event-related potential (ERP) correlates of SL and language abilities while controlling for the general level of selective attention. Seventeen adults completed tests of visual SL, receptive vocabulary, grammatical ability, and sentence completion. Response times and ERPs showed that SL is related to receptive vocabulary and grammatical ability. ERPs indicated that the relationship between SL and grammatical ability was independent of attention while the association between SL and receptive vocabulary depended on attention. The implications of these dissociative relationships in terms of underlying mechanisms of SL and language are discussed. These results further elucidate the cognitive nature of the links between SL mechanisms and language abilities.

| S-EPMC5293669 | biostudies-literature

A Standardized Project Gutenberg Corpus for Statistical Analysis of Natural Language and Quantitative Linguistics.

Project description:The use of Project Gutenberg (PG) as a text corpus has been extremely popular in statistical analysis of language for more than 25 years. However, in contrast to other major linguistic datasets of similar importance, no consensual full version of PG exists to date. In fact, most PG studies so far either consider only a small number of manually selected books, leading to potential biased subsets, or employ vastly different pre-processing strategies (often specified in insufficient details), raising concerns regarding the reproducibility of published results. In order to address these shortcomings, here we present the Standardized Project Gutenberg Corpus (SPGC), an open science approach to a curated version of the complete PG data containing more than 50,000 books and more than 3 × 10 9 word-tokens. Using different sources of annotated metadata, we not only provide a broad characterization of the content of PG, but also show different examples highlighting the potential of SPGC for investigating language variability across time, subjects, and authors. We publish our methodology in detail, the code to download and process the data, as well as the obtained corpus itself on three different levels of granularity (raw text, timeseries of word tokens, and counts of words). In this way, we provide a reproducible, pre-processed, full-size version of Project Gutenberg as a new scientific resource for corpus linguistics, natural language processing, and information retrieval.

| S-EPMC7516435 | biostudies-literature