Nonlinear Information Bottleneck
Ontology highlight
ABSTRACT:
SUBMITTER: Kolchinsky A
PROVIDER: S-EPMC7514526 | biostudies-literature | 2019 Nov
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Extracting relevant information from data is crucial for all forms of learning. The information bottleneck (IB) method formalizes this, offering a mathematically precise and conceptually appealing framework for understanding learning phenomena. However the nonlinearity of the IB problem makes it computationally expensive and analytically intractable in general. Here we derive a perturbation theory for the IB method and report the first complete characterization of the learning onset-the limit of maximum relevant information per bit extracted from data. We test our results on synthetic probability distributions, finding good agreement with the exact numerical solution near the onset of learning. We explore the difference and subtleties in our derivation and previous attempts at deriving a perturbation theory for the learning onset and attribute the discrepancy to a flawed assumption. Our work also provides a fresh perspective on the intimate relationship between the IB method and the strong data processing inequality.
| S-EPMC9806839 | biostudies-literature
Project description:Decision-making emerges from distributed computations across multiple brain areas, but it is unclear why the brain distributes the computation. In deep learning, artificial neural networks use multiple areas (or layers) to form optimal representations of task inputs. These optimal representations are sufficient to perform the task well, but minimal so they are invariant to other irrelevant variables. We recorded single neurons and multiunits in dorsolateral prefrontal cortex (DLPFC) and dorsal premotor cortex (PMd) in monkeys during a perceptual decision-making task. We found that while DLPFC represents task-related inputs required to compute the choice, the downstream PMd contains a minimal sufficient, or optimal, representation of the choice. To identify a mechanism for how cortex may form these optimal representations, we trained a multi-area recurrent neural network (RNN) to perform the task. Remarkably, DLPFC and PMd resembling representations emerged in the early and late areas of the multi-area RNN, respectively. The DLPFC-resembling area partially orthogonalized choice information and task inputs and this choice information was preferentially propagated to downstream areas through selective alignment with inter-area connections, while remaining task information was not. Our results suggest that cortex uses multi-area computation to form minimal sufficient representations by preferential propagation of relevant information between areas.
| S-EPMC10369960 | biostudies-literature
Project description:Responding to stimuli requires that organisms encode information about the external world. Not all parts of the input are important for behavior, and resource limitations demand that signals be compressed. Prediction of the future input is widely beneficial in many biological systems. We compute the trade-offs between representing the past faithfully and predicting the future using the information bottleneck approach, for input dynamics with different levels of complexity. For motion prediction, we show that, depending on the parameters in the input dynamics, velocity or position information is more useful for accurate prediction. We show which motion representations are easiest to re-use for accurate prediction in other motion contexts, and identify and quantify those with the highest transferability. For non-Markovian dynamics, we explore the role of long-term memory in shaping the internal representation. Lastly, we show that prediction in evolutionary population dynamics is linked to clustering allele frequencies into non-overlapping memories.
| S-EPMC7971903 | biostudies-literature
Project description:MotivationWe present a multi-sequence generalization of Variational Information Bottleneck and call the resulting model Attentive Variational Information Bottleneck (AVIB). Our AVIB model leverages multi-head self-attention to implicitly approximate a posterior distribution over latent encodings conditioned on multiple input sequences. We apply AVIB to a fundamental immuno-oncology problem: predicting the interactions between T-cell receptors (TCRs) and peptides.ResultsExperimental results on various datasets show that AVIB significantly outperforms state-of-the-art methods for TCR-peptide interaction prediction. Additionally, we show that the latent posterior distribution learned by AVIB is particularly effective for the unsupervised detection of out-of-distribution amino acid sequences.Availability and implementationThe code and the data used for this study are publicly available at: https://github.com/nec-research/vibtcr.Supplementary informationSupplementary data are available at Bioinformatics online.
| S-EPMC9825246 | biostudies-literature
Project description:Background:In the classical psychological refractory period (PRP) paradigm, two stimuli are presented in brief succession, and participants are asked to make separate speeded responses to both stimuli. Due to a central cognitive bottleneck, responses to the second stimulus are delayed, especially at short stimulus-onset asynchrony (SOA) between the two stimuli. Although the mechanisms of dual-task interference in the classical PRP paradigm have been extensively investigated, specific mechanisms underlying the cross-modal PRP paradigm are not well understood. In particular, it remains unknown whether the dominance of vision over audition manifests in the cross-modal PRP tasks. The present study aimed to investigate whether the visual dominance effect manifests in the cross-modal PRP paradigm. Methods:We adapted the classical PRP paradigm by manipulating the order of a visual and an auditory task: the visual task could either precede the auditory task or vice versa, at either short or long SOAs. Twenty-five healthy participants took part in Experiment 1, and thirty-three new participants took part in Experiment 2. Reaction time and accuracy data were calculated and further analyzed by repeated-measures analysis of variance. Results:The results showed that visual precedence in the Visual-Auditory condition caused larger impairments to the subsequent auditory processing than vice versa in the Auditory-Visual condition: a larger delay of second response was revealed in the Visual-Auditory condition (135 ± 10 ms) than the Auditory-Visual condition (88 ± 9 ms). This effect was found only at the short SOAs under the existence of the central bottleneck, but not at the long SOAs. Moreover, this effect occurred both when the single visual and the single auditory task were of equal difficulty in Experiment 1 and when the single auditory task was more difficult than the single visual task in Experiment 2. Conclusion:Results of the two experiments suggested that the visual dominance effect occurred under the central bottleneck of cognitive processing.
| S-EPMC6085859 | biostudies-other
Project description:In magnetic resonance (MR) imaging, a lack of standardization in acquisition often causes pulse sequence-based contrast variations in MR images from site to site, which impedes consistent measurements in automatic analyses. In this paper, we propose an unsupervised MR image harmonization approach, CALAMITI (Contrast Anatomy Learning and Analysis for MR Intensity Translation and Integration), which aims to alleviate contrast variations in multi-site MR imaging. Designed using information bottleneck theory, CALAMITI learns a globally disentangled latent space containing both anatomical and contrast information, which permits harmonization. In contrast to supervised harmonization methods, our approach does not need a sample population to be imaged across sites. Unlike traditional unsupervised harmonization approaches which often suffer from geometry shifts, CALAMITI better preserves anatomy by design. The proposed method is also able to adapt to a new testing site with a straightforward fine-tuning process. Experiments on MR images acquired from ten sites show that CALAMITI achieves superior performance compared with other harmonization approaches.
| S-EPMC10473284 | biostudies-literature
Project description:MotivationThe Gene Ontology (GO) is a controlled vocabulary designed to represent the biological concepts pertaining to gene products. This study investigates the methods for identifying informative subsets of GO terms in an automatic and objective fashion. This task in turn requires addressing the following issues: how to represent the semantic context of GO terms, what metrics are suitable for measuring the semantic differences between terms, how to identify an informative subset that retains as much as possible of the original semantic information of GO.ResultsWe represented the semantic context of a GO term using the word-usage-profile associated with the term, which enables one to measure the semantic differences between terms based on the differences in their semantic contexts. We further employed the information bottleneck methods to automatically identify subsets of GO terms that retain as much as possible of the semantic information in an annotation database. The automatically retrieved informative subsets align well with an expert-picked GO slim subset, cover important concepts and proteins, and enhance literature-based GO annotation.Availabilityhttp://carcweb.musc.edu/TextminingProjects/.
| S-EPMC2944202 | biostudies-literature
Project description:When humans attempt to perform two tasks at once, execution of the first task usually leads to postponement of the second one. This task delay is thought to result from a bottleneck occurring at a central, amodal stage of information processing that precludes two response selection or decision-making operations from being concurrently executed. Using time-resolved functional magnetic resonance imaging (fMRI), here we present a neural basis for such dual-task limitations, e.g. the inability of the posterior lateral prefrontal cortex, and possibly the superior medial frontal cortex, to process two decision-making operations at once. These results suggest that a neural network of frontal lobe areas acts as a central bottleneck of information processing that severely limits our ability to multitask.
| S-EPMC2527865 | biostudies-literature
Project description:Nonlinear parametric processes involving ultrashort pulses are typically carried out in time domain, which mathematically corresponds to a convolution of their frequency spectra. In contrast, this spectral convolution changes into a multiplication operation when performing the nonlinear interaction in frequency domain. Here, we extend the scope of frequency-domain nonlinear optics by demonstrating its ability to perform a temporal convolution. Through this approach, nonlinear optical operations that are inaccessible in time domain can be realised: specific optical information can be coherently advanced by picoseconds within a pulse sequence-a newly generated second harmonic pulse carries the amplitude and phase information of two input pulses. This central pulse is isolated when using an input field consisting of two cross-polarized input pulses in combination with type-II second harmonic generation. The effects of nonlinear temporal convolution can be viewed from the aspect of signal processing and pulse shaping, where the nonlinear interaction in the parametric crystal plays the role of a dynamic linear optical filter-in contrast to conventional static filters-with a shaping mask instantaneously adapting to the laser field.
| S-EPMC7486391 | biostudies-literature
Project description:Wireless transfer of information is the basis of modern communication. It includes cellular, WiFi, Bluetooth, and GPS systems, all of which use electromagnetic radio waves with frequencies ranging from typically 100 MHz to a few GHz. However, several long-standing challenges with standard radio-wave wireless transmission still exist, including keeping secure transmission of data from potential compromise. Here, we demonstrate wireless information transfer using a line-of-sight optical architecture with a micromechanical element. In this fundamentally new approach, a laser beam encoded with information impinges on a nonlinear micromechanical resonator located a distance from the laser. The force generated by the radiation pressure of the laser light on the nonlinear micromechanical resonator produces a sideband modulation signal, which carries the precise information encoded in the subtle changes in the radiation pressure. Using this, we demonstrate data and image transfer with one hundred percent fidelity with a single 96-by-270 μm silicon resonator element in an optical frequency band. This mechanical approach relies only on the momentum of the incident photons and is therefore able to use any portion of the optical frequency band-a band that is 10 000 times wider than the radio frequency band. Our line-of-sight architecture using highly scalable micromechanical resonators offers new possibilities in wireless communication. Due to their small size, these resonators can be easily arrayed while maintaining a small form factor to provide redundancy and parallelism.
| S-EPMC6444992 | biostudies-literature