Nystromformer: A Nystom-based Algorithm for Approximating Self-Attention
Ontology highlight
ABSTRACT:
SUBMITTER: Xiong Y
PROVIDER: S-EPMC8570649 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Differential equations are commonly used to model various types of real life applications. The complexity of these models may often hinder the ability to acquire an analytical solution. To overcome this drawback, numerical methods were introduced to approximate the solutions. Initially when developing a numerical algorithm, researchers focused on the key aspect which is accuracy of the method. As numerical methods becomes more and more robust, accuracy alone is not sufficient hence begins the pursuit of efficiency which warrants the need for reducing computational cost. The current research proposes a numerical algorithm for solving initial value higher order ordinary differential equations (ODEs). The proposed algorithm is derived as a three point block multistep method, developed in an Adams type formulae (3PBCS) and will be used to solve various types of ODEs and systems of ODEs. Type of ODEs that are selected varies from linear to nonlinear, artificial and real life problems. Results will illustrate the accuracy and efficiency of the proposed three point block method. Order, stability and convergence of the method are also presented in the study.
| S-EPMC7880453 | biostudies-literature
Project description:A current challenge in RNA structure prediction is the description of global helical arrangements compatible with a given secondary structure. Here we address this problem by developing a hierarchical graph sampling/data mining approach to reduce conformational space and accelerate global sampling of candidate topologies. Starting from a 2D structure, we construct an initial graph from size measures deduced from solved RNAs and junction topologies predicted by our data-mining algorithm RNAJAG trained on known RNAs. We sample these graphs in 3D space guided by knowledge-based statistical potentials derived from bending and torsion measures of internal loops as well as radii of gyration for known RNAs. Graph sampling results for 30 representative RNAs are analyzed and compared with reference graphs from both solved structures and predicted structures by available programs. This comparison indicates promise for our graph-based sampling approach for characterizing global helical arrangements in large RNAs: graph rmsds range from 2.52 to 28.24 Å for RNAs of size 25-158 nucleotides, and more than half of our graph predictions improve upon other programs. The efficiency in graph sampling, however, implies an additional step of translating candidate graphs into atomic models. Such models can be built with the same idea of graph partitioning and build-up procedures we used for RNA design.
| S-EPMC3964109 | biostudies-literature
Project description:Motor imagery (MI) electroencephalography (EEG) classification is an important part of the brain-computer interface (BCI), allowing people with mobility problems to communicate with the outside world via assistive devices. However, EEG decoding is a challenging task because of its complexity, dynamic nature, and low signal-to-noise ratio. Designing an end-to-end framework that fully extracts the high-level features of EEG signals remains a challenge. In this study, we present a parallel spatial-temporal self-attention-based convolutional neural network for four-class MI EEG signal classification. This study is the first to define a new spatial-temporal representation of raw EEG signals that uses the self-attention mechanism to extract distinguishable spatial-temporal features. Specifically, we use the spatial self-attention module to capture the spatial dependencies between the channels of MI EEG signals. This module updates each channel by aggregating features over all channels with a weighted summation, thus improving the classification accuracy and eliminating the artifacts caused by manual channel selection. Furthermore, the temporal self-attention module encodes the global temporal information into features for each sampling time step, so that the high-level temporal features of the MI EEG signals can be extracted in the time domain. Quantitative analysis shows that our method outperforms state-of-the-art methods for intra-subject and inter-subject classification, demonstrating its robustness and effectiveness. In terms of qualitative analysis, we perform a visual inspection of the new spatial-temporal representation estimated from the learned architecture. Finally, the proposed method is employed to realize control of drones based on EEG signal, verifying its feasibility in real-time applications.
| S-EPMC7759669 | biostudies-literature
Project description:The diffusion map is a dimensionality reduction method. The reduction coordinates are associated with the leading eigenfunctions of the backward Fokker-Planck operator, providing a dynamic meaning for these coordinates. One of the key factors that affect the accuracy of diffusion map embedding is the dynamic measure implemented in the Gaussian kernel. A common practice in diffusion map study of molecular systems is to approximate dynamic proximity with RMSD (root-mean-square deviation). In this paper, we present a hybrid geometry-energy based kernel. Since high energy-barriers may exist between geometrically similar conformations, taking both RMSD and energy difference into account in the kernel can better describe conformational transitions between neighboring conformations and lead to accurate embedding. We applied our diffusion map method to the ?-hairpin of the B1 domain of streptococcal protein G and to Trp-cage. Our results in ?-hairpin show that the diffusion map embedding achieves better results with the hybrid kernel than that with the RMSD-based kernel in terms of free energy landscape characterization and a new correlation measure between the cluster center Euclidean distances in the reduced-dimension space and the reciprocals of the total net flow between these clusters. In addition, our diffusion map analysis of the ultralong molecular dynamics trajectory of Trp-cage has provided a unified view of its folding mechanism. These promising results demonstrate the effectiveness of our diffusion map approach in the analysis of the dynamics and thermodynamics of molecular systems. The hybrid geometry-energy criterion could be also useful as a general dynamic measure for other purposes.
| S-EPMC6733700 | biostudies-literature
Project description:The dual functional lncRNAs have been intensively studied and identified to be involved in various fundamental cellular processes recently. It is essential to understand in which context when a dual functional lncRNA serves as a non-coding RNA or a template for coding peptide, particularly in some pathological conditions. However, apart from time consuming and cell type specific experiments, there is virtually no in-silico method for predicting the identity of dual functional lncRNAs. Here, we developed a deep-learning model with multi-head self-attention mechanism, LncReader, to identify dual functional lncRNAs based on their sequence, physicochemical and secondary structural features. Our data demonstrated that LncReader showed multiple advantage compared to various classical machine learning methods. Moreover, to obtain independent in-house datasets for robust testing, mass spectrometry proteomics combined with RNA-seq were applied in four leukemia cell lines. Remarkably, LncReader achieved the best performance among all these datasets. Therefore, LncReader provides a sophisticated and practical tool that enables fast dual functional lncRNAs identification.
2023-04-26 | PXD030066 | Pride
Project description:In this paper, a new method is proposed for low-rank matrix completion which is based on the least squares approximating to the known elements in the manifold formed by the singular vectors of the partial singular value decomposition alternatively. The method can achieve a reduction of the rank of the manifold by gradually reducing the number of the singular value of the thresholding and get the optimal low-rank matrix. It is proven that the manifold-alternative approximating method is convergent under some conditions. Furthermore, compared with the augmented Lagrange multiplier and the orthogonal rank-one matrix pursuit algorithms by random experiments, it is more effective as regards the CPU time and the low-rank property.
| S-EPMC6290672 | biostudies-other
Project description:Base editors are chimeric ribonucleoprotein complexes consisting of a DNA-targeting CRISPR-Cas module and a single-stranded DNA deaminase. They enable transition of C•G into T•A base pairs and vice versa on genomic DNA. While base editors have great potential as genome editing tools for basic research and gene therapy, their application has been hampered by a broad variation in editing efficiencies on different genomic loci. Here we perform an extensive analysis of adenine- and cytosine base editors on a library of 28,294 lentivirally integrated genetic sequences and establish BE-DICT, an attention-based deep learning algorithm capable of predicting base editing outcomes with high accuracy. BE-DICT is a versatile tool that in principle can be trained on any novel base editor variant, facilitating the application of base editing for research and therapy.
| S-EPMC8387386 | biostudies-literature
Project description:The clinical presentation of anorexia nervosa (AN) is characterized by preoccupation with body experience, intrusive concerns regarding shape, and pathological fears of weight gain. These symptoms are suggestive of unrelenting self-focused attention. No research to date has characterized self-focused attention (SFA) in AN nor examined neurocognitive features that may facilitate an excessive, rigid, or sustained focus on one's appearance.This study examined SFA, body image disturbance, and executive functioning in women with current anorexia nervosa (AN-C; n?=?24), a history of AN who were weight-restored at the time of the study (WR; n?=?19), and healthy controls (n?=?24).Private and public SFA were highest among WR and lowest among AN-C. Shape concerns were negatively correlated with SFA, especially among AN-C, after controlling for depression and social anxiety symptoms.Lower levels of SFA among AN-C were unexpected and suggest the acute state of AN may lessen pathological self-focus, negatively reinforcing symptoms. In addition, body image concerns may distract from general SFA. Deficits in executive attention may explain these findings, as each one unit increase in perseverative errors among AN-C participants was associated with an almost one-half unit decrease in public SFA.
| S-EPMC4474370 | biostudies-literature
Project description:Previous research has demonstrated abstract concepts associated with spatial location (e.g., God in the Heavens) could direct visual attention upward or downward, because thinking about the abstract concepts activates the corresponding vertical perceptual symbols. For self-concept, there are similar metaphors (e.g., "I am above others"). However, whether thinking about the self can induce visual attention orientation is still unknown. Therefore, the current study tested whether self-reflection can direct visual attention. Individuals often display the tendency of self-enhancement in social comparison, which reminds the individual of the higher position one possesses relative to others within the social environment. As the individual is the agent of the attention orientation, and high status tends to make an individual look down upon others to obtain a sense of pride, it was hypothesized that thinking about the self would lead to a downward attention orientation. Using reflection of personality traits and a target discrimination task, Study 1 found that, after self-reflection, visual attention was directed downward. Similar effects were also found after friend-reflection, with the level of downward attention being correlated with the likability rating scores of the friend. Thus, in Study 2, a disliked other was used as a control and the positive self-view was measured with above-average judgment task. We found downward attention orientation after self-reflection, but not after reflection upon the disliked other. Moreover, the attentional bias after self-reflection was correlated with above-average self-view. The current findings provide the first evidence that thinking about the self could direct visual-spatial attention downward, and suggest that this effect is probably derived from a positive self-view within the social context.
| S-EPMC5591849 | biostudies-literature
Project description:This SuperSeries is composed of the SubSeries listed below.
2022-04-26 | GSE201485 | GEO