Committee machines-a universal method to deal with non-idealities in memristor-based neural networks.
Ontology highlight
ABSTRACT:
SUBMITTER: Joksas D
PROVIDER: S-EPMC7450095 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Among the recent innovative technologies, memristor (memory-resistor) has attracted researchers attention as a fundamental computation element. It has been experimentally shown that memristive elements can emulate synaptic dynamics and are even capable of supporting spike timing dependent plasticity (STDP), an important adaptation rule that is gaining particular interest because of its simplicity and biological plausibility. The overall goal of this work is to provide a novel (theoretical) analog computing platform based on memristor devices and recurrent neural networks that exploits the memristor device physics to implement two variations of the backpropagation algorithm: recurrent backpropagation and equilibrium propagation. In the first learning technique, the use of memristor-based synaptic weights permits to propagate the error signals in the network by means of the nonlinear dynamics via an analog side network. This makes the processing non-digital and different from the current procedures. However, the necessity of a side analog network for the propagation of error derivatives makes this technique still highly biologically implausible. In order to solve this limitation, it is therefore proposed an alternative solution to the use of a side network by introducing a learning technique used for energy-based models: equilibrium propagation. Experimental results show that both approaches significantly outperform conventional architectures used for pattern reconstruction. Furthermore, due to the high suitability for VLSI implementation of the equilibrium propagation learning rule, additional results on the classification of the MNIST dataset are here reported.
| S-EPMC7105894 | biostudies-literature
Project description:Artificial Intelligence (AI) is currently experiencing a bloom driven by deep learning (DL) techniques, which rely on networks of connected simple computing units operating in parallel. The low communication bandwidth between memory and processing units in conventional von Neumann machines does not support the requirements of emerging applications that rely extensively on large sets of data. More recent computing paradigms, such as high parallelization and near-memory computing, help alleviate the data communication bottleneck to some extent, but paradigm- shifting concepts are required. Memristors, a novel beyond-complementary metal-oxide-semiconductor (CMOS) technology, are a promising choice for memory devices due to their unique intrinsic device-level properties, enabling both storing and computing with a small, massively-parallel footprint at low power. Theoretically, this directly translates to a major boost in energy efficiency and computational throughput, but various practical challenges remain. In this work we review the latest efforts for achieving hardware-based memristive artificial neural networks (ANNs), describing with detail the working principia of each block and the different design alternatives with their own advantages and disadvantages, as well as the tools required for accurate estimation of performance metrics. Ultimately, we aim to provide a comprehensive protocol of the materials and methods involved in memristive neural networks to those aiming to start working in this field and the experts looking for a holistic approach.
| S-EPMC10912231 | biostudies-literature
Project description:Neural networks trained by backpropagation have achieved tremendous successes on numerous intelligent tasks. However, naïve gradient-based training and updating methods on memristors impede applications due to intrinsic material properties. Here, we built a 39 nm 1 Gb phase change memory (PCM) memristor array and quantified the unique resistance drift effect. On this basis, spontaneous sparse learning (SSL) scheme that leverages the resistance drift to improve PCM-based memristor network training is developed. During training, SSL regards the drift effect as spontaneous consistency-based distillation process that reinforces the array weights at the high-resistance state continuously unless the gradient-based method switches them to low resistance. Experiments show that the SSL not only helps the convergence of network with better performance and sparsity controllability without additional computation in handwritten digit classification. This work promotes the learning algorithms with the intrinsic properties of memristor devices, opening a new direction for development of neuromorphic computing chips.
| S-EPMC7803975 | biostudies-literature
Project description:The ability to efficiently analyze the activities of biological neural networks can significantly promote our understanding of neural communications and functionalities. However, conventional neural signal analysis approaches need to transmit and store large amounts of raw recording data, followed by extensive processing offline, posing significant challenges to the hardware and preventing real-time analysis and feedback. Here, we demonstrate a memristor-based reservoir computing (RC) system that can potentially analyze neural signals in real-time. We show that the perovskite halide-based memristor can be directly driven by emulated neural spikes, where the memristor state reflects temporal features in the neural spike train. The RC system is successfully used to recognize neural firing patterns, monitor the transition of the firing patterns, and identify neural synchronization states among different neurons. Advanced neuroelectronic systems with such memristor networks can enable efficient neural signal analysis with high spatiotemporal precision, and possibly closed-loop feedback control.
| S-EPMC7228921 | biostudies-literature
Project description:Hardware neural networks with mechanical flexibility are promising next-generation computing systems for smart wearable electronics. Several studies have been conducted on flexible neural networks for practical applications; however, developing systems with complete synaptic plasticity for combinatorial optimization remains challenging. In this study, the metal-ion injection density is explored as a diffusive parameter of the conductive filament in organic memristors. Additionally, a flexible artificial synapse with bio-realistic synaptic plasticity is developed using organic memristors that have systematically engineered metal-ion injections, for the first time. In the proposed artificial synapse, short-term plasticity (STP), long-term plasticity, and homeostatic plasticity are independently achieved and are analogous to their biological counterparts. The time windows of the STP and homeostatic plasticity are controlled by the ion-injection density and electric-signal conditions, respectively. Moreover, stable capabilities for complex combinatorial optimization in the developed synapse arrays are demonstrated under spike-dependent operations. This effective concept for realizing flexible neuromorphic systems for complex combinatorial optimization is an essential building block for achieving a new paradigm of wearable smart electronics associated with artificial intelligent systems.
| S-EPMC10323658 | biostudies-literature
Project description:Conventional respiration measurement requires a separate device and/or can cause discomfort, so it is difficult to perform routinely, even for patients with respiratory diseases. The development of contactless respiration measurement technology would reduce discomfort and help detect and prevent fatal diseases. Therefore, we propose a respiration measurement method using a learning-based region-of-interest detector and a clustering-based respiration pixel estimation technique. The proposed method consists of a model for classifying whether a pixel conveys respiration information based on its variance and a method for classifying pixels with clear breathing components using the symmetry of the respiration signals. The proposed method was evaluated with the data of 14 men and women acquired in an actual environment, and it was confirmed that the average error was within approximately 0.1 bpm. In addition, a Bland-Altman analysis confirmed that the measurement result had no error bias, and regression analysis confirmed that the correlation of the results with the reference is high. The proposed method, designed to be inexpensive, fast, and robust to noise, is potentially suitable for practical use in clinical scenarios.
| S-EPMC8157066 | biostudies-literature
Project description:Memristors with tunable resistance states are emerging building blocks of artificial neural networks. However, in situ learning on a large-scale multiple-layer memristor network has yet to be demonstrated because of challenges in device property engineering and circuit integration. Here we monolithically integrate hafnium oxide-based memristors with a foundry-made transistor array into a multiple-layer neural network. We experimentally demonstrate in situ learning capability and achieve competitive classification accuracy on a standard machine learning dataset, which further confirms that the training algorithm allows the network to adapt to hardware imperfections. Our simulation using the experimental parameters suggests that a larger network would further increase the classification accuracy. The memristor neural network is a promising hardware platform for artificial intelligence with high speed-energy efficiency.
| S-EPMC6008303 | biostudies-literature
Project description:Oscillatory Neural Networks (ONNs) are currently arousing interest in the research community for their potential to implement very fast, ultra-low-power computing tasks by exploiting specific emerging technologies. From the architectural point of view, ONNs are based on the synchronization of oscillatory neurons in cognitive processing, as occurs in the human brain. As emerging technologies, VO2 and memristive devices show promising potential for the efficient implementation of ONNs. Abundant literature is now becoming available pertaining to the study and building of ONNs based on VO2 devices and resistive coupling, such as memristors. One drawback of direct resistive coupling is that physical resistances cannot be negative, but from the architectural and computational perspective this would be a powerful advantage when interconnecting weights in ONNs. Here we solve the problem by proposing a hardware implementation technique based on differential oscillatory neurons for ONNs (DONNs) with VO2-based oscillators and memristor-bridge circuits. Each differential oscillatory neuron is made of a pair of VO2 oscillators operating in anti-phase. This way, the neurons provide a pair of differential output signals in opposite phase. The memristor-bridge circuit is used as an adjustable coupling function that is compatible with differential structures and capable of providing both positive and negative weights. By combining differential oscillatory neurons and memristor-bridge circuits, we propose the hardware implementation of a fully connected differential ONN (DONN) and use it as an associative memory. The standard Hebbian rule is used for training, and the weights are then mapped to the memristor-bridge circuit through a proposed mapping rule. The paper also introduces some functional and hardware specifications to evaluate the design. Evaluation is performed by circuit-level electrical simulations and shows that the retrieval accuracy of the proposed design is comparable to that of classic Hopfield Neural Networks.
| S-EPMC8322448 | biostudies-literature
Project description:Spiking Neural Networks (SNNs) have shown favorable performance recently. Nonetheless, the time-consuming computation on neuron level and complex optimization limit their real-time application. Curiosity has shown great performance in brain learning, which helps biological brains grasp new knowledge efficiently and actively. Inspired by this leaning mechanism, we propose a curiosity-based SNN (CBSNN) model, which contains four main learning processes. Firstly, the network is trained with biologically plausible plasticity principles to get the novelty estimations of all samples in only one epoch; secondly, the CBSNN begins to repeatedly learn the samples whose novelty estimations exceed the novelty threshold and dynamically update the novelty estimations of samples according to the learning results in five epochs; thirdly, in order to avoid the overfitting of the novel samples and forgetting of the learned samples, CBSNN retrains all samples in one epoch; finally, step two and step three are periodically taken until network convergence. Compared with the state-of-the-art Voltage-driven Plasticity-centric SNN (VPSNN) under standard architecture, our model achieves a higher accuracy of 98.55% with only 54.95% of its computation cost on the MNIST hand-written digit recognition dataset. Similar conclusion can also be found out in other datasets, i.e., Iris, NETtalk, Fashion-MNIST, and CIFAR-10, respectively. More experiments and analysis further prove that such curiosity-based learning theory is helpful in improving the efficiency of SNNs. As far as we know, this is the first practical combination of the curiosity mechanism and SNN, and these improvements will make the realistic application of SNNs possible on more specific tasks within the von Neumann framework.
| S-EPMC7020337 | biostudies-literature
Project description:Memristors have emerged as a promising candidate for critical applications such as non-volatile memory as well as non-Von Neumann computing architectures based on neuromorphic and machine learning systems. In this study, we demonstrate that memristors can be used to perform principal component analysis (PCA), an important technique for machine learning and data feature learning. The conductance changes of memristors in response to voltage pulses are studied and modeled with an internal state variable to trace the analog behavior of the device. Unsupervised, online learning is achieved in a memristor crossbar using Sanger's learning rule, a derivative of Hebb's rule, to obtain the principal components. The details of weights evolution during training is investigated over learning epochs as a function of training parameters. The effects of device non-uniformity on the PCA network performance are further analyzed. We show that the memristor-based PCA network is capable of linearly separating distinct classes from sensory data with high clarification success of 97.6% even in the presence of large device variations.
| S-EPMC4650685 | biostudies-literature