Dataset Information

Gradient Decomposition Methods for Training Neural Networks With Non-ideal Synaptic Devices.

ABSTRACT: While promising for high-capacity machine learning accelerators, memristor devices have non-idealities that prevent software-equivalent accuracies when used for online training. This work uses a combination of Mini-Batch Gradient Descent (MBGD) to average gradients, stochastic rounding to avoid vanishing weight updates, and decomposition methods to keep the memory overhead low during mini-batch training. Since the weight update has to be transferred to the memristor matrices efficiently, we also investigate the impact of reconstructing the gradient matrixes both internally (rank-seq) and externally (rank-sum) to the memristor array. Our results show that streaming batch principal component analysis (streaming batch PCA) and non-negative matrix factorization (NMF) decomposition algorithms can achieve near MBGD accuracy in a memristor-based multi-layer perceptron trained on the MNIST (Modified National Institute of Standards and Technology) database with only 3 to 10 ranks at significant memory savings. Moreover, NMF rank-seq outperforms streaming batch PCA rank-seq at low-ranks making it more suitable for hardware implementation in future memristor-based accelerators.

SUBMITTER: Zhao J

PROVIDER: S-EPMC8645649 | biostudies-literature | 2021

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Gradient Decomposition Methods for Training Neural Networks With Non-ideal Synaptic Devices.

Zhao Junyun J Huang Siyuan S Yousuf Osama O Gao Yutong Y Hoskins Brian D BD Adam Gina C GC

Frontiers in neuroscience 20211122

While promising for high-capacity machine learning accelerators, memristor devices have non-idealities that prevent software-equivalent accuracies when used for online training. This work uses a combination of Mini-Batch Gradient Descent (MBGD) to average gradients, stochastic rounding to avoid vanishing weight updates, and decomposition methods to keep the memory overhead low during mini-batch training. Since the weight update has to be transferred to the memristor matrices efficiently, we also ...[more]

PMID: 34880721

Similar Datasets

Project description:Spiking neural networks (SNNs) are a computational tool in which the information is coded into spikes, as in some parts of the brain, differently from conventional neural networks (NNs) that compute over real-numbers. Therefore, SNNs can implement intelligent information extraction in real-time at the edge of data acquisition and correspond to a complementary solution to conventional NNs working for cloud-computing. Both NN classes face hardware constraints due to limited computing parallelism and separation of logic and memory. Emerging memory devices, like resistive switching memories, phase change memories, or memristive devices in general are strong candidates to remove these hurdles for NN applications. The well-established training procedures of conventional NNs helped in defining the desiderata for memristive device dynamics implementing synaptic units. The generally agreed requirements are a linear evolution of memristive conductance upon stimulation with train of identical pulses and a symmetric conductance change for conductance increase and decrease. Conversely, little work has been done to understand the main properties of memristive devices supporting efficient SNN operation. The reason lies in the lack of a background theory for their training. As a consequence, requirements for NNs have been taken as a reference to develop memristive devices for SNNs. In the present work, we show that, for efficient CMOS/memristive SNNs, the requirements for synaptic memristive dynamics are very different from the needs of a conventional NN. System-level simulations of a SNN trained to classify hand-written digit images through a spike timing dependent plasticity protocol are performed considering various linear and non-linear plausible synaptic memristive dynamics. We consider memristive dynamics bounded by artificial hard conductance values and limited by the natural dynamics evolution toward asymptotic values (soft-boundaries). We quantitatively analyze the impact of resolution and non-linearity properties of the synapses on the network training and classification performance. Finally, we demonstrate that the non-linear synapses with hard boundary values enable higher classification performance and realize the best trade-off between classification accuracy and required training time. With reference to the obtained results, we discuss how memristive devices with non-linear dynamics constitute a technologically convenient solution for the development of on-line SNN training.

Project description:IntroductionEffective leadership improves patient care during medical and trauma resuscitations. While dedicated training programs can improve leadership in trauma resuscitation, we have a limited understanding of the optimal training methods. Our objective was to explore learners' and teachers' perceptions of effective methods of leadership training for trauma resuscitation.MethodsWe performed a qualitative exploration of learner and teacher perceptions of leadership training methods using a modified grounded theory approach. We interviewed 28 participants, including attending physicians, residents, fellows, and nurses who regularly participated in trauma team activations. We then analyzed transcripts in an iterative manner to form codes, identify themes, and explore relationships between themes.ResultsBased on interviewees' perceptions, we identified seven methods used to train leadership in trauma resuscitation: reflection; feedback; hands-on learning; role modeling; simulation; group reflection; and didactic. We also identified three major themes in perceived best practices in training leaders in trauma resuscitation: formal vs informal curriculum; training techniques for novice vs more senior learner; and interprofessional training. Participants felt that informal training methods were the most important part of training, and that a significant part of a training program for leaders in trauma resuscitation should use informal methods. Learners who were earlier in their training preferred more supervision and guidance, while learners who were more advanced in their training preferred a greater degree of autonomy. Finally, participants believed leadership training for trauma resuscitation should be multidisciplinary and interprofessional.ConclusionWe identified several important themes for training leaders in trauma resuscitation, including using a variety of different training methods, adapting the methods used based on the learner's level of training, and incorporating opportunities for multidisciplinary and interprofessional training. More research is needed to determine the optimal balance of informal and formal training, how to standardize and increase consistency in informal training, and the optimal way to incorporate multidisciplinary and interprofessional learning into a leadership in trauma resuscitation training program.

Dataset Information

Gradient Decomposition Methods for Training Neural Networks With Non-ideal Synaptic Devices.

Publications

Gradient Decomposition Methods for Training Neural Networks With Non-ideal Synaptic Devices.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets