Real-Time Selective Markerless Tracking of Forepaws of Head Fixed Mice Using Deep Neural Networks.
Ontology highlight
ABSTRACT:
SUBMITTER: Forys BJ
PROVIDER: S-EPMC7307631 | biostudies-literature | 2020 May/Jun
REPOSITORIES: biostudies-literature
Publications
Real-Time Selective Markerless Tracking of Forepaws of Head Fixed Mice Using Deep Neural Networks.
Forys Brandon J BJ Xiao Dongsheng D Gupta Pankaj P Murphy Timothy H TH
eNeuro 20200501 3
Here, we describe a system capable of tracking specific mouse paw movements at high frame rates (70.17 Hz) with a high level of accuracy (mean<i> </i>=<i> </i>0.95, SD<i> </i><<i> </i>0.01). Short-latency markerless tracking of specific body parts opens up the possibility of manipulating motor feedback. We present a software and hardware scheme built on DeepLabCut-a robust movement-tracking deep neural network framework-which enables real-time estimation of paw and digit movements of mice. Using ...[more]
Similar Datasets
Project description:The ability to control a behavioral task or stimulate neural activity based on animal behavior in real-time is an important tool for experimental neuroscientists. Ideally, such tools are noninvasive, low-latency, and provide interfaces to trigger external hardware based on posture. Recent advances in pose estimation with deep learning allows researchers to train deep neural networks to accurately quantify a wide variety of animal behaviors. Here, we provide a new DeepLabCut-Live! package that achieves low-latency real-time pose estimation (within 15 ms, >100 FPS), with an additional forward-prediction module that achieves zero-latency feedback, and a dynamic-cropping mode that allows for higher inference speeds. We also provide three options for using this tool with ease: (1) a stand-alone GUI (called DLC-Live! GUI), and integration into (2) Bonsai, and (3) AutoPilot. Lastly, we benchmarked performance on a wide range of systems so that experimentalists can easily decide what hardware is required for their needs.
| S-EPMC7781595 | biostudies-literature
Project description:The functional movement screen (FMS) test is a seven-test battery used to assess fundamental movement abilities of individuals. It is commonly used to predict sports injuries but relies on clinical expertise and is not suitable for self-examination. This study presents an automatic FMS movement assessment framework using a multi-view deep neural network called MVDNN. The framework combines automatic skeleton extraction with manual feature selection to extract 3D trajectory features of human skeleton joints from two different directions. Three mainstream methods of time-series modeling are then used to learn high-level feature representation from skeleton sequences, and motion features from two views are fused to provide complementary information. Results of public FMS movements dataset demonstrate that our MVDNN outperforms current state-of-the-art methods with an average miF1 score of 0.857, maF1 score of 0.768, and Kappa score of 0.640 over ten runs.
| S-EPMC10784700 | biostudies-literature
Project description:Differences in popularity are a key aspect of status in virtually all human groups and shape social interactions within them. Little is known, however, about how we track and neurally represent others' popularity. We addressed this question in two real-world social networks using sociometric methods to quantify popularity. Each group member (perceiver) viewed faces of every other group member (target) while whole-brain functional MRI data were collected. Independent functional localizer tasks were used to identify brain systems supporting affective valuation (ventromedial prefrontal cortex, ventral striatum, amygdala) and social cognition (dorsomedial prefrontal cortex, precuneus, temporoparietal junction), respectively. During the face-viewing task, activity in both types of neural systems tracked targets' sociometric popularity, even when controlling for potential confounds. The target popularity-social cognition system relationship was mediated by valuation system activity, suggesting that observing popular individuals elicits value signals that facilitate understanding their mental states. The target popularity-valuation system relationship was strongest for popular perceivers, suggesting enhanced sensitivity to differences among other group members' popularity. Popular group members also demonstrated greater interpersonal sensitivity by more accurately predicting how their own personalities were perceived by other individuals in the social network. These data offer insights into the mechanisms by which status guides social behavior.
| S-EPMC4679039 | biostudies-other
Project description:PurposeSurgery is the predominant treatment modality of human glioma but suffers difficulty on clearly identifying tumor boundaries in clinic. Conventional practice involves neurosurgeon's visual evaluation and intraoperative histological examination of dissected tissues using frozen section, which is time-consuming and complex. The aim of this study was to develop fluorescent imaging coupled with artificial intelligence technique to quickly and accurately determine glioma in real-time during surgery.MethodsGlioma patients (N = 23) were enrolled and injected with indocyanine green for fluorescence image-guided surgery. Tissue samples (N = 1874) were harvested from surgery of these patients, and the second near-infrared window (NIR-II, 1000-1700 nm) fluorescence images were obtained. Deep convolutional neural networks (CNNs) combined with NIR-II fluorescence imaging (named as FL-CNN) were explored to automatically provide pathological diagnosis of glioma in situ in real-time during patient surgery. The pathological examination results were used as the gold standard.ResultsThe developed FL-CNN achieved the area under the curve (AUC) of 0.945. Comparing to neurosurgeons' judgment, with the same level of specificity >80%, FL-CNN achieved a much higher sensitivity (93.8% versus 82.0%, P < 0.001) with zero time overhead. Further experiments demonstrated that FL-CNN corrected >70% of the errors made by neurosurgeons. FL-CNN was also able to rapidly predict grade and Ki-67 level (AUC 0.810 and 0.625) of tumor specimens intraoperatively.ConclusionOur study demonstrates that deep CNNs are better at capturing important information from fluorescence images than surgeons' evaluation during patient surgery. FL-CNN is highly promising to provide pathological diagnosis intraoperatively and assist neurosurgeons to obtain maximum resection safely.Trial registrationChiCTR ChiCTR2000029402. Registered 29 January 2020, retrospectively registered.
| S-EPMC8440289 | biostudies-literature
Project description:Deep neural networks (DNNs) have substantial computational requirements, which greatly limit their performance in resource-constrained environments. Recently, there are increasing efforts on optical neural networks and optical computing based DNNs hardware, which bring significant advantages for deep learning systems in terms of their power efficiency, parallelism and computational speed. Among them, free-space diffractive deep neural networks (D2NNs) based on the light diffraction, feature millions of neurons in each layer interconnected with neurons in neighboring layers. However, due to the challenge of implementing reconfigurability, deploying different DNNs algorithms requires re-building and duplicating the physical diffractive systems, which significantly degrades the hardware efficiency in practical application scenarios. Thus, this work proposes a novel hardware-software co-design method that enables first-of-its-like real-time multi-task learning in D22NNs that automatically recognizes which task is being deployed in real-time. Our experimental results demonstrate significant improvements in versatility, hardware efficiency, and also demonstrate and quantify the robustness of proposed multi-task D2NN architecture under wide noise ranges of all system components. In addition, we propose a domain-specific regularization algorithm for training the proposed multi-task architecture, which can be used to flexibly adjust the desired performance for each task.
| S-EPMC8155121 | biostudies-literature
Project description:Here, we describe an automated optical method for tracking animal behavior in both head-fixed and freely moving animals, in real time and offline. It takes advantage of an off-the-shelf camera system, the Pixy camera, designed as a fast vision sensor for robotics that uses a color-based filtering algorithm at 50 Hz to track objects. Using customized software, we demonstrate the versatility of our approach by first tracking the rostro-caudal motion of individual adjacent row (D1, D2) or arc whiskers (β, γ), or a single whisker and points on the whisker pad, in head-fixed mice performing a tactile task. Next, we acquired high-speed video and Pixy data simultaneously and applied the pixy-based real-time tracking to high-speed video data. With this approach, we expand the temporal resolution of the Pixy camera and track motion (post hoc) at the limit of high-speed video frame rates. Finally, we show that this system is flexible: it can be used to track individual whisker or limb position without any sophisticated object tracking algorithm, it can be used in many lighting conditions including infrared (IR); it can be used to track head rotation and location of multiple animals simultaneously. Our system makes behavioral monitoring possible in virtually any biological setting.
| S-EPMC5318546 | biostudies-literature
Project description:PurposeAccurate verification of tumor position during irradiation could reduce the probability of target miss. We investigated whether a commercial gantry-mounted 2-dimensional (2D) kilo-voltage (kV) imaging system could be used for real-time 3D tumor tracking during volumetric modulated arc therapy (VMAT) lung stereotactic body radiation therapy (SBRT). Markerless tumor tracking on kV fluoroscopic images was validated using a life-like moving thorax phantom and subsequently performed on kV images continuously acquired before and during free-breathing VMAT lung SBRT.Methods and materialsThe 3D-printed/molded phantom containing 3 lung tumors was moved in 3D in TrueBeam developer mode, using simulated regular/irregular breathing patterns. Planar kV images were acquired at 7 frames/s during 11 Gy/fraction 10 MV flattening filter free VMAT. 2D reference templates were created for each gantry angle using the planning 4D computed tomography inspiration phase. kV images and templates were matched using normalized cross correlation to determine 2D tumor position, and triangulation of 2D matched projections determined the third dimension. 3D target tracking performed on cone beam computed tomography projection data from 18 patients (20 tumors) and real-time online tracking data from 2 of the 18 patients who underwent free-breathing VMAT lung SBRT are presented.ResultsFor target 1 and 2 of the phantom (upper lung and middle/medial lung, mean density -130 Hounsfield units), 3D results within 2 mm of the known position were present in 92% and 96% of the kV projections, respectively. For target 3 (inferior lung, mean density -478 Hounsfield units) this dropped to 80%. Benchmarking against the respiratory signal, 13/20 (65%) tumors (10.5 ± 11.1 cm3) were considered successfully tracked on the cone beam computed tomography data. Tracking was less successful (≤50% of the time) in 7/20 (1.2 ± 1.5 cm3). Successful online tracking during lung SBRT was demonstrated.Conclusions3D markerless tumor tracking on a standard linear accelerator using template matching and triangulation of free-breathing kV fluoroscopic images was possible in 65% of small lung tumors. The smallest tumors were most challenging.
| S-EPMC8170355 | biostudies-literature
Project description:Intraoperative diagnosis is essential for providing safe and effective care during cancer surgery1. The existing workflow for intraoperative diagnosis based on hematoxylin and eosin staining of processed tissue is time, resource and labor intensive2,3. Moreover, interpretation of intraoperative histologic images is dependent on a contracting, unevenly distributed, pathology workforce4. In the present study, we report a parallel workflow that combines stimulated Raman histology (SRH)5-7, a label-free optical imaging method and deep convolutional neural networks (CNNs) to predict diagnosis at the bedside in near real-time in an automated fashion. Specifically, our CNNs, trained on over 2.5 million SRH images, predict brain tumor diagnosis in the operating room in under 150 s, an order of magnitude faster than conventional techniques (for example, 20-30 min)2. In a multicenter, prospective clinical trial (n = 278), we demonstrated that CNN-based diagnosis of SRH images was noninferior to pathologist-based interpretation of conventional histologic images (overall accuracy, 94.6% versus 93.9%). Our CNNs learned a hierarchy of recognizable histologic feature representations to classify the major histopathologic classes of brain tumors. In addition, we implemented a semantic segmentation method to identify tumor-infiltrated diagnostic regions within SRH images. These results demonstrate how intraoperative cancer diagnosis can be streamlined, creating a complementary pathway for tissue diagnosis that is independent of a traditional pathology laboratory.
| S-EPMC6960329 | biostudies-literature
Project description:In general, animal behavior can be described as the neuronal-driven sequence of reoccurring postures through time. Most of the available current technologies focus on offline pose estimation with high spatiotemporal resolution. However, to correlate behavior with neuronal activity it is often necessary to detect and react online to behavioral expressions. Here we present DeepLabStream, a versatile closed-loop tool providing real-time pose estimation to deliver posture dependent stimulations. DeepLabStream has a temporal resolution in the millisecond range, can utilize different input, as well as output devices and can be tailored to multiple experimental designs. We employ DeepLabStream to semi-autonomously run a second-order olfactory conditioning task with freely moving mice and optogenetically label neuronal ensembles active during specific head directions.
| S-EPMC7846585 | biostudies-literature
Project description:The ability to efficiently analyze the activities of biological neural networks can significantly promote our understanding of neural communications and functionalities. However, conventional neural signal analysis approaches need to transmit and store large amounts of raw recording data, followed by extensive processing offline, posing significant challenges to the hardware and preventing real-time analysis and feedback. Here, we demonstrate a memristor-based reservoir computing (RC) system that can potentially analyze neural signals in real-time. We show that the perovskite halide-based memristor can be directly driven by emulated neural spikes, where the memristor state reflects temporal features in the neural spike train. The RC system is successfully used to recognize neural firing patterns, monitor the transition of the firing patterns, and identify neural synchronization states among different neurons. Advanced neuroelectronic systems with such memristor networks can enable efficient neural signal analysis with high spatiotemporal precision, and possibly closed-loop feedback control.
| S-EPMC7228921 | biostudies-literature