Deep-Learning-Based Recovery of Missing Optical Marker Trajectories in 3D Motion Capture Systems
Ontology highlight
ABSTRACT:
SUBMITTER: Yuhai O
PROVIDER: S-EPMC11200691 | biostudies-literature | 2024 Jun
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Missing information in motion capture data caused by occlusion or detachment of markers is a common problem that is difficult to avoid entirely. The aim of this study was to develop and test an algorithm for reconstruction of corrupted marker trajectories in datasets representing human gait. The reconstruction was facilitated using information of marker inter-correlations obtained from a principal component analysis, combined with a novel weighting procedure. The method was completely data-driven, and did not require any training data. We tested the algorithm on datasets with movement patterns that can be considered both well suited (healthy subject walking on a treadmill) and less suited (transitioning from walking to running and the gait of a subject with cerebral palsy) to reconstruct. Specifically, we created 50 copies of each dataset, and corrupted them with gaps in multiple markers at random temporal and spatial positions. Reconstruction errors, quantified by the average Euclidian distance between predicted and measured marker positions, was ≤ 3 mm for the well suited dataset, even when there were gaps in up to 70% of all time frames. For the less suited datasets, median reconstruction errors were in the range 5-6 mm. However, a few reconstructions had substantially larger errors (up to 29 mm). Our results suggest that the proposed algorithm is a viable alternative both to conventional gap-filling algorithms and state-of-the-art reconstruction algorithms developed for motion capture systems. The strengths of the proposed algorithm are that it can fill gaps anywhere in the dataset, and that the gaps can be considerably longer than when using conventional interpolation techniques. Limitations are that it does not enforce musculoskeletal constraints, and that the reconstruction accuracy declines if applied to datasets with less predictable movement patterns.
| S-EPMC4816448 | biostudies-literature
Project description:In this paper, a marker-based, single-person optical motion capture method (DeepMoCap) is proposed using multiple spatio-temporally aligned infrared-depth sensors and retro-reflective straps and patches (reflectors). DeepMoCap explores motion capture by automatically localizing and labeling reflectors on depth images and, subsequently, on 3D space. Introducing a non-parametric representation to encode the temporal correlation among pairs of colorized depthmaps and 3D optical flow frames, a multi-stage Fully Convolutional Network (FCN) architecture is proposed to jointly learn reflector locations and their temporal dependency among sequential frames. The extracted reflector 2D locations are spatially mapped in 3D space, resulting in robust 3D optical data extraction. The subject's motion is efficiently captured by applying a template-based fitting technique on the extracted optical data. Two datasets have been created and made publicly available for evaluation purposes; one comprising multi-view depth and 3D optical flow annotated images (DMC2.5D), and a second, consisting of spatio-temporally aligned multi-view depth images along with skeleton, inertial and ground truth MoCap data (DMC3D). The FCN model outperforms its competitors on the DMC2.5D dataset using 2D Percentage of Correct Keypoints (PCK) metric, while the motion capture outcome is evaluated against RGB-D and inertial data fusion approaches on DMC3D, outperforming the next best method by 4 . 5 % in total 3D PCK accuracy.
| S-EPMC6359336 | biostudies-literature
Project description:Marker-based Optical Motion Capture (OMC) systems and associated musculoskeletal (MSK) modelling predictions offer non-invasively obtainable insights into muscle and joint loading at an in vivo level, aiding clinical decision-making. However, an OMC system is lab-based, expensive, and requires a line of sight. Inertial Motion Capture (IMC) techniques are widely-used alternatives, which are portable, user-friendly, and relatively low-cost, although with lesser accuracy. Irrespective of the choice of motion capture technique, one typically uses an MSK model to obtain the kinematic and kinetic outputs, which is a computationally expensive tool increasingly well approximated by machine learning (ML) methods. Here, an ML approach is presented that maps experimentally recorded IMC input data to the human upper-extremity MSK model outputs computed from ('gold standard') OMC input data. Essentially, this proof-of-concept study aims to predict higher-quality MSK outputs from the much easier-to-obtain IMC data. We use OMC and IMC data simultaneously collected for the same subjects to train different ML architectures that predict OMC-driven MSK outputs from IMC measurements. In particular, we employed various neural network (NN) architectures, such as Feed-Forward Neural Networks (FFNNs) and Recurrent Neural Networks (RNNs) (vanilla, Long Short-Term Memory, and Gated Recurrent Unit) and a comprehensive search for the best-fit model in the hyperparameters space in both subject-exposed (SE) as well as subject-naive (SN) settings. We observed a comparable performance for both FFNN and RNN models, which have a high degree of agreement (ravg,SE,FFNN=0.90±0.19, ravg,SE,RNN=0.89±0.17, ravg,SN,FFNN=0.84±0.23, and ravg,SN,RNN=0.78±0.23) with the desired OMC-driven MSK estimates for held-out test data. The findings demonstrate that mapping IMC inputs to OMC-driven MSK outputs using ML models could be instrumental in transitioning MSK modelling from 'lab to field'.
| S-EPMC10215337 | biostudies-literature
Project description:Motion capture of unrestrained moving animals is a major analytic tool in neuroethology and behavioral physiology. At present, several motion capture methodologies have been developed, all of which have particular limitations regarding experimental application. Whereas marker-based motion capture systems are very robust and easily adjusted to suit different setups, tracked species, or body parts, they cannot be applied in experimental situations where markers obstruct the natural behavior (e.g., when tracking delicate, elastic, and/or sensitive body structures). On the other hand, marker-less motion capture systems typically require setup- and animal-specific adjustments, for example by means of tailored image processing, decision heuristics, and/or machine learning of specific sample data. Among the latter, deep-learning approaches have become very popular because of their applicability to virtually any sample of video data. Nevertheless, concise evaluation of their training requirements has rarely been done, particularly with regard to the transfer of trained networks from one application to another. To address this issue, the present study uses insect locomotion as a showcase example for systematic evaluation of variation and augmentation of the training data. For that, we use artificially generated video sequences with known combinations of observed, real animal postures and randomized body position, orientation, and size. Moreover, we evaluate the generalization ability of networks that have been pre-trained on synthetic videos to video recordings of real walking insects, and estimate the benefit in terms of reduced requirement for manual annotation. We show that tracking performance is affected only little by scaling factors ranging from 0.5 to 1.5. As expected from convolutional networks, the translation of the animal has no effect. On the other hand, we show that sufficient variation of rotation in the training data is essential for performance, and make concise suggestions about how much variation is required. Our results on transfer from synthetic to real videos show that pre-training reduces the amount of necessary manual annotation by about 50%.
| S-EPMC8100444 | biostudies-literature
Project description:Optical motion capture (OMC) systems are commonly used to capture in-vivo three-dimensional joint kinematics. However, the skin-based markers may not reflect the underlying bone movement, a source of error known as soft tissue artifact (STA). This study examined STA during wrist motion by evaluating the agreement between OMC and biplanar videoradiography (BVR). Nine subjects completed 7 different wrist motion tasks: doorknob rotation to capture supination and pronation, radial-ulnar deviation, flexion-extension, circumduction, hammering, and pitcher pouring. BVR and OMC captured the motion simultaneously. Wrist kinematics were quantified using helical motion parameters of rotation and translation, and Bland-Altman analysis quantified the mean difference (bias) and 95% limit of agreement (LOA). The rotational bias of doorknob pronation, a median bias of -4.9°, was significantly larger than the flexion-extension (0.7°, p < 0.05) and radial-ulnar deviation (1.8°, p < 0.01) tasks. The rotational LOA range was significantly smaller in the flexion-extension task (5.9°) compared to pitcher (11.6°, p < 0.05) and doorknob pronation (17.9°, p < 0.05) tasks. The translation bias did not differ between tasks. The translation LOA range was significantly larger in circumduction (9.8°) compared to the radial-ulnar deviation (6.3°, p < 0.05) and pitcher (3.4°, p < 0.05) tasks. While OMC technology has a wide-range of successful applications, we demonstrated it has relatively poor agreement with BVR in tracking wrist motion, and that the agreement depends on the nature and direction of wrist motion.
| S-EPMC8089049 | biostudies-literature
Project description:Obtaining quantitative data describing the movements of animals is an essential step in understanding their locomotor biology. Outside the laboratory, measuring animal locomotion often relies on video-based approaches and analysis is hampered because of difficulties in calibration and often the limited availability of possible camera positions. It is also usually restricted to two dimensions, which is often an undesirable over-simplification given the essentially three-dimensional nature of many locomotor performances. In this paper we demonstrate a fully three-dimensional approach based on 3D photogrammetric reconstruction using multiple, synchronised video cameras. This approach allows full calibration based on the separation of the individual cameras and will work fully automatically with completely unmarked and undisturbed animals. As such it has the potential to revolutionise work carried out on free-ranging animals in sanctuaries and zoological gardens where ad hoc approaches are essential and access within enclosures often severely restricted. The paper demonstrates the effectiveness of video-based 3D photogrammetry with examples from primates and birds, as well as discussing the current limitations of this technique and illustrating the accuracies that can be obtained. All the software required is open source so this can be a very cost effective approach and provides a methodology of obtaining data in situations where other approaches would be completely ineffective.
| S-EPMC4154302 | biostudies-literature
Project description:In clinical gait analysis, measurement errors impede the reliability and repeatability of the measurements. This extrinsic variability can potentially mislead the clinical interpretation of the analysis and should thus be minimised. Skin marker misplacement has been identified as the largest source of extrinsic variability between measurements. The goal of this study was to test whether the fusion of motion capture and 3D medical imaging could reduce extrinsic variability due to skin marker misplacement. The fusion method consists in using anatomical landmarks identified with 3D medical imaging to correct marker misplacements. To assess the reduction of variability accountable to the fusion method, skin marker misplacements were voluntarily introduced in the measurement of the pelvis and hip kinematics during gait for two patients scheduled for unilateral hip arthroplasty and two patients that underwent unilateral hip arthroplasty. The root mean square deviation was reduced by -78 ± 15% and the range of variability by -80 ± 16% for the pelvis and hip kinematics in average. These results showed that the fusion method could significantly reduce the extrinsic variability due to skin marker misplacement and thus increase the reliability and repeatability of motion capture measurements. However, the identification of anatomical landmarks via medical imaging is a new source of extrinsic variability that should be assessed before considering the fusion method for clinical applications.
| S-EPMC6988975 | biostudies-literature
Project description:Optical motion capture is a mature contemporary technique for the acquisition of motion data; alas, it is non-error-free. Due to technical limitations and occlusions of markers, gaps might occur in such recordings. The article reviews various neural network architectures applied to the gap-filling problem in motion capture sequences within the FBM framework providing a representation of body kinematic structure. The results are compared with interpolation and matrix completion methods. We found out that, for longer sequences, simple linear feedforward neural networks can outperform the other, sophisticated architectures, but these outcomes might be affected by the small amount of data availabe for training. We were also able to identify that the acceleration and monotonicity of input sequence are the parameters that have a notable impact on the obtained results.
| S-EPMC8472986 | biostudies-literature
Project description:Optical motion capture systems are state-of-the-art in motion acquisition; however, like any measurement system they are not error-free: noise is their intrinsic feature. The works so far mostly employ a simple noise model, expressing the uncertainty as a simple variance. In the work, we demonstrate that it might be not sufficient and we prove the existence of several types of noise and demonstrate how to quantify them using Allan variance. Such a knowledge is especially important for using optical motion capture to calibrate other techniques, and for applications requiring very fine quality of recording. For the automated readout of the noise coefficients, we solve the multidimensional regression problem using sophisticated metaheuristics in the exploration-exploitation scheme. We identified in the laboratory the notable contribution to the overall noise from white noise and random walk, and a minor contribution from blue noise and flicker, whereas the violet noise is absent. Besides classic types of noise we identified the presence of the correlated noises and periodic distortion. We analyzed also how the noise types scale with an increasing number of cameras. We had also the opportunity to observe the influence of camera failure on the overall performance.
| S-EPMC6832304 | biostudies-literature
Project description:BACKGROUND: Monitoring joint angles through wearable systems enables human posture and gesture to be reconstructed as a support for physical rehabilitation both in clinics and at the patient's home. A new generation of wearable goniometers based on knitted piezoresistive fabric (KPF) technology is presented. METHODS: KPF single-and double-layer devices were designed and characterized under stretching and bending to work as strain sensors and goniometers. The theoretical working principle and the derived electromechanical model, previously proved for carbon elastomer sensors, were generalized to KPF. The devices were used to correlate angles and piezoresistive fabric behaviour, to highlight the differences in terms of performance between the single layer and the double layer sensors. A fast calibration procedure is also proposed. RESULTS: The proposed device was tested both in static and dynamic conditions in comparison with standard electrogoniometers and inertial measurement units respectively. KPF goniometer capabilities in angle detection were experimentally proved and a discussion of the device measurement errors of is provided. The paper concludes with an analysis of sensor accuracy and hysteresis reduction in particular configurations. CONCLUSIONS: Double layer KPF goniometers showed a promising performance in terms of angle measurements both in quasi-static and dynamic working mode for velocities typical of human movement. A further approach consisting of a combination of multiple sensors to increase accuracy via sensor fusion technique has been presented.
| S-EPMC3996949 | biostudies-other