Reinforcement Learning and Control of a Lower Extremity Exoskeleton for Squat Assistance.
Ontology highlight
ABSTRACT:
SUBMITTER: Luo S
PROVIDER: S-EPMC8326457 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Few studies have systematically investigated robust controllers for lower limb rehabilitation exoskeletons (LLREs) that can safely and effectively assist users with a variety of neuromuscular disorders to walk with full autonomy. One of the key challenges for developing such a robust controller is to handle different degrees of uncertain human-exoskeleton interaction forces from the patients. Consequently, conventional walking controllers either are patient-condition specific or involve tuning of many control parameters, which could behave unreliably and even fail to maintain balance. We present a novel, deep neural network, reinforcement learning-based robust controller for a LLRE based on a decoupled offline human-exoskeleton simulation training with three independent networks, which aims to provide reliable walking assistance against various and uncertain human-exoskeleton interaction forces. The exoskeleton controller is driven by a neural network control policy that acts on a stream of the LLRE's proprioceptive signals, including joint kinematic states, and subsequently predicts real-time position control targets for the actuated joints. To handle uncertain human interaction forces, the control policy is trained intentionally with an integrated human musculoskeletal model and realistic human-exoskeleton interaction forces. Two other neural networks are connected with the control policy network to predict the interaction forces and muscle coordination. To further increase the robustness of the control policy to different human conditions, we employ domain randomization during training that includes not only randomization of exoskeleton dynamics properties but, more importantly, randomization of human muscle strength to simulate the variability of the patient's disability. Through this decoupled deep reinforcement learning framework, the trained controller of LLREs is able to provide reliable walking assistance to patients with different degrees of neuromuscular disorders without any control parameter tuning. A universal, RL-based walking controller is trained and virtually tested on a LLRE system to verify its effectiveness and robustness in assisting users with different disabilities such as passive muscles (quadriplegic), muscle weakness, or hemiplegic conditions without any control parameter tuning. Analysis of the RMSE for joint tracking, CoP-based stability, and gait symmetry shows the effectiveness of the controller. An ablation study also demonstrates the strong robustness of the control policy under large exoskeleton dynamic property ranges and various human-exoskeleton interaction forces. The decoupled network structure allows us to isolate the LLRE control policy network for testing and sim-to-real transfer since it uses only proprioception information of the LLRE (joint sensory state) as the input. Furthermore, the controller is shown to be able to handle different patient conditions without the need for patient-specific control parameter tuning.
| S-EPMC10024861 | biostudies-literature
Project description:Walking on different terrains leads to different biomechanics, which motivates the development of exoskeletons for assisting on walking according to the type of a terrain. The design of a lightweight soft exoskeleton that simultaneously assists multiple joints in the lower limb is presented in this paper. It is used to assist both hip and knee joints in a single system, the assistance force is directly applied to the hip joint flexion and the knee joint extension, while indirectly to the hip extension also. Based on the biological torque of human walking at three different slopes, a novel strategy is developed to improve the performance of assistance. A parameter optimal iterative learning control (POILC) method is introduced to reduce the error generated due to the difference between the wearing position and the biological features of the different wearers. In order to obtain the metabolic rate, three subjects walked on a treadmill, for 10 min on each terrain, at a speed of 4 km/h under both conditions of wearing and not wearing the soft exoskeleton. Results showed that the metabolic rate was decreased with the increasing slope of the terrain. The reductions in the net metabolic rate in the experiments on the downhill, flat ground, and uphill were, respectively, 9.86%, 12.48%, and 22.08% compared to the condition of not wearing the soft exoskeleton, where their corresponding absolute values were 0.28 W/kg, 0.72 W/kg, and 1.60 W/kg.
| S-EPMC7435451 | biostudies-literature
Project description:Among the infinite number of possible movements that can be produced, humans are commonly assumed to choose those that optimize criteria such as minimizing movement time, subject to certain movement constraints like signal-dependent and constant motor noise. While so far these assumptions have only been evaluated for simplified point-mass or planar models, we address the question of whether they can predict reaching movements in a full skeletal model of the human upper extremity. We learn a control policy using a motor babbling approach as implemented in reinforcement learning, using aimed movements of the tip of the right index finger towards randomly placed 3D targets of varying size. We use a state-of-the-art biomechanical model, which includes seven actuated degrees of freedom. To deal with the curse of dimensionality, we use a simplified second-order muscle model, acting at each degree of freedom instead of individual muscles. The results confirm that the assumptions of signal-dependent and constant motor noise, together with the objective of movement time minimization, are sufficient for a state-of-the-art skeletal model of the human upper extremity to reproduce complex phenomena of human movement, in particular Fitts' Law and the [Formula: see text] Power Law. This result supports the notion that control of the complex human biomechanical system can plausibly be determined by a set of simple assumptions and can easily be learned.
| S-EPMC8280157 | biostudies-literature
Project description:Assistive exoskeletons can reduce the metabolic cost of walking, and recent advances in exoskeleton device design and control have resulted in large metabolic savings. Most exoskeleton devices provide assistance at either the ankle or hip. Exoskeletons that assist multiple joints have the potential to provide greater metabolic savings, but can require many actuators and complicated controllers, making it difficult to design effective assistance. Coupled assistance, when two or more joints are assisted using one actuator or control signal, could reduce control dimensionality while retaining metabolic benefits. However, it is unknown which combinations of assisted joints are most promising and if there are negative consequences associated with coupled assistance. Since designing assistance with human experiments is expensive and time-consuming, we used musculoskeletal simulation to evaluate metabolic savings from multi-joint assistance and identify promising joint combinations. We generated 2D muscle-driven simulations of walking while simultaneously optimizing control strategies for simulated lower-limb exoskeleton assistive devices to minimize metabolic cost. Each device provided assistance either at a single joint or at multiple joints using massless, ideal actuators. To assess if control could be simplified for multi-joint exoskeletons, we simulated different control strategies in which the torque provided at each joint was either controlled independently or coupled between joints. We compared the predicted optimal torque profiles and changes in muscle and total metabolic power consumption across the single joint and multi-joint assistance strategies. We found multi-joint devices-whether independent or coupled-provided 50% greater metabolic savings than single joint devices. The coupled multi-joint devices were able to achieve most of the metabolic savings produced by independently-controlled multi-joint devices. Our results indicate that device designers could simplify multi-joint exoskeleton designs by reducing the number of torque control parameters through coupling, while still maintaining large reductions in metabolic cost.
| S-EPMC8730392 | biostudies-literature
Project description:BackgroundMany gait impairments are characterized by asymmetry and result in reduced mobility. Exoskeletons could be useful for restoring gait symmetry by assisting only one leg. However, we still have limited understanding of the effects of unilateral exoskeleton assistance. Our aim was to compare the effects of unilateral and bilateral assistance using a within-subject study design.MethodsEleven participants walked in different exoskeleton conditions. In the Unilateral conditions, only one leg was assisted. In Bilateral Matched Total Work, half of the assistance from the Unilateral conditions was applied to both legs such that the bilateral sum was equal to that of the Unilateral conditions. In Bilateral Matched Work Per Leg, the same assistance as in the Unilateral conditions was provided to both legs such that the bilateral sum was the double of that of the Unilateral conditions. In the Powered-Off condition, no assistance was provided. We measured metabolic energy consumption, exoskeleton mechanics and kinematics.ResultsOn average, the Unilateral, Bilateral Matched Total Work and Bilateral Matched Work Per Leg conditions reduced the metabolic rate by 7, 11 and 15%, respectively, compared with the Powered-Off condition. A possible explanation for why the Unilateral conditions effectively reduced the metabolic rate could be that they caused only very little asymmetry in gait biomechanics, except at the ankle and in the horizontal center-of-mass velocity. We found the highest ratio of metabolic rate reduction versus positive work assistance with bilateral assistance and low work per leg (Bilateral Matched Total Work). Statistical analysis indicated that assistance symmetry and assistance per leg are more important than the bilateral summed assistance for reducing the metabolic rate of walking.ConclusionsThese data bridge the gap between conclusions from studies with unilateral and bilateral exoskeletons and inform how unilateral assistance can be used to influence gait parameters, such as center-of-mass velocity.
| S-EPMC6085709 | biostudies-literature
Project description:Recent years have witnessed an explosion in the application of microfluidic techniques to a wide variety of problems in the chemical and biological sciences. Despite the many considerable advantages that microfluidic systems bring to experimental science, microfluidic platforms often exhibit inconsistent system performance when operated over extended timescales. Such variations in performance are because of a multiplicity of factors, including microchannel fouling, substrate deformation, temperature and pressure fluctuations, and inherent manufacturing irregularities. The introduction and integration of advanced control algorithms in microfluidic platforms can help mitigate such inconsistencies, paving the way for robust and repeatable long-term experiments. Herein, two state-of-the-art reinforcement learning algorithms, based on Deep Q-Networks and model-free episodic controllers, are applied to two experimental "challenges," involving both continuous-flow and segmented-flow microfluidic systems. The algorithms are able to attain superhuman performance in controlling and processing each experiment, highlighting the utility of novel control algorithms for automated high-throughput microfluidic experimentation.
| S-EPMC6644574 | biostudies-literature
Project description:Self-selected walking speed is an important aspect of mobility. Exoskeletons can increase walking speed, but the mechanisms behind these changes and the upper limits on performance are unknown. Human-in-the-loop optimization is a technique for identifying exoskeleton characteristics that maximize the benefits of assistance, which has been critical to achieving large improvements in energy economy. In this study, we used human-in-the-loop optimization to test whether large improvements in self-selected walking speed are possible through ankle exoskeleton assistance. Healthy participants (N =10) were instructed to walk at a comfortable speed on a self-paced treadmill while wearing tethered ankle exoskeletons. An algorithm sequentially applied different patterns of exoskeleton torque and estimated the speed-optimal pattern, which was then evaluated in separate trials. With torque optimized for speed, participants walked 42% faster than in normal shoes (1.83 ms-1 vs. 1.31 ms-1; Tukey HSD, p = 4 ×10-8 ), with speed increases ranging from 6% to 91%. Participants walked faster with speed-optimized torque than with torque optimized for energy consumption (1.55 ms-1) or torque chosen to induce slow walking (1.18 ms-1). Gait characteristics with speed-optimized torque were highly variable across participants, and changes in metabolic cost of transport ranged from a 31% decrease to a 78% increase, with a decrease of 2% on average. These results demonstrate that ankle exoskeletons can facilitate large increases in self-selected walking speed, which could benefit older adults and others with reduced walking speed.
| S-EPMC8172358 | biostudies-literature
Project description:Adaptive decision making in dynamic environments requires multiple reinforcement-learning steps that may be implemented by dissociable neural circuits. Here, we used a novel directionally specific viral ablation approach to investigate the function of several anatomically defined orbitofrontal cortex (OFC) circuits during adaptive, flexible decision making in rats trained on a probabilistic reversal learning task. Ablation of OFC neurons projecting to the nucleus accumbens selectively disrupted performance following a reversal, by disrupting the use of negative outcomes to guide subsequent choices. Ablation of amygdala neurons projecting to the OFC also impaired reversal performance, but due to disruptions in the use of positive outcomes to guide subsequent choices. Ablation of OFC neurons projecting to the amygdala, by contrast, enhanced reversal performance by destabilizing action values. Our data are inconsistent with a unitary function of the OFC in decision making. Rather, distinct OFC-amygdala-striatal circuits mediate distinct components of the action-value updating and maintenance necessary for decision making.
| S-EPMC6893860 | biostudies-literature
Project description:Stroke is the second leading cause of death and one of the leading causes of disability in the world. According to the World Health Organization, 11 million people suffer a stroke yearly. The cost of the disease is exorbitant, and the most widely used treatment is conventional physiotherapy. Therefore, assistive technology emerges to optimize rehabilitation and functional capabilities, but cost, robustness, usability, and long-term results still restrict the technology selection. This work aimed to develop a low-cost ankle orthosis, the G-Exos, a wearable exoskeleton to increase motor capability by assisting dorsiflexion, plantarflexion, and ankle stability. A hybrid system provided near-natural gait movements using active, motor, and passive assistance, elastic band. The system was validated with 10 volunteers with foot drop: seven with stroke, two with incomplete spinal cord injury (SCI), and one with acute inflammatory transverse myelitis (ATM). The G-Exos showed assistive functionality for gait movement. A Friedman test showed a significant difference in dorsiflexion amplitude with the use of the G-Exos compared to gait without the use of the G-Exos [x 2 (3) = 98.56, p < 0.001]. In addition, there was also a significant difference in ankle eversion and inversion comparing walking with and without the G-Exos [x 2 (3) = 36.12, p < 0.001]. The G-Exos is a robust, lightweight, and flexible assistive technology device to detect the gait phase accurately and provide better human-machine interaction. G-Exos training improved capability to deal with gait disorders, usability, and motor and functional recovery. Wearable assistive technologies lead to a better quality of life and contribute using in activities of daily living.
| S-EPMC9684314 | biostudies-literature
Project description:Deep reinforcement learning (DRL) is applied to control a nonlinear, chaotic system governed by the one-dimensional Kuramoto-Sivashinsky (KS) equation. DRL uses reinforcement learning principles for the determination of optimal control solutions and deep neural networks for approximating the value function and the control policy. Recent applications have shown that DRL may achieve superhuman performance in complex cognitive tasks. In this work, we show that using restricted localized actuation, partial knowledge of the state based on limited sensor measurements and model-free DRL controllers, it is possible to stabilize the dynamics of the KS system around its unstable fixed solutions, here considered as target states. The robustness of the controllers is tested by considering several trajectories in the phase space emanating from different initial conditions; we show that DRL is always capable of driving and stabilizing the dynamics around target states. The possibility of controlling the KS system in the chaotic regime by using a DRL strategy solely relying on local measurements suggests the extension of the application of RL methods to the control of more complex systems such as drag reduction in bluff-body wakes or the enhancement/diminution of turbulent mixing.
| S-EPMC6894543 | biostudies-literature