Dataset Information

Reference-point centering and range-adaptation enhance human reinforcement learning at the cost of irrational preferences.

ABSTRACT: In economics and perceptual decision-making contextual effects are well documented, where decision weights are adjusted as a function of the distribution of stimuli. Yet, in reinforcement learning literature whether and how contextual information pertaining to decision states is integrated in learning algorithms has received comparably little attention. Here, we investigate reinforcement learning behavior and its computational substrates in a task where we orthogonally manipulate outcome valence and magnitude, resulting in systematic variations in state-values. Model comparison indicates that subjects' behavior is best accounted for by an algorithm which includes both reference point-dependence and range-adaptation-two crucial features of state-dependent valuation. In addition, we find that state-dependent outcome valuation progressively emerges, is favored by increasing outcome information and correlated with explicit understanding of the task structure. Finally, our data clearly show that, while being locally adaptive (for instance in negative valence and small magnitude contexts), state-dependent valuation comes at the cost of seemingly irrational choices, when options are extrapolated out from their original contexts.

SUBMITTER: Bavard S

PROVIDER: S-EPMC6206161 | biostudies-literature | 2018 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Reference-point centering and range-adaptation enhance human reinforcement learning at the cost of irrational preferences.

Bavard Sophie S Lebreton Maël M Khamassi Mehdi M Coricelli Giorgio G Palminteri Stefano S

Nature communications 20181029 1

In economics and perceptual decision-making contextual effects are well documented, where decision weights are adjusted as a function of the distribution of stimuli. Yet, in reinforcement learning literature whether and how contextual information pertaining to decision states is integrated in learning algorithms has received comparably little attention. Here, we investigate reinforcement learning behavior and its computational substrates in a task where we orthogonally manipulate outcome valence ...[more]

PMID: 30374019

Similar Datasets

Project description:Aim The European Union (EU) has received criticism for being slow to secure coronavirus disease (COVID-19) vaccine contracts in 2020 before the approval of the first COVID-19 vaccine. This study aimed to retrospectively analyze the EU’s COVID-19 vaccine procurement strategy. To this end, the study retrospectively determined the minimum vaccine efficacy that made vaccination cost-effective from a societal perspective in Germany before clinical trial announcements in late 2020. The results were compared with the expected vaccine efficacy before the announcements. Methods Two strategies were analyzed: vaccination followed by the complete lifting of mitigation measures and a long-term mitigation strategy. A decision model was constructed using, for example, information on age-specific fatality rates, intensive care unit costs and outcomes, and herd protection thresholds. The base-case time horizon was 5 years. Cost-effectiveness of vaccination was determined in terms of the costs per life-year gained. The value of an additional life-year was borrowed from new, innovative oncological drugs, as cancer is a condition with a perceived threat similar to that of COVID-19. Results A vaccine with 50% efficacy against death due to COVID-19 was not clearly cost-effective compared with a long-term mitigation strategy if mitigation measures were planned to be lifted after vaccine rollout. The minimum vaccine efficacy required to achieve cost-effectiveness was 40% in the base case. The sensitivity analysis showed considerable variation around the minimum vaccine efficacy, extending above 50% for some of the input variables. Conclusions This study showed that vaccine efficacy levels expected before clinical trial announcements did not clearly justify lifting mitigation measures from a cost-effectiveness standpoint. Hence, the EU’s sluggish procurement strategy still appeared to be rational at the time of decision making. Supplementary Information The online version contains supplementary material available at 10.1186/s12913-022-08726-4.

Project description:To investigate deep reinforcement learning (DRL) based on historical treatment plans for developing automated radiation adaptation protocols for nonsmall cell lung cancer (NSCLC) patients that aim to maximize tumor local control at reduced rates of radiation pneumonitis grade 2 (RP2).In a retrospective population of 114 NSCLC patients who received radiotherapy, a three-component neural networks framework was developed for deep reinforcement learning (DRL) of dose fractionation adaptation. Large-scale patient characteristics included clinical, genetic, and imaging radiomics features in addition to tumor and lung dosimetric variables. First, a generative adversarial network (GAN) was employed to learn patient population characteristics necessary for DRL training from a relatively limited sample size. Second, a radiotherapy artificial environment (RAE) was reconstructed by a deep neural network (DNN) utilizing both original and synthetic data (by GAN) to estimate the transition probabilities for adaptation of personalized radiotherapy patients' treatment courses. Third, a deep Q-network (DQN) was applied to the RAE for choosing the optimal dose in a response-adapted treatment setting. This multicomponent reinforcement learning approach was benchmarked against real clinical decisions that were applied in an adaptive dose escalation clinical protocol. In which, 34 patients were treated based on avid PET signal in the tumor and constrained by a 17.2% normal tissue complication probability (NTCP) limit for RP2. The uncomplicated cure probability (P+) was used as a baseline reward function in the DRL.Taking our adaptive dose escalation protocol as a blueprint for the proposed DRL (GAN + RAE + DQN) architecture, we obtained an automated dose adaptation estimate for use at ?2/3 of the way into the radiotherapy treatment course. By letting the DQN component freely control the estimated adaptive dose per fraction (ranging from 1-5 Gy), the DRL automatically favored dose escalation/de-escalation between 1.5 and 3.8 Gy, a range similar to that used in the clinical protocol. The same DQN yielded two patterns of dose escalation for the 34 test patients, but with different reward variants. First, using the baseline P+ reward function, individual adaptive fraction doses of the DQN had similar tendencies to the clinical data with an RMSE = 0.76 Gy; but adaptations suggested by the DQN were generally lower in magnitude (less aggressive). Second, by adjusting the P+ reward function with higher emphasis on mitigating local failure, better matching of doses between the DQN and the clinical protocol was achieved with an RMSE = 0.5 Gy. Moreover, the decisions selected by the DQN seemed to have better concordance with patients eventual outcomes. In comparison, the traditional temporal difference (TD) algorithm for reinforcement learning yielded an RMSE = 3.3 Gy due to numerical instabilities and lack of sufficient learning.We demonstrated that automated dose adaptation by DRL is a feasible and a promising approach for achieving similar results to those chosen by clinicians. The process may require customization of the reward function if individual cases were to be considered. However, development of this framework into a fully credible autonomous system for clinical decision support would require further validation on larger multi-institutional datasets.

Dataset Information

Reference-point centering and range-adaptation enhance human reinforcement learning at the cost of irrational preferences.

Publications

Reference-point centering and range-adaptation enhance human reinforcement learning at the cost of irrational preferences.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets