Dataset Information

Adaptive user interface design and analysis using emotion recognition through facial expressions and body posture from an RGB-D sensor.

ABSTRACT: This work presents the design and analysis of an Adaptive User Interface (AUI) for a desktop application that uses a novel solution for the recognition of the emotional state of a user through both facial expressions and body posture from an RGB-D sensor. Six basic emotions are recognized through facial expressions in addition to the physiological state, which is recognized through the body posture. The facial expressions and body posture are acquired in real-time from a Kinect sensor. A scoring system is used to improve recognition by minimizing the confusion between the different emotions. The implemented solution achieves an accuracy rate of above 90%. The recognized emotion is then used to derive an Automatic AUI where the user can use speech commands to modify the User Interface (UI) automatically. A comprehensive user study is performed to compare the usability of an Automatic, Manual, and a Hybrid AUI. The AUIs are evaluated in terms of their efficiency, effectiveness, productivity, and error safety. Additionally, a comprehensive analysis is performed to evaluate the results from the viewpoint of different genders and age groups. Results show that the hybrid adaptation improves usability in terms of productivity and efficiency. Finally, a combination of both automatic and hybrid AUIs result in significantly positive user experience compared to the manual adaptation.

SUBMITTER: Medjden S

PROVIDER: S-EPMC7365406 | biostudies-literature | 2020

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Adaptive user interface design and analysis using emotion recognition through facial expressions and body posture from an RGB-D sensor.

Medjden Selma S Ahmed Naveed N Lataifeh Mohammed M

PloS one 20200716 7

This work presents the design and analysis of an Adaptive User Interface (AUI) for a desktop application that uses a novel solution for the recognition of the emotional state of a user through both facial expressions and body posture from an RGB-D sensor. Six basic emotions are recognized through facial expressions in addition to the physiological state, which is recognized through the body posture. The facial expressions and body posture are acquired in real-time from a Kinect sensor. A scoring ...[more]

PMID: 32673325

Similar Datasets

Project description:During real-life interactions, facial expressions of emotion are perceived dynamically with multimodal sensory information. In the absence of auditory sensory channel inputs, it is unclear how facial expressions are recognised and internally represented by deaf individuals. Few studies have investigated facial expression recognition in deaf signers using dynamic stimuli, and none have included all six basic facial expressions of emotion (anger, disgust, fear, happiness, sadness, and surprise) with stimuli fully controlled for their low-level visual properties, leaving the question of whether or not a dynamic advantage for deaf observers exists unresolved. We hypothesised, in line with the enhancement hypothesis, that the absence of auditory sensory information might have forced the visual system to better process visual (unimodal) signals, and predicted that this greater sensitivity to visual stimuli would result in better recognition performance for dynamic compared to static stimuli, and for deaf-signers compared to hearing non-signers in the dynamic condition. To this end, we performed a series of psychophysical studies with deaf signers with early-onset severe-to-profound deafness (dB loss >70) and hearing controls to estimate their ability to recognize the six basic facial expressions of emotion. Using static, dynamic, and shuffled (randomly permuted video frames of an expression) stimuli, we found that deaf observers showed similar categorization profiles and confusions across expressions compared to hearing controls (e.g., confusing surprise with fear). In contrast to our hypothesis, we found no recognition advantage for dynamic compared to static facial expressions for deaf observers. This observation shows that the decoding of dynamic facial expression emotional signals is not superior even in the deaf expert visual system, suggesting the existence of optimal signals in static facial expressions of emotion at the apex. Deaf individuals match hearing individuals in the recognition of facial expressions of emotion.

Project description:BackgroundThe emotion recognition task (ERT) was developed to overcome shortcomings of static emotion recognition paradigms, by identifying more subtle deficits in emotion recognition across different intensity levels. In this study, we used the ERT to investigate emotion recognition deficits across the frontotemporal (FTD) and Alzheimer's Dementia (AD) spectrum.MethodsWith the ERT, we assessed the recognition of facial emotional expressions (anger-disgust-fear-happiness-sadness-surprise) across four intensities (40-60-80-100%) in patients with behavioural variant FTD (bvFTD; n = 32), and AD (n = 32), presymptomatic FTD mutation carriers (n = 47) and controls (n = 49). We examined group differences using multilevel linear regression with age, sex and education level as covariates, and performed post hoc analyses on presymptomatic (MAPT, GRN and C9orf72) mutation carriers. Classification abilities were investigated by means of logistic regression.ResultsLowest ERT total scores were found in patients with bvFTD and AD, whereas equal highest performance was found in presymptomatic mutation carriers and controls. For all emotions, significantly lower subscores were found in patients with bvFTD than in presymptomatic mutation carriers and in controls (highest p value = 0.025). Patients with bvFTD performed lower than patients with AD on anger (p = 0.005) and a trend towards significance was found for a lower performance on happiness (p = 0.065). Task performance increased with higher emotional intensity, and classification was better at the lowest than at the highest intensity. C9orf72 mutation carriers performed worse on recognizing anger at the lowest intensity than GRN mutation carriers (p = 0.047) and controls (p = 0.038). The ERT differentiated between patients with bvFTD and controls, and between patients with AD and controls (both p < 0.001).DiscussionOur results demonstrate emotion recognition deficits in both bvFTD and AD, and suggest the presence of subtle emotion recognition changes in presymptomatic C9orf72-FTD. This highlights the importance of incorporating emotion recognition paradigms into standard neuropsychological assessment for early differential diagnosis, and as clinical endpoints in upcoming therapeutic trials.

Project description:Facial mimicry is commonly defined as the tendency to imitate-at a sub-threshold level-facial expressions of other individuals. Numerous studies support a role of facial mimicry in recognizing others' emotions. However, the underlying functional mechanism is unclear. A prominent hypothesis considers facial mimicry as based on an action-perception loop, leading to the prediction that facial mimicry should be observed only when processing others' facial expressions. Nevertheless, previous studies have also detected facial mimicry during observation of emotional bodily expressions. An emergent alternative hypothesis is that facial mimicry overtly reflects the simulation of an "emotion", rather than the reproduction of a specific observed motor pattern. In the present study, we tested whether blocking mimicry ("Bite") on the lower face disrupted recognition of happy expressions conveyed by either facial or body expressions. In Experiment 1, we tested participants' ability to identify happy, fearful and neutral expressions in the Bite condition and in two control conditions. In Experiment 2, to ensure that such a manipulation selectively affects emotion recognition, we tested participants' ability to recognize emotional expressions, as well as the actors' gender, under the Bite condition and a control condition. Finally, we investigated the relationship between dispositional empathy and emotion recognition under the condition of blocked mimicry. Our findings demonstrated that blocking mimicry on the lower face hindered recognition of happy facial and body expressions, while the recognition of neutral and fearful expressions was not affected by the mimicry manipulation. The mimicry manipulation did not affect the gender discrimination task. Furthermore, the impairment of happy expression recognition correlated with empathic traits. These results support the role of facial mimicry in emotion recognition and suggest that facial mimicry reflects a global sensorimotor simulation of others' emotions rather than a muscle-specific reproduction of an observed motor expression.

Dataset Information

Adaptive user interface design and analysis using emotion recognition through facial expressions and body posture from an RGB-D sensor.

Publications

Adaptive user interface design and analysis using emotion recognition through facial expressions and body posture from an RGB-D sensor.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets