Emotion Recognition Using Eye-Tracking: Taxonomy, Review and Current Challenges.
Ontology highlight
ABSTRACT:
SUBMITTER: Lim JZ
PROVIDER: S-EPMC7219342 | biostudies-literature | 2020 Apr
REPOSITORIES: biostudies-literature
Publications
Emotion Recognition Using Eye-Tracking: Taxonomy, Review and Current Challenges.
Lim Jia Zheng JZ Mountstephens James J Teo Jason J
Sensors (Basel, Switzerland) 20200422 8
The ability to detect users' emotions for the purpose of emotion engineering is currently one of the main endeavors of machine learning in affective computing. Among the more common approaches to emotion detection are methods that rely on electroencephalography (EEG), facial image processing and speech inflections. Although eye-tracking is fast in becoming one of the most commonly used sensor modalities in affective computing, it is still a relatively new approach for emotion detection, especial ...[more]
Similar Datasets
Project description:BackgroundCurrent tasks measuring social cognition are usually 'pen and paper' tasks, have ceiling effects and include complicated test instructions that may be difficult to understand for those with cognitive impairment. We therefore aimed to develop a set of simple, instructionless, quantitative, tasks of emotion recognition using the methodology of eye tracking, with the subsequent aim of assessing their utility in individuals with behavioural variant frontotemporal dementia (bvFTD).MethodsUsing the Eyelink 1000 Plus eye tracker, 18 bvFTD and 22 controls completed tasks of simple and complex emotion recognition that involved viewing four images (one target face (simple) or pair of eyes (complex) and the others non-target) followed by a target emotion word and lastly the original four images alongside the emotion word. A dwell time change score was then calculated as the main outcome measure by subtracting the percentage dwell time for the target image before the emotion word appeared away from the percentage dwell time for the target image after the emotion word appeared. All participants also underwent a standard cognitive battery and volumetric T1-weighted magnetic resonance imaging.ResultsAnalysis using a mixed effects model showed that the average (standard deviation) mean dwell time change score in the target interest area was 35 (27)% for the control group compared with only 4 (18)% for the bvFTD group (p < 0.05) for the simple emotion recognition task, and 15 (26)% for the control group compared with only 2 (18)% for the bvFTD group (p < 0.05) for the complex emotion recognition task. Worse performance in the bvFTD group correlated with atrophy in the right ventromedial prefrontal and orbitofrontal cortices, brain regions previously implicated in social cognition.ConclusionsIn summary, eye tracking is a viable tool for assessing social cognition in individuals with bvFTD, being well-tolerated and able to overcome some of the problems associated with standard psychometric tasks.
| S-EPMC7871635 | biostudies-literature
Project description:Strabismus is one of the most common vision diseases that would cause amblyopia and even permanent vision loss. Timely diagnosis is crucial for well treating strabismus. In contrast to manual diagnosis, automatic recognition can significantly reduce labor cost and increase diagnosis efficiency. In this paper, we propose to recognize strabismus using eye-tracking data and convolutional neural networks. In particular, an eye tracker is first exploited to record a subject's eye movements. A gaze deviation (GaDe) image is then proposed to characterize the subject's eye-tracking data according to the accuracies of gaze points. The GaDe image is fed to a convolutional neural network (CNN) that has been trained on a large image database called ImageNet. The outputs of the full connection layers of the CNN are used as the GaDe image's features for strabismus recognition. A dataset containing eye-tracking data of both strabismic subjects and normal subjects is established for experiments. Experimental results demonstrate that the natural image features can be well transferred to represent eye-tracking data, and strabismus can be effectively recognized by our proposed method.
| S-EPMC5944293 | biostudies-literature
Project description:BackgroundCriminal associates such as terrorist members are likely to deny knowing members of their network when questioned by police. Eye tracking research suggests that lies about familiar faces can be detected by distinct markers of recognition (e.g. fewer fixations and longer fixation durations) across multiple eye fixation parameters. However, the effect of explicit eye movement strategies to concealed recognition on such markers has not been examined. Our aim was to assess the impact of fixed-sequence eye movement strategies (across the forehead, ears, eyes, nose, mouth and chin) on markers of familiar face recognition. Participants were assigned to one of two groups: a standard guilty group who were simply instructed to conceal knowledge but with no specific instructions on how to do so; and a countermeasures group who were instructed to look at every familiar and unfamiliar face in the same way by executing a consistent sequence of fixations.ResultsIn the standard guilty group, lies about recognition of familiar faces showed longer average fixation durations, a lower proportion of fixations to the inner face regions, and proportionately more viewing of the eyes than honest responses to genuinely unknown faces. In the countermeasures condition, familiar face recognition was detected by longer fixations durations, fewer fixations to the inner regions of the face, and fewer interest areas of the face viewed. Longer fixation durations were a consistent marker of recognition across both conditions for most participants; differences were detectable from the first fixation.ConclusionThe results suggest that individuals can exert a degree of executive control over fixation patterns but that: the eyes are particularly attention-grabbing for familiar faces; the more viewers look around the face, the more they give themselves away; and attempts to deploy the same fixation patterns to familiar and unfamiliar faces were unsuccessful. The results suggest that the best strategy for concealing recognition might be to keep the eyes fixated in the centre of the screen but, even then, recognition is apparent in longer fixation durations. We discuss potential optimal conditions for detecting concealed knowledge of faces.
| S-EPMC6684707 | biostudies-literature
Project description:Adolescents with Conduct Disorder (CD) show deficits in recognizing facial expressions of emotion, but it is not known whether these difficulties extend to other social cues, such as emotional body postures. Moreover, in the absence of eye-tracking data, it is not known whether such deficits, if present, are due to a failure to attend to emotionally informative regions of the body. Male and female adolescents with CD and varying levels of callous-unemotional (CU) traits (n = 45) and age- and sex-matched typically-developing controls (n = 51) categorized static and dynamic emotional body postures. The emotion categorization task was paired with eye-tracking methods to investigate relationships between fixation behavior and recognition performance. Having CD was associated with impaired recognition of static and dynamic body postures and atypical fixation behavior. Furthermore, males were less likely to fixate emotionally-informative regions of the body than females. While we found no effects of CU traits on body posture recognition, the effects of CU traits on fixation behavior varied according to CD status and sex, with CD males with lower levels of CU traits showing the most atypical fixation behavior. Critically, atypical fixation behavior did not explain the body posture recognition deficits observed in CD. Our findings suggest that CD-related impairments in recognition of body postures of emotion are not due to attentional issues. Training programmes designed to ameliorate the emotion recognition difficulties associated with CD may need to incorporate a body posture component.
| S-EPMC8154802 | biostudies-literature
Project description:Self-face recognition has been shown to be impaired in schizophrenia (SZ), according to studies using behavioral tasks implicating cognitive demands. Here, we employed an eye-tracking methodology, which is a relevant tool to understand impairments in self-face recognition deficits in SZ because it provides a natural, continuous and online record of face processing. Moreover, it allows collecting the most relevant and informative features each individual looks at during the self-face recognition. These advantages are especially relevant considering the fundamental role played by the patterns of visual exploration on face processing. Thus, this paper aims to investigate self-face recognition deficits in SZ using eye-tracking methodology. Visual scan paths were monitored in 20 patients with SZ and 20 healthy controls. Self, famous, and unknown faces were morphed in steps of 20%. Location, number, and duration of fixations on relevant areas were recorded with an eye-tracking system. Participants performed a passive exploration task (no specific instruction was provided), followed by an active decision making task (individuals were explicitly requested to recognize the different faces). Results showed that patients with SZ had fewer and longer fixations compared to controls. Nevertheless, both groups focused their attention on relevant facial features in a similar way. No significant difference was found between groups when participants were requested to recognize the faces (active task). In conclusion, using an eye tracking methodology and two tasks with low levels of cognitive demands, our results suggest that patients with SZ are able to: (1) explore faces and focus on relevant features of the face in a similar way as controls; and (2) recognize their own face.
| S-EPMC4748053 | biostudies-literature
Project description:Emotions are fundamental for human beings and play an important role in human cognition. Emotion is commonly associated with logical decision making, perception, human interaction, and to a certain extent, human intelligence itself. With the growing interest of the research community towards establishing some meaningful “emotional” interactions between humans and computers, the need for reliable and deployable solutions for the identification of human emotional states is required. Recent developments in using electroencephalography (EEG) for emotion recognition have garnered strong interest from the research community as the latest developments in consumer-grade wearable EEG solutions can provide a cheap, portable, and simple solution for identifying emotions. Since the last comprehensive review was conducted back from the years 2009 to 2016, this paper will update on the current progress of emotion recognition using EEG signals from 2016 to 2019. The focus on this state-of-the-art review focuses on the elements of emotion stimuli type and presentation approach, study size, EEG hardware, machine learning classifiers, and classification approach. From this state-of-the-art review, we suggest several future research opportunities including proposing a different approach in presenting the stimuli in the form of virtual reality (VR). To this end, an additional section devoted specifically to reviewing only VR studies within this research domain is presented as the motivation for this proposed new approach using VR as the stimuli presentation device. This review paper is intended to be useful for the research community working on emotion recognition using EEG signals as well as for those who are venturing into this field of research.
| S-EPMC7516734 | biostudies-literature
Project description:ObjectivesFor children to understand the emotional behavior of others, the first two steps involve emotion encoding and emotion interpreting, according to the Social Information Processing model. Access to daily social interactions is prerequisite to a child acquiring these skills, and barriers to communication such as hearing loss impede this access. Therefore, it could be challenging for children with hearing loss to develop these two skills. The present study aimed to understand the effect of prelingual hearing loss on children's emotion understanding, by examining how they encode and interpret nonverbal emotional cues in dynamic social situations.DesignSixty deaf or hard-of-hearing (DHH) children and 71 typically hearing (TH) children (3-10 years old, mean age 6.2 years, 54% girls) watched videos of prototypical social interactions between a target person and an interaction partner. At the end of each video, the target person did not face the camera, rendering their facial expressions out of view to participants. Afterward, participants were asked to interpret the emotion they thought the target person felt at the end of the video. As participants watched the videos, their encoding patterns were examined by an eye tracker, which measured the amount of time participants spent looking at the target person's head and body and at the interaction partner's head and body. These regions were preselected for analyses because they had been found to provide cues for interpreting people's emotions and intentions.ResultsWhen encoding emotional cues, both the DHH and TH children spent more time looking at the head of the target person and at the head of the interaction partner than they spent looking at the body or actions of either person. Yet, compared with the TH children, the DHH children looked at the target person's head for a shorter time (b = -0.03, p = 0.030), and at the target person's body (b = 0.04, p = 0.006) and at the interaction partner's head (b = 0.03, p = 0.048) for a longer time. The DHH children were also less accurate when interpreting emotions than their TH peers (b = -0.13, p = 0.005), and their lower scores were associated with their distinctive encoding pattern.ConclusionsThe findings suggest that children with limited auditory access to the social environment tend to collect visually observable information to compensate for ambiguous emotional cues in social situations. These children may have developed this strategy to support their daily communication. Yet, to fully benefit from such a strategy, these children may need extra support for gaining better social-emotional knowledge.
| S-EPMC8221710 | biostudies-literature
Project description:Facial emotion recognition (FER) is a computer vision process aimed at detecting and classifying human emotional expressions. FER systems are currently used in a vast range of applications from areas such as education, healthcare, or public safety; therefore, detection and recognition accuracies are very important. Similar to any computer vision task based on image analyses, FER solutions are also suitable for integration with artificial intelligence solutions represented by different neural network varieties, especially deep neural networks that have shown great potential in the last years due to their feature extraction capabilities and computational efficiency over large datasets. In this context, this paper reviews the latest developments in the FER area, with a focus on recent neural network models that implement specific facial image analysis algorithms to detect and recognize facial emotions. This paper's scope is to present from historical and conceptual perspectives the evolution of the neural network architectures that proved significant results in the FER area. This paper endorses convolutional neural network (CNN)-based architectures against other neural network architectures, such as recurrent neural networks or generative adversarial networks, highlighting the key elements and performance of each architecture, and the advantages and limitations of the proposed models in the analyzed papers. Additionally, this paper presents the available datasets that are currently used for emotion recognition from facial expressions and micro-expressions. The usage of FER systems is also highlighted in various domains such as healthcare, education, security, or social IoT. Finally, open issues and future possible developments in the FER area are identified.
| S-EPMC10458371 | biostudies-literature
Project description:Studies of high-acuity visual cortical processing have been limited by the inability to track eye position with sufficient accuracy to precisely reconstruct the visual stimulus on the retina. As a result, studies of primary visual cortex (V1) have been performed almost entirely on neurons outside the high-resolution central portion of the visual field (the fovea). Here we describe a procedure for inferring eye position using multi-electrode array recordings from V1 coupled with nonlinear stimulus processing models. We show that this method can be used to infer eye position with 1 arc-min accuracy--significantly better than conventional techniques. This allows for analysis of foveal stimulus processing, and provides a means to correct for eye movement-induced biases present even outside the fovea. This method could thus reveal critical insights into the role of eye movements in cortical coding, as well as their contribution to measures of cortical variability.
| S-EPMC4159777 | biostudies-literature
Project description:Eye tracking systems have recently experienced a diversity of novel calibration procedures, including smooth pursuit and vestibulo-ocular reflex based calibrations. These approaches allowed collecting more data compared to the standard 9-point calibration. However, the computation of the mapping function which provides planar gaze positions from pupil features given as input is mostly based on polynomial regressions, and little work has investigated alternative approaches. This paper fills this gap by providing a new calibration computation method based on symbolic regression. Instead of making prior assumptions on the polynomial transfer function between input and output records, symbolic regression seeks an optimal model among different types of functions and their combinations. This approach offers an interesting perspective in terms of flexibility and accuracy. Therefore, we designed two experiments in which we collected ground truth data to compare vestibulo-ocular and smooth pursuit calibrations based on symbolic regression, both using a marker or a finger as a target, resulting in four different calibrations. As a result, we improved calibration accuracy by more than 30%, with reasonable extra computation time.
| S-EPMC6420251 | biostudies-literature