Dataset Information

Multimodal Neural Network for Rapid Serial Visual Presentation Brain Computer Interface.

ABSTRACT: Brain computer interfaces allow users to preform various tasks using only the electrical activity of the brain. BCI applications often present the user a set of stimuli and record the corresponding electrical response. The BCI algorithm will then have to decode the acquired brain response and perform the desired task. In rapid serial visual presentation (RSVP) tasks, the subject is presented with a continuous stream of images containing rare target images among standard images, while the algorithm has to detect brain activity associated with target images. In this work, we suggest a multimodal neural network for RSVP tasks. The network operates on the brain response and on the initiating stimulus simultaneously, providing more information for the BCI application. We present two variants of the multimodal network, a supervised model, for the case when the targets are known in advanced, and a semi-supervised model for when the targets are unknown. We test the neural networks with a RSVP experiment on satellite imagery carried out with two subjects. The multimodal networks achieve a significant performance improvement in classification metrics. We visualize what the networks has learned and discuss the advantages of using neural network models for BCI applications.

SUBMITTER: Manor R

PROVIDER: S-EPMC5168930 | biostudies-literature | 2016

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Multimodal Neural Network for Rapid Serial Visual Presentation Brain Computer Interface.

Manor Ran R Mishali Liran L Geva Amir B AB

Frontiers in computational neuroscience 20161220

Brain computer interfaces allow users to preform various tasks using only the electrical activity of the brain. BCI applications often present the user a set of stimuli and record the corresponding electrical response. The BCI algorithm will then have to decode the acquired brain response and perform the desired task. In rapid serial visual presentation (RSVP) tasks, the subject is presented with a continuous stream of images containing rare target images among standard images, while the algorit ...[more]

PMID: 28066220

Similar Datasets

Project description:Background and objectives: Many brain-computer interfaces (BCIs) for people with severe disabilities present stimuli in the visual modality with little consideration of the visual skills required for successful use. The primary objective of this tutorial is to present researchers and clinical professionals with basic information about the visual skills needed for functional use of visual BCIs, and to offer modifications that would render BCI technology more accessible for persons with vision impairments.Methods: First, we provide a background on BCIs that rely on a visual interface. We then describe the visual skills required for BCI technologies that are used for augmentative and alternative communication (AAC), as well as common eye conditions or impairments that can impact the user's performance. We summarize screening tools that can be administered by the non-eye care professional in a research or clinical setting, as well as the role of the eye care professional. Finally, we explore potential BCI design modifications to compensate for identified functional impairments. Information was generated from literature review and the clinical experience of vision experts.Results and conclusions: This in-depth description culminates in foundational information about visual skills and functional visual impairments that affect the design and use of visual interfaces for BCI technologies. The visual interface is a critical component of successful BCI systems. We can determine a BCI system for potential users with visual impairments and design BCI visual interfaces based on sound anatomical and physiological visual clinical science.Implications for RehabilitationAs brain-computer interfaces (BCIs) become possible access methods for people with severe motor impairments, it is critical that clinicians have a basic knowledge of the visual skills necessary for use of visual BCI interfaces.Rehabilitation providers must have a knowledge of objectively gathering information regarding a potential BCI user's functional visual skills.Rehabilitation providers must understand how to modify BCI visual interfaces for the potential user with visual impairments.Rehabilitation scientists should understand the visual demands of BCIs as they develop and evaluate these new access methods.

Project description:Visual ERP (P300) based brain-computer interfaces (BCIs) allow for fast and reliable spelling and are intended as a muscle-independent communication channel for people with severe paralysis. However, they require the presentation of visual stimuli in the field of view of the user. A head-mounted display could allow convenient presentation of visual stimuli in situations, where mounting a conventional monitor might be difficult or not feasible (e.g., at a patient's bedside). To explore if similar accuracies can be achieved with a virtual reality (VR) headset compared to a conventional flat screen monitor, we conducted an experiment with 18 healthy participants. We also evaluated it with a person in the locked-in state (LIS) to verify that usage of the headset is possible for a severely paralyzed person. Healthy participants performed online spelling with three different display methods. In one condition a 5 × 5 letter matrix was presented on a conventional 22 inch TFT monitor. Two configurations of the VR headset were tested. In the first (glasses A), the same 5 × 5 matrix filled the field of view of the user. In the second (glasses B), single letters of the matrix filled the field of view of the user. The participant in the LIS tested the VR headset on three different occasions (glasses A condition only). For healthy participants, average online spelling accuracies were 94% (15.5 bits/min) using three flash sequences for spelling with the monitor and glasses A and 96% (16.2 bits/min) with glasses B. In one session, the participant in the LIS reached an online spelling accuracy of 100% (10 bits/min) using the glasses A condition. We also demonstrated that spelling with one flash sequence is possible with the VR headset for healthy users (mean: 32.1 bits/min, maximum reached by one user: 71.89 bits/min at 100% accuracy). We conclude that the VR headset allows for rapid P300 BCI communication in healthy users and may be a suitable display option for severely paralyzed persons.

Dataset Information

Multimodal Neural Network for Rapid Serial Visual Presentation Brain Computer Interface.

Publications

Multimodal Neural Network for Rapid Serial Visual Presentation Brain Computer Interface.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets