Dataset Information

A general method for assessing brain-computer interface performance and its limitations.

ABSTRACT: When researchers evaluate brain-computer interface (BCI) systems, we want quantitative answers to questions such as: How good is the system's performance? How good does it need to be? and: Is it capable of reaching the desired level in future? In response to the current lack of objective, quantitative, study-independent approaches, we introduce methods that help to address such questions. We identified three challenges: (I) the need for efficient measurement techniques that adapt rapidly and reliably to capture a wide range of performance levels; (II) the need to express results in a way that allows comparison between similar but non-identical tasks; (III) the need to measure the extent to which certain components of a BCI system (e.g. the signal processing pipeline) not only support BCI performance, but also potentially restrict the maximum level it can reach.For challenge (I), we developed an automatic staircase method that adjusted task difficulty adaptively along a single abstract axis. For challenge (II), we used the rate of information gain between two Bernoulli distributions: one reflecting the observed success rate, the other reflecting chance performance estimated by a matched random-walk method. This measure includes Wolpaw's information transfer rate as a special case, but addresses the latter's limitations including its restriction to item-selection tasks. To validate our approach and address challenge (III), we compared four healthy subjects' performance using an EEG-based BCI, a 'Direct Controller' (a high-performance hardware input device), and a 'Pseudo-BCI Controller' (the same input device, but with control signals processed by the BCI signal processing pipeline).Our results confirm the repeatability and validity of our measures, and indicate that our BCI signal processing pipeline reduced attainable performance by about 33% (21 bits min(-1)).Our approach provides a flexible basis for evaluating BCI performance and its limitations, across a wide range of tasks and task difficulties.

SUBMITTER: Hill NJ

PROVIDER: S-EPMC4113089 | biostudies-literature | 2014 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A general method for assessing brain-computer interface performance and its limitations.

Hill N Jeremy NJ Häuser Ann-Katrin AK Schalk Gerwin G

Journal of neural engineering 20140324 2

<h4>Objective</h4>When researchers evaluate brain-computer interface (BCI) systems, we want quantitative answers to questions such as: How good is the system's performance? How good does it need to be? and: Is it capable of reaching the desired level in future? In response to the current lack of objective, quantitative, study-independent approaches, we introduce methods that help to address such questions. We identified three challenges: (I) the need for efficient measurement techniques that ada ...[more]

PMID: 24658406

Similar Datasets

Project description:Performance variation is a critical issue in motor imagery brain-computer interface (MI-BCI), and various neurophysiological, psychological, and anatomical correlates have been reported in the literature. Although the main aim of such studies is to predict MI-BCI performance for the prescreening of poor performers, studies which focus on the user's sense of the motor imagery process and directly estimate MI-BCI performance through the user's self-prediction are lacking. In this study, we first test each user's self-prediction idea regarding motor imagery experimental datasets. Fifty-two subjects participated in a classical, two-class motor imagery experiment and were asked to evaluate their easiness with motor imagery and to predict their own MI-BCI performance. During the motor imagery experiment, an electroencephalogram (EEG) was recorded; however, no feedback on motor imagery was given to subjects. From EEG recordings, the offline classification accuracy was estimated and compared with several questionnaire scores of subjects, as well as with each subject's self-prediction of MI-BCI performance. The subjects' performance predictions during motor imagery task showed a high positive correlation (r = 0.64, p < 0.01). Interestingly, it was observed that the self-prediction became more accurate as the subjects conducted more motor imagery tasks in the Correlation coefficient (pre-task to 2nd run: r = 0.02 to r = 0.54, p < 0.01) and root mean square error (pre-task to 3rd run: 17.7% to 10%, p < 0.01). We demonstrated that subjects may accurately predict their MI-BCI performance even without feedback information. This implies that the human brain is an active learning system and, by self-experiencing the endogenous motor imagery process, it can sense and adopt the quality of the process. Thus, it is believed that users may be able to predict MI-BCI performance and results may contribute to a better understanding of low performance and advancing BCI.

Project description:Motor imagery (MI) is an endogenous mental process and is commonly used as an electroencephalogram (EEG)-based brain-computer interface (BCI) strategy. Previous studies of P300 and MI-based (without online feedback) BCI have shown that mental states like fatigue can negatively affect participants' EEG signatures. However, exogenous stimuli cause visual fatigue, which might have a different mechanism than endogenous tasks do. Furthermore, subjects could adjust themselves if online feedback is provided. In this sense, it is still unclear how fatigue affects online MI-based BCI performance. With this question, 12 healthy subjects are recruited to investigate this issue, and an MI-based online BCI experiment is performed for four sessions on different days. The first session is for training, and the other three sessions differ in rest condition and duration-no rest, 16-min eyes-open rest, and 16-min eyes-closed rest-arranged in a pseudo-random order. Multidimensional fatigue inventory (MFI) and short stress state questionnaire (SSSQ) reveal that general fatigue, mental fatigue, and distress have increased, while engagement has decreased significantly within certain sessions. However, the BCI performances, including percent valid correct (PVC) and information transfer rate (ITR), show no significant change across 400 trials. The results suggest that although the repetitive MI task has affected subjects' mental states, their BCI performances and feature separability within a session are not affected by the task significantly. Further electrophysiological analysis reveals that the alpha-band power in the sensorimotor area has an increasing tendency, while event-related desynchronization (ERD) modulation level has a decreasing trend. During the rest time, no physiological difference has been found in the eyes-open rest condition; on the contrary, the alpha-band power increase and subsequent decrease appear in the eyes-closed rest condition. In summary, this experiment shows evidence that mental states can change dramatically in the intensive MI-BCI practice, but BCI performances could be maintained.

Project description:Brain-computer interface (BCI) paradigms are usually tested when environmental and biological artifacts are intentionally avoided. In this study, we deliberately introduced different perturbations in order to test the robustness of a steady state visual evoked potential (SSVEP) based BCI. Specifically we investigated to what extent a drop in performance is related to the degraded quality of EEG signals or rather due to increased cognitive load. In the online tasks, subjects focused on one of the four circles and gave feedback on the correctness of the classification under four conditions randomized across subjects: Control (no perturbation), Speaking (counting loudly and repeatedly from one to ten), Thinking (mentally counting repeatedly from one to ten), and Listening (listening to verbal counting from one to ten). Decision tree, Naïve Bayes and K-Nearest Neighbor classifiers were used to evaluate the classification performance using features generated by canonical correlation analysis. During the online condition, Speaking and Thinking decreased moderately the mean classification accuracy compared to Control condition whereas there was no significant difference between Listening and Control conditions across subjects. The performances were sensitive to the classification method and to the perturbation conditions. We have not observed significant artifacts in EEG during perturbations in the frequency range of interest except in theta band. Therefore we concluded that the drop in the performance is likely to have a cognitive origin. During the Listening condition relative alpha power in a broad area including central and temporal regions primarily over the left hemisphere correlated negatively with the performance thus most likely indicating active suppression of the distracting presentation of the playback. This is the first study that systematically evaluates the effects of natural artifacts (i.e. mental, verbal and audio perturbations) on SSVEP-based BCIs. The results can be used to improve individual classification performance taking into account effects of perturbations.

Dataset Information

A general method for assessing brain-computer interface performance and its limitations.

Publications

A general method for assessing brain-computer interface performance and its limitations.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets