AUC Confidence Bounds for Performance Evaluation of Brain-Computer InterfaceReport as inadecuate

AUC Confidence Bounds for Performance Evaluation of Brain-Computer Interface - Download this document for free, or read online. Document in PDF available to download.

1 SPMC - Signal Processing and Multimedia Communications research group

Abstract : Currently most performance evaluation of Brain- Computer Interface BCI systems is simply reported in terms of accuracy. In this paper we propose a novel approach to evaluate the true performance of BCI systems based on Receiver Operating Characteristic ROC analysis, that removes the limitations of the accuracy performance measure. We demonstrate the need to provide, and particularly for small sample size, Confidence Interval CI bounds to indicate reliability of the BCI system performance. The ROC-based methodology makes it possible to calculate CI, shown as a contour at each any points of the ROC curve, with value of the lower bound of the Area Under the Curve AUC. We illustrate the usefulness of the methodology using the results the BCI Competition IV data set 3, dealing with the classification of wrist movements from four directions recorded using agnetoencephalogram MEG. Plotting the 95% CI contours overlayed on the ROC curves revealed some overlap with the chance level, thus revealing potential different interpretation from claims based on single accuracy value. The ROC-based methodology will also help to determine minimal sample size, an important requirement for future BCI studies and competitions.

keyword : Receiver Operating Characteristic ROC Area Under the Curve AUC Confidence Interval CI Performance Evaluation Brain-Computer Interface BCI

Author: Brahim Hamadicharef -



Related documents