On robust parameter estimation in brain–computer interfacing

Samek, Wojciech,Nakajima, Shinichi,Kawanabe, Motoaki,Mü,ller, Klaus-Robert IOP 2017 Journal of neural engineering Vol.14 No.6

<P> <I>Objective</I>. The reliable estimation of parameters such as mean or covariance matrix from noisy and high-dimensional observations is a prerequisite for successful application of signal processing and machine learning algorithms in brain–computer interfacing (BCI). This challenging task becomes significantly more difficult if the data set contains outliers, e.g. due to subject movements, eye blinks or loose electrodes, as they may heavily bias the estimation and the subsequent statistical analysis. Although various robust estimators have been developed to tackle the outlier problem, they ignore important structural information in the data and thus may not be optimal. Typical structural elements in BCI data are the <I>trials</I> consisting of a few hundred EEG samples and indicating the start and end of a task. <I>Approach</I>. This work discusses the parameter estimation problem in BCI and introduces a novel hierarchical view on robustness which naturally comprises different types of outlierness occurring in structured data. Furthermore, the class of minimum divergence estimators is reviewed and a robust mean and covariance estimator for structured data is derived and evaluated with simulations and on a benchmark data set. <I>Main results</I>. The results show that state-of-the-art BCI algorithms benefit from robustly estimated parameters. <I>Significance</I>. Since parameter estimation is an integral part of various machine learning algorithms, the presented techniques are applicable to many problems beyond BCI.</P>

Evaluating the Visualization of What a Deep Neural Network Has Learned

Samek, Wojciech,Binder, Alexander,Montavon, Gregoire,Lapuschkin, Sebastian,Muller, Klaus-Robert IEEE 2017 IEEE transactions on neural networks and learning Vol.28 No.11

<P>Deep neural networks (DNNs) have demonstrated impressive performance in complex machine learning tasks such as image classification or speech recognition. However, due to their multilayer nonlinear structure, they are not transparent, i.e., it is hard to grasp what makes them arrive at a particular classification or recognition decision, given a new unseen data sample. Recently, several approaches have been proposed enabling one to understand and interpret the reasoning embodied in a DNN for a single test image. These methods quantify the 'importance' of individual pixels with respect to the classification decision and allow a visualization in terms of a heatmap in pixel/input space. While the usefulness of heatmaps can be judged subjectively by a human, an objective quality measure is missing. In this paper, we present a general methodology based on region perturbation for evaluating ordered collections of pixels such as heatmaps. We compare heatmaps computed by three different methods on the SUN397, ILSVRC2012, and MIT Places data sets. Our main result is that the recently proposed layer-wise relevance propagation algorithm qualitatively and quantitatively provides a better explanation of what made a DNN arrive at a particular classification decision than the sensitivity-based approach or the deconvolution method. We provide theoretical arguments to explain this result and discuss its practical implications. Finally, we investigate the use of heatmaps for unsupervised assessment of the neural network performance.</P>

Multiscale temporal neural dynamics predict performance in a complex sensorimotor task

Samek, Wojciech,Blythe, Duncan A.J.,Curio, Gabriel,Mü,ller, Klaus-Robert,Blankertz, Benjamin,Nikulin, Vadim V. Elsevier 2016 NeuroImage Vol.141 No.-

<P><B>Abstract</B></P> <P>Ongoing neuronal oscillations are pivotal in brain functioning and are known to influence subjects' performance. This modulation is usually studied on short time scales whilst multiple time scales are rarely considered. In our study we show that Long-Range Temporal Correlations (LRTCs) estimated from the amplitude of EEG oscillations over a range of time-scales predict performance in a complex sensorimotor task, based on Brain-Computer Interfacing (BCI). Our paradigm involved eighty subjects generating covert motor responses to dynamically changing visual cues and thus controlling a computer program through the modulation of neuronal oscillations. The neuronal dynamics were estimated with multichannel EEG. Our results show that: (a) BCI task accuracy may be predicted on the basis of LRTCs measured during the preceding training session, and (b) this result was not due to signal-to-noise ratio of the ongoing neuronal oscillations. Our results provide direct empirical evidence in addition to previous theoretical work suggesting that scale-free neuronal dynamics are important for optimal brain functioning.</P> <P><B>Highlights</B></P> <P> <UL> <LI> Functional relevance of Long-Range Temporal Correlations (LRTCs) was investigated. </LI> <LI> LRTCs were measured with EEG during complex sensorimotor task. </LI> <LI> Alpha-band LRTCs predicted task performance. </LI> <LI> Power-law neuronal dynamics are likely to be beneficial for brain functioning. </LI> </UL> </P>

Well-prepared middle school teachers: Common ground or subtle divide between practitioners and university faculty in the State of Oregon, United States

Linda Samek,Younghee M. Kim,Jay Casbon,Micki M. Caskey,William L. Greene,P. Maureen Musser 한국교육개발원 2010 KEDI Journal of Educational Policy Vol.7 No.2

This qualitative study followed a survey study that investigated university faculty, classroom teachers, and principals’ perceptions of well-prepared middle school teachers in the state of Oregon in the United States. A qualitative approach allowed the researchers to explore and interpret the participants’ views (Denzin &Lincoln, 1998). In spite of many similarities, a number of differences in emphasis or priority were found among the groups, including views on assessment, curriculum development, and the importance of family and community connections for beginning classroom teachers. This study provides a foundation for deeper analysis and discussion among university faculty and practitioners concerning the "what" of middle school teacher preparation programs.

Methods for interpreting and understanding deep neural networks

Montavon, Gré,goire,Samek, Wojciech,Mü,ller, Klaus-Robert Elsevier 2018 Digital signal processing Vol.73 No.-

<P><B>Abstract</B></P> <P>This paper provides an entry point to the problem of interpreting a deep neural network model and explaining its predictions. It is based on a tutorial given at ICASSP 2017. As a tutorial paper, the set of methods covered here is not exhaustive, but sufficiently representative to discuss a number of questions in interpretability, technical challenges, and possible applications. The second part of the tutorial focuses on the recently proposed layer-wise relevance propagation (LRP) technique, for which we provide theory, recommendations, and tricks, to make most efficient use of it on real data.</P>

Learning From More Than One Data Source: Data Fusion Techniques for Sensorimotor Rhythm-Based Brain–Computer Interfaces

Fazli, Siamac,Dahne, Sven,Samek, Wojciech,Bieszmann, Felix,Muller, Klaus-Robert IEEE 2015 Proceedings of the IEEE Vol.103 No.6

<P>Brain-computer interfaces (BCIs) are successfully used in scientific, therapeutic and other applications. Remaining challenges are among others a low signal-to-noise ratio of neural signals, lack of robustness for decoders in the presence of inter-trial and inter-subject variability, time constraints on the calibration phase and the use of BCIs outside a controlled lab environment. Recent advances in BCI research addressed these issues by novel combinations of complementary analysis as well as recording techniques, so called hybrid BCIs. In this paper, we review a number of data fusion techniques for BCI along with hybrid methods for BCI that have recently emerged. Our focus will be on sensorimotor rhythm-based BCIs. We will give an overview of the three main lines of research in this area, integration of complementary features of neural activation, integration of multiple previous sessions and of multiple subjects, and show how these techniques can be used to enhance modern BCI systems.</P>

Multivariate Machine Learning Methods for Fusing Multimodal Functional Neuroimaging Data

Dahne, Sven,Biessmann, Felix,Samek, Wojciech,Haufe, Stefan,Goltz, Dominique,Gundlach, Christopher,Villringer, Arno,Fazli, Siamac,Muller, Klaus-Robert IEEE 2015 Proceedings of the Institute of Electrical and Ele Vol.103 No.9

<P>Multimodal data are ubiquitous in engineering, communications, robotics, computer vision, or more generally speaking in industry and the sciences. All disciplines have developed their respective sets of analytic tools to fuse the information that is available in all measured modalities. In this paper, we provide a review of classical as well as recent machine learning methods (specifically factor models) for fusing information from functional neuroimaging techniques such as: LFP, EEG, MEG, fNIRS, and fMRI. Early and late fusion scenarios are distinguished, and appropriate factor models for the respective scenarios are presented along with example applications from selected multimodal neuroimaging studies. Further emphasis is given to the interpretability of the resulting model parameters, in particular by highlighting how factor models relate to physical models needed for source localization. The methods we discuss allow for the extraction of information from neural data, which ultimately contributes to 1) better neuroscientific understanding; 2) enhance diagnostic performance; and 3) discover neural signals of interest that correlate maximally with a given cognitive paradigm. While we clearly study the multimodal functional neuroimaging challenge, the discussed machine learning techniques have a wide applicability, i.e., in general data fusion, and may thus be informative to the general interested reader.</P>

Explaining nonlinear classification decisions with deep Taylor decomposition

Montavon, Gré,goire,Lapuschkin, Sebastian,Binder, Alexander,Samek, Wojciech,Mü,ller, Klaus-Robert Elsevier 2017 Pattern recognition Vol.65 No.-

<P><B>Abstract</B></P> <P>Nonlinear methods such as Deep Neural Networks (DNNs) are the gold standard for various challenging machine learning problems such as image recognition. Although these methods perform impressively well, they have a significant disadvantage, the lack of transparency, limiting the interpretability of the solution and thus the scope of application in practice. Especially DNNs act as black boxes due to their multilayer nonlinear structure. In this paper we introduce a novel methodology for interpreting generic multilayer neural networks by decomposing the network classification decision into contributions of its input elements. Although our focus is on image classification, the method is applicable to a broad set of input data, learning tasks and network architectures. Our method called deep Taylor decomposition efficiently utilizes the structure of the network by backpropagating the explanations from the output to the input layer. We evaluate the proposed method empirically on the MNIST and ILSVRC data sets.</P> <P><B>Highlights</B></P> <P> <UL> <LI> A novel method to explain nonlinear classification decisions in terms of input variables is introduced. </LI> <LI> The method is based on Taylor expansions and decomposes the output of a deep neural network in terms of input variables. </LI> <LI> The resulting deep Taylor decomposition can be applied directly to existing neural networks without retraining. </LI> <LI> The method is tested on two large-scale neural networks for image classification: BVLC CaffeNet and GoogleNet. </LI> </UL> </P>

Deep Neural Networks for No-Reference and Full-Reference Image Quality Assessment

Bosse, Sebastian,Maniry, Dominique,Muller, Klaus-Robert,Wiegand, Thomas,Samek, Wojciech IEEE 2018 IEEE TRANSACTIONS ON IMAGE PROCESSING - Vol.27 No.1

<P>We present a deep neural network-based approach to image quality assessment (IQA). The network is trained end-to-end and comprises ten convolutional layers and five pooling layers for feature extraction, and two fully connected layers for regression, which makes it significantly deeper than related IQA models. Unique features of the proposed architecture are that: 1) with slight adaptations it can be used in a no-reference (NR) as well as in a full-reference (FR) IQA setting and 2) it allows for joint learning of local quality and local weights, i.e., relative importance of local quality to the global quality estimate, in an unified framework. Our approach is purely data-driven and does not rely on hand-crafted features or other types of prior domain knowledge about the human visual system or image statistics. We evaluate the proposed approach on the LIVE, CISQ, and TID2013 databases as well as the LIVE In the wild image quality challenge database and show superior performance to state-of-the-art NR and FR IQA methods. Finally, cross-database evaluation shows a high ability to generalize between different databases, indicating a high robustness of the learned features.</P>

Brain–computer interfacing under distraction: an evaluation study

Brandl, Stephanie,Frølich, Laura,Hö,hne, Johannes,Mü,ller, Klaus-Robert,Samek, Wojciech IOP 2016 Journal of neural engineering Vol.13 No.5

<P> <I>Objective.</I> While motor-imagery based brain–computer interfaces (BCIs) have been studied over many years by now, most of these studies have taken place in controlled lab settings. Bringing BCI technology into everyday life is still one of the main challenges in this field of research. <I>Approach.</I> This paper systematically investigates BCI performance under 6 types of distractions that mimic out-of-lab environments. <I>Main results.</I> We report results of 16 participants and show that the performance of the standard common spatial patterns (CSP)?+?regularized linear discriminant analysis classification pipeline drops significantly in this ‘simulated’ out-of-lab setting. We then investigate three methods for improving the performance: (1) artifact removal, (2) ensemble classification, and (3) a 2-step classification approach. While artifact removal does not enhance the BCI performance significantly, both ensemble classification and the 2-step classification combined with CSP significantly improve the performance compared to the standard procedure. <I>Significance.</I> Systematically analyzing out-of-lab scenarios is crucial when bringing BCI into everyday life. Algorithms must be adapted to overcome nonstationary environments in order to tackle real-world challenges.</P>