1 Introduction↩︎

Tracking eye position is a subject of active research due to its multiple applications across different fields such as behavioural science [1], assistive technology [2], or user experience [3]. Eye tracking in combination with conventional research methods, like behavioral measures, can help assess and potentially diagnose neurological diseases such as Autism Spectrum Disorder [4], Obsessive Compulsive Disorder [5], Schizophrenia [6], Parkinson’s [7], and Alzheimer’s disease [8]. Additionally, eye tracking technology can be used to detect states of drowsiness [9], to support communication for locked-in patients [10], and to measure attention in marketing [11]. In the last decade, technological advances have allowed complementing eye-tracking technology with EEG — a non-invasive, minimally restrictive, and, relatively low-cost measure of mesoscale brain dynamics with high temporal resolution. Combining behavioral information gained from eye tracking with the neurophysiological markers provided by EEG enables researchers to study perceptual, attentional or cognitive processes in naturalistic situations [12]. Notably, estimating gaze position using EEG would make available gaze position information in a wide variety of studies that cannot acquire eye tracking data otherwise, either because of the unavailability of ET hardware or the lack of in-house expertise. This, in turn, could accelerate scientific discovery on human behavior and neurological and psychiatric diseases, particularly during free viewing of complex stimuli (i.e. naturalistic paradigms) and in clinical settings, in which the installation of an eye-tracker is impractical (e.g. hospital bed).

Advancing cognitive research at the intersection of brain dynamics and eye movement requires synchronized data from EEG and eye-tracking. Such data contains gaze pattern of eye movements recorded with eye- tracking as well as the neurophysiological markers provided by EEG, allowing researchers to study attention and reaction time [13], or to improve brain-computer interfaces [14]. In the same line, estimating gaze position from EEG signals is typically approached with machine learning and deep learning models [15], [16], which need significant amounts of annotated data for training. However, collecting and annotating simultaneous EEG and eye position data is time-consuming and expensive since it requires equipment and expertise for both EEG acquisition and eye-tracking. Hence, the access to concurrently recorded EEG-ET data is highly restricted, which significantly slows down progress in this field. To help bridging this gap, we release EEGEyeNet, a large dataset of EEG data synchronized with precise eye-tracking recordings. Furthermore, given the multiple benefits that EEG-based eye tracking can bring to different domains, in conjunction with our dataset, we present a benchmark for evaluating gaze estimation from EEG. This benchmark comprises three tasks of increasing difficulty and is conceived as a tool to facilitate comparable and reproducible research on gaze estimation from EEG data. We run extensive experiments on the proposed benchmark to establish baseline performance. In order to foster further research in this field we make all of our code and infrastructure available in the following site: http://www.eegeye.net.

To conclude, our key contributions can be summarized as:

• A dataset of high-density 128-channel EEG data synchronized with video-infrared eye tracking from 356 healthy adults that amounts to a total of more than 47 hours of recording. Along with the raw dataset we provide preprocessed data and code for preprocessing and feature extraction.

• A benchmark for gaze estimation from EEG signals that consists of three evaluation tasks with increasing difficulty. This benchmark is built on a subset of the EEGEyeNet dataset.

• An extensive experimentation that establishes baseline performance on the proposed benchmark.

2 Related Work↩︎

Gaze prediction is an active research topic with applications in human behaviour analysis [17], advertisement [18], and human-computer interaction [19], to name a few. Previous research found evidence suggesting that action selection is facilitated by attention [1]. Furthermore, [20] demonstrated the possibility of performing activity recognition from eye movements. To predict gaze location, some models use saliency maps [21], [22], while others leverage machine learning techniques to estimate gaze position from indirect data: [23] use webcam images, and [24] and [25] use functional magnetic resonance imaging (fMRI). Similarly, [26] reconstructed fixations maps, which can predict eye movement patterns, directly from fMRI data. Interestingly, our recent work indicates that it might be possible to infer gaze direction directly from EEG data [16]. Although to our knowledge no previous work has employed deep learning to estimate eye position from EEG, there are studies demonstrating that combining EEG and ET can improve performance in various tasks, as compared with a single modality. In particular, this has been reported in the vigilance estimation [27], information extraction and sentiment analysis [28], or analysing users’ behaviour when performing a web search [29].

2.0.0.1 Related Datasets.

There exist some openly available datasets that combine EEG and ET data. However, in comparison to EEGEyeNet, they are acquired from a smaller sample of individuals [28], [30] or using a less advanced EEG-ET setup [31], [32]. A prominent example of a multimodal neurophysiological dataset, including EEG and ET data collected from a significant sample of participants (126 individuals), was proposed by [33]. However, it is devised as a resource for assessing information processing in the developing brain and contains data from young participants only. This way, EEGEyeNet dataset is the first large-scale and precisely annotated EEG-ET dataset containing recording from participant across the adult lifespan (18-80 years old). In Table ¿tbl:tab:datasets:comparison? we compare the EEGEyeNet dataset with the cited datasets in terms of the number of participants, their age, and recording/session length.

3 EEGEyeNet Dataset↩︎

In this section, we provide a detailed description of the EEGEyeNet dataset. Together with the raw data, we release two sets of preprocessed data: minimally and maximally preprocessed; as well as the preprocessing code. This way, we give users the freedom to manipulate raw data while easing the experimentation barrier by additionally providing ready-to-use clean data.

3.1 Data Acquisition↩︎

3.1.0.1 Participants.

Data were recorded from 356 healthy adults. The study included 190 female and 166 male participants, of ages between 18 and 80 years. All participants gave their written informed consent before participation in the experiment and received a monetary compensation (the local currency equivalent of 50 US Dollars). The data was collected according to the principles expressed in the Declaration of Helsinki [34].

3.1.0.2 Recording setup.

High-density EEG data was recorded at a sampling rate of 500 Hz, with midline central recording reference, using a 128-channel EEG Geodesic Hydrocel system. The impedance of each electrode was checked prior to each recording session and kept below 40 kOhm. Simultaneously, eye position was recorded with an infrared video-based ET EyeLink 1000 Plus from SR Research at a sampling rate of 500 Hz and an instrument spatial resolution of less than 0.01\(°\) root mean square (RMS) of the distances between successive samples. The ET was calibrated with a 9-point grid before each recording. In a validation step, the ET calibration was repeated until the error between two measurements at any point was less than 0.5\(°\), or the average error for all points was less than 1\(°\). Participants were seated at a distance of 68 cm from a 24-inch monitor with a resolution of \(800 \times 600\) pixels. A stable head position was ensured with a chin rest. The illustration of the recording setup can be seen in Figure 1.

3.2 Preprocessing↩︎

EEG data is often contaminated by artifacts produced by environmental factors, e.g., temperature, air humidity, as well as other sources of electromagnetic noise, such as line noise [35]. These artifacts interact in a complex manner with participant-related artifacts, typically reflecting unwanted physiological signals such as eye movements, eye blinks, muscular noise, heart signals or sweating, which differ from participant to participant. The resulting artifacts in the EEG data are typically more prominent than the signal of interest (i.e. brain activity). Therefore, EEG data requires preprocessing in the form of artifact cleaning or artifact correction [36]. We preprocessed the entire EEGEyeNet dataset using the openly available toolbox from [37] in two ways: minimally and maximally. Minimal preprocessing includes the detection and interpolation of bad electrodes, and filtering the data with 40 Hz high-pass filter and 0.5 Hz low-pass filter (cf. Appendix 8). The difference between these two types of preprocessing is that maximal preprocessing removes a much larger number of artifacts (muscles, heart, eyes, line noise, channel noise). To do this, independent component analysis (ICA) is applied in combination with IClabel [38], a pre-trained classifier that estimates the probability of a component reflecting artifactual activity. If a component receives a probability estimation larger than \(0.8\) for any class of artifact we remove it from the data. Minimally preprocessed data includes ocular artifacts, which is expected to make the estimation of gaze position easier. On the other hand, maximal preprocessing is a state-of-the-art technique for neuroscientific applications [37] that aims to keep only neurophysiological information in the data.

After preprocessing (both minimally and maximally), the EEG and eye-tracking data were synchronized using “EYE EEG” [39] to enable EEG analyses time-locked to the onsets of relevant events depending on the experimental paradigm. Synchronization quality was ensured by comparing the trigger latencies recorded in the EEG and eye-tracker data. All synchronization errors did not exceed 2 ms.

3.3 Data Annotation↩︎

Existing literature studying eye movement generally distinguishes between three different events [40]: saccades, fixations, and blinks. Saccades are rapid, ballistic eye movements that instantly change the gaze position. Fixations are defined as time periods without saccades, and blinks are considered a special case of fixation, where the pupil diameter is zero. For each of the experimental paradigms described in Section 3.4, we provide annotations in the form of start and end time of each event, as well as the start and end position of saccades and the average position of fixations. See Appendix 9 for further details.

3.4 Experimental Paradigms↩︎

3.4.0.1 Pro- and Antisaccade.

The pro- and antisaccade paradigm was designed according to the internationally standardized protocol for antisaccade testing developed by [41] and is described in detail in [42]. Each trial starts with a central fixation square. The participants are asked to focus on the center of the screen for a randomized time-period between 1 and 3.5 seconds. Subsequently, the cue (i.e. a dot) appears horizontally on the left or the right hand-side of the central fixation square. In the prosaccade trials, the participants are asked to focus their gaze on the cue as fast as possible, while in the antisaccade trials the participants are instructed to perform a saccade towards the opposite side of the cue. In both cases, the cue is shown for one second. As soon as the cue disappears, the participants shift their focus back to the center of the screen; this is illustrated in Figure 2. Data recorded following this paradigm may be used for different research purposes, such as estimating gaze direction or examining responses to inhibition.

3.4.0.2 Large Grid.

Participants are asked to fixate on a series of dots that are sequentially presented, each at one of 25 different screen positions. Unlike the others, the dot at the center of the screen appears three times, resulting in 27 trials (displayed dots) per block, each dot is displayed for \(1.5\) to \(1.8\) seconds. The positions of the dots were selected to ensure coverage of all corners of the screen as well as the center (see Figure 3). The shape of the grid and its use for eye gaze estimation follows the work from [24]. Given that [24] used the Large Grid paradigm for functional Magnetic Resonance Imaging (fMRI), we have adapted the length of the stimulus and the number of repetitions to our setup. To record a larger number of trials and reduce the predictability of the subsequent positions in the primary sequence of the stimulus, we use different pseudo-randomized orderings of the dots presentation, distributed in five experimental blocks, as shown in Figure 3. The entire procedure is repeated 6 times during the measurement, resulting in 810 stimuli for each participant.

3.4.0.3 Visual Symbol Search.

Visual Symbol Search (VSS) is a computerized version of a clinical assessment to measure processing speed (Symbol Search Subtest of the Wechsler Intelligence Scale for Children IV (WISC-IV) [43]) and the Wechsler Adult Intelligence Scale (WAIS-III)  [44]). Participants are shown 15 rows at a time, where each row consists of two target symbols, five search symbols and two additional symbols that contain respectively the words “YES” and “NO”. For each row, participants need to indicate by clicking with the mouse button on the “YES” or “NO” symbol, whether or not one of the two target symbols appears among the five search symbols. Each recording of the VSS paradigm takes 120 seconds with a maximum of 60 trials, where one trial corresponds to one row; in \(50\%\) of the trials one of the target symbols does appear in the search symbols and in the remaining \(50\%\) none does. Once participants finish a set of 15 rows, they press a “next page” button which displays a new set of 15 rows. Participants are instructed to solve as many rows, or trials, as possible within the given 120 seconds. Before beginning the actual recording, participants perform a training of four trials, for which they receive feedback to ensure they understand the task. No feedback is provided throughout the actual recording. Data collected according to this paradigm may be used for investigating behavioral and neurophysiological correlates of processing speed.

The EEGEyeNet dataset contains data recorded following all three different experimental paradigms mentioned above. These paradigms cover typical cognitive tasks and provide a wealth of eye movement patterns.

Finally, in Table ¿tbl:tab:summary? we report for each paradigm the number of appearances of the three events that we extract: fixations, saccades and blinks; we report these after both minimal (min) and maximal (max) preprocessing. In Appendix 10 we give further details about the characteristics of each of these events. These numbers make apparent the large size of the EEGEyeNet dataset, with a total of over 47 hours of recorded events.

4 Benchmark↩︎

Based on the EEGEyeNet dataset we propose a benchmark to assess EEG-based eye tracking methods. This benchmark consists of three different tasks with increasing difficulty. For each of the tasks, we provide data preparation modules that cut the data into samples of one second with a temporal resolution of \(2 ms\) from all 128 EEG channels; This way all samples have a shape of \(500 \times 128\), i.e., \(500\) time points for each of the \(128\) EEG channels. This is also illustrated in Figure 5. We use exclusively minimally preprocessed EEG data as it produces better performance as compared to maximal preprocessed data. In Appendix 12, we report our results for the maximally preprocessed data, as well as the results of the experiments ran on the Zuco 2.0 dataset [30]. For comparability, we also provide a stable split of the data across participants, with 70% of the participants for training, 15% for validation and 15% for test. Note that each participant’s data is contained only in one of the three sets, i.e., the same participant does not appear during training and validation or test. In Table ¿tbl:tab:benStats? we include a summary of how the data is split in each task of the benchmark.

4.1 Task 1: Left-right↩︎

The first and simplest task of our benchmark consists of determining the direction of the subject’s gaze along the horizontal axis. This task is performed exclusively on the first experimental paradigm (Pro- and Antisaccade). We make sure that in all the samples provided for this task, after the cue, the participant performs a saccade towards the correct direction and then fixates on the cue; otherwise, the sample is not included in the data. Given that the antisaccade task has been viewed as an indicator of behavioral inhibition abilities [45] and that here we are interested in saccade direction, for the sake of simplicity, we use only data from the prosaccade trials of the antisaccade experimental paradigm. From the \(30,842\) samples used in this task, \(14,827\) correspond to the “left” label and \(16,015\) to “right”.

Each sample starts at the instant when a cue is shown on the screen and contains one saccade: the goal is to determine whether it is a left or a right saccade. This is a binary classification task, and therefore, performance is measured in terms of accuracy with a random baseline of \(52.3\%\), given by the majority class. Although seemingly simple, determining the horizontal direction of the human gaze is of paramount importance in multiple real-world applications, e.g., gaze-based writing systems for people with disabilities [46].

4.2 Task 2: Angle/Amplitude↩︎

The second task of our benchmark consists of determining the angle and amplitude of a saccade and is performed on data from the Large Grid paradigm. The one-second samples in this task contain a saccade onset in the middle of the sample. This way, each sample contains a complete saccade as well as a part of the preceding and succeeding fixations. During the data preparation for this task, we remove samples with fixations that are shorter than \(150ms\) and with saccades with an amplitude of less than \(1°\) [47].

Given a sample, the task of the model is to regress the two target values, i.e., angle and amplitude of the relative change of the gaze position during the saccade. The evaluation metric for this task is the square root of the mean squared error (RMSE) for the angle (in radians along the shortest path of the unit circle) and amplitude (in millimeters) separately. The naive baseline is given by the mean angle and amplitude in the training set and amounts to \(1.90\) RMSE radians for the angle and \(74.7\) RMSE mm for the amplitude. This task is significantly harder than the Left-Right task and aims at serving as an intermediate step for the development of a purely EEG-based ET. Despite its difficulty, there is evidence that EEG recordings contain angle information [48].

4.3 Task 3: Absolute Position↩︎

Finally, we propose the task of determining the absolute position of the subject’s gaze in the screen, described in terms of XY-coordinates. Again, this task is performed on the Large Grid paradigm. We provide samples of one second during which the participant is performing only one fixation. The data preparation module ensures that in this time window there is no other event happening, i.e., the participant performs only a fixation. However, we note that in order to estimate the current gaze position we expect past information, e.g., the previous gaze position, to be helpful. For this reason, we also encourage experimentation on different ways of processing and cutting the full EEG recordings provided in the dataset.

Performance is measured as the euclidean distance in millimeters between the actual and the estimated gaze position in the XY-coordinate plane. Random performance is again calculated as the mean position across the training set and corresponds to a distance of \(123.3\) mm. This is the hardest task in the proposed benchmark and aims at simulating a purely EEG-based eye-tracker. We expect this task to help in the development of general methods to further improve gaze estimation systems as well as in setting an upper bound on the reach of EEG-based eye-tracking.

5 Baselines↩︎

We run extensive experiments on the proposed benchmark in order to provide baselines of different complexity. In our repository we provide an intuitive interface to reproduce our results and to use the methods presented here as a starting point for further research. We consider both models based on classical machine learning as well as large neural networks.

5.1 Models↩︎

5.1.0.1 Machine Learning.

These models operate on features extracted from the data rather than on the raw data. Therefore, in a feature extraction step, we apply a band-pass filter in the alpha band [\(8-13\) Hz] on the continuous EEG signals across the entire trial. The choice of the alpha band is motivated by growing evidence suggesting that alpha activity is integral to spatial attention, and therefore plays a central role in covert orienting in a range of paradigms [49]. After band-passing the signal, the Hilbert transform was applied, resulting in a complex time series from which we extract phase and amplitude. Using the resulting features, we experiment with different models. Left-Right is a classification task while Angle/Amplitude and Absolute Position are regression tasks and thus, some of the considered models can be applied only to either of those two types of tasks. In particular, the classification-only models that we study are Gaussian Naive Bayes (NB) Linear Support Vector Classification (SVC) and Radial Basis Function (RBF) kernel SVC, whereas the regression-only models are Linear, Ridge and Lasso Regression as well as Elastic Net and RBF Support Vector Machine for regression (SVR). Furthermore, we use K-Nearest Neighbours (KNN) and four tree-based models, Random Forest, Gradient Boost, AdaBoost and XGBoost, which can be used for both, classification and regression. In all cases we use the implementation from the Sklearn library [50], for detailed model hyperparameters see Appendix 11.

5.1.0.2 Deep Learning.

We evaluate five state-of-the-art architectures for time series on the proposed benchmark: a standard one-dimensional convolutional neural network (CNN), a CNN with pyramidal shape, the EEGNet model by [51], an InceptionTime model [52], and an Xception model [53]. All of these models use convolutions as the primary operation (see Appendix 11 for architectural details). We tune the learning rate and other hyperparameters on the validation set of the Left-Right task and use those values in all the reported results (cf. Appendix 11). We use binary cross entropy loss to train the models for Left-Right, and mean square error (MSE) loss for the other two tasks; in all cases we use the Adam optimizer [54] and early stopping on the validation sets.

5.2 Results↩︎

In Table ¿tbl:tab:results? we provide the results of our evaluation for each of the models considered and for each of the three tasks. We tune the learning rate of the different deep learning models on the validation set of the Left-Right task and use the same rate in the other two tasks, i.e., \(1e^{-4}\). We run 5 times each experiment and report mean performance and standard deviation.

5.2.0.1 Left-Right.

We see that classical machine learning models achieve high performance in this task, much above the naive baseline of \(52.26\%\). In particular, tree-based models reach a performance of over \(96\%\), which confirms Left-Right as the easiest task in the proposed benchmark. Notably, although classical models obtain high scores, all the deep learning models reach a performance consistently higher, with an accuracy of over \(98\%\) in all cases (except for InceptionTime, \(97.9\%)\). This shows that despite the high performance, differences in performance can still be noticeable.

5.2.0.2 Angle/Amplitude.

The results in Table ¿tbl:tab:results? clearly show that this task is harder than Left-Right. Except for RBF SVC/SVR, which performs close to random, all classical machine learning models reach a similar performance for the estimation of both amplitude and angle. This result is above the naive baseline although not by a big margin. Tree-based models perform the best among classical statistical models, however, they are clearly inferior to deep learning models in this task. Somewhat surprisingly, among the deep learning models the simplest ones, i.e., CNN and PyramidalCNN perform the best, with an RMSE of \(0.33\) and \(0.34\) radians in angle estimation, and \(32\) and \(30.7\) mm in amplitude estimation, respectively. Overall, we see that there is still a considerable gap between the best results reported here and ideal performance. Our results constitute a baseline that should orient future work on estimating angle and amplitude of saccades from EEG data.

5.2.0.3 Absolute Position.

Finally, we see in the last column of Table ¿tbl:tab:results? the results of our experimentation on absolute position estimation. We see that classical models generally fail in this task, with performances very close to the naive baseline. On the other hand, deep learning models reach an euclidean distance with respect to the true location in the range of \(70\) to \(80\) mm. Although these results are far from ideal performance, the considerable gap with the naive baseline shows that EEG-based eye tracking can potentially be achieved to an acceptable degree of accuracy. Our results, set the baseline for this task in \(70.2\) mm, as reached by CNN, again one of the simpler deep learning models. However, as explained in Section 4.3, exploiting previous information is likely to improve performance, and thus, it would be interesting to see how sequence models, such as Recurrent Neural Networks or Transformers would perform in this task. There is a lot of room for improvement and we hope that this task will help future work in advancing EEG-based eye tracking.

In summary, our evaluation reveals that deep learning models are superior to other statistical techniques in estimating gaze position from EEG data. Although this is not surprising, given the complexity of the task and the larger expressive capacity of neural networks, it confirms that EEGEyeNet is a valuable resource for developing large neural models. We expect that future work will surpass our scores advancing EEG-based eye tracking.

6 Discussion and Future Work↩︎

The dataset and benchmark presented in this work provide an approachable framework to conduct research in the intersection of brain activity and eye movement. We will actively maintain and continuously extend both the dataset and the benchmark with further measurements from more participants and new experimental paradigms. We acknowledge as current limitations the relatively small number of participants recorded for the Large Grid paradigm and the fact that the test sets of our benchmark are publicly available. We will address both points by (1) recording more data and (2) building an automatic evaluation service that keeps the test set hidden from the users and includes a leaderboard.

To facilitate extensive use of EEGEyeNet for various research purposes, we provide in our repository data preparation tools with an easy-to-use interface where users can define their own benchmarking tasks or extract other information from the dataset. In particular, the user can specify whether some recording blocks from the experimental paradigm should be ignored, which events to extract from the data and how the data should be preprocessed. Additionally, the user can also decide whether feature extraction should be performed or not. As with the other resources presented in this work, the data preparation module is in continuous development. We plan to adapt this software tool in the future according to the users’ needs. Overall, we expect that EEGEyeNet will become a central resource for a broad range of EEG and Eye-Tracking related research, specifically:

6.0.0.1 Research in Cognitive Area.

EEGEyeNet’s rich structure and high-density coverage of EEG and Eye-Tracking data may help advance other areas that study the combination of gaze position and brain activity to identify variations in attention, arousal and participant’s compliance with the task demands. Moreover, the behavioral information gained from eye tracking with the high temporal resolution and neurophysiological markers provided by EEG enables research of the perceptual, attentional, cognitive processes.

6.0.0.2 Benchmarking.

We expect the high quality, diversity and large scale of the EEGEyeNet dataset to be leveraged, in order to include new tasks in the proposed benchmark, as well as to build benchmarks for related domains. In particular, we plan to include segmentation tasks that evaluate the ability of a given model to detect and distinguish events such as fixations or saccades.

7 Conclusion↩︎

Recording eye-tracking data is a complex and expensive process that requires specialized hardware, trained operators and participants’ consent. Collecting such data in combination with EEG adds an additional level of complexity to the data acquisition process. Therefore, behavioral research studying the combination of brain activity and eye movements is typically restricted by the lack of appropriate data. In this work, we have introduced EEGEyeNet, a large dataset of EEG and eye tracking data, with the view of making basic cognitive neuroscience research more approachable. Furthermore, given the potential benefits that EEG-based eye-tracking can bring in different domains, we have proposed a benchmark to facilitate the development of new methods tackling this challenge. Our experiments on this benchmark show that deep learning models perform better than classical statistical models. This confirms that the amount of data contained in EEGEyeNet is large enough to reliably train large neural models, which is a promising direction for further developing EEG-based eye tracking.

Author statement↩︎

Hereby we confirm that we bear all responsibility in case of violation of rights, etc., and confirmation of the data license. All data are de-identified and participants gave permission for their data to be openly shared as part of the informed consent process.

Public data are distributed under the the Creative Commons Attribution 4.0 International Public License (https://creativecommons.org/licenses/by/4.0/).

We release our code used for our experiments, data collection and data preparation at the following repository : https://github.com/ardkastrati/EEGEyeNet and our dataset at https://doi.org/10.17605/OSF.IO/KTV7M.

Acknowledgments↩︎

We thank Tzvetan Popov for useful discussions related to this project and Marta Marciniak for the language editing, and proofreading. Dataset collection and maintenance is generously supported by the Velux Foundation and Swiss National Science Foundation.

We thank the participants for taking the time to be in the study, as well as their willingness to have their data shared with the scientific community.

Supplementary Material↩︎

8 Preprocessing↩︎

8.0.0.1 EEG Minimal Preprocessing

In this preprocessing strategy first, we identify bad channels using the EEGLAB [55] clean rawdata algorithm ². This algorithm consists of three steps: (1) it detects channels that have no signal variation for more than 5 seconds and filters the EEG of the remaining channels with a forward-backwards (non-causal) filter, here we use a high-pass filter with a transition band of [\(0.25\), \(0.75\)] Hz; (2), it searches for channels with a lower correlation to its robust estimate than a given threshold that we set to \(0.85\); and (3) it removes EEG channels with excessive line noise that in our case corresponds to \(4\) standard deviations. Subsequently, the high Variance Criterion (HVC) was applied, with a pre- specified threshold of \(100 \mu V\) as the upper limit for identifying bad channels. Later, the EEG data were band-pass filtered at \(0.5\) and \(40\) Hz and then, Zapline toolkit [56] was applied to remove line noise artifacts, discarding \(7\) power line components. Finally, all the channels marked as bad were interpolated using spherical spline interpolation.

8.0.0.2 EEG Maximal Preprocessing.

In addition to the whole minimal preprocessing pipeline, in the Maximal Preprocessing Pipeline, the Independent component analysis (ICA) is used to isolate the various source generator processes underlying those recordings. Non-Brain artifactual source components are removed based on the automatic classification result as provided by Independent Component Label (ICLabel) [38]. ICA included Optimizing the ICA-based removal of ocular EEG artifacts from free viewing experiments (Dimigen 2020) as implemented in [37].

8.0.0.3 Objective Quality Classification

After the bad channel interpolation, all EEG datasets were rated with the objective quality criteria of the Automagic Toolbox [37]. Any data file rated as bad, meaning that the proportion of high-amplitude data points in the signal (> \(30 \mu V\)) was larger than 0.2, more than 20% of the time points showed a variance larger than \(15 \mu V\) across channels, 40% of the channels showed high variance (\(15 \mu V\)), or the ratio of bad channels was higher than 0.4 was not included in the further data processing.

9 Annotations↩︎

Saccades are detected by the velocity and acceleration of the eye movements. We used the default system (SR Research, http://www.sr-research.com/) parameters to define saccades: an acceleration threshold of 8000° per second, a velocity threshold of 30° per second, and a deflection threshold of 0.1°.

Fixations were defined as time periods without saccades. Fixation might include small saccades (i.e., microsaccades), which fall below the threshold for saccade detection. Furthermore, a blink was be regarded as a special case of a fixation, where the pupil diameter is either zero or outside a dynamically computed valid pupil, or the horizontal and vertical gaze positions are zero.

We extracted the following information about the saccades, fixations and blinks: start and end time, duration, coordinates of start positions on the computer screen in pixels, and for saccades additionally: end positions, and amplitudes.

10 Events Characteristics↩︎

In this appendix we show further details of each dataset. More precisely, for each experimental paradigm we present the distribution of the fixation duration, the distribution of the fixation positions, the distribution of the saccade amplitude (in pixels, where 1 pixel corresponds to 0.5 mm), and the distribution of the saccade angle.

11 Details of the Models↩︎

In this appendix we present the details of our models used during our experimentations.

11.1 Deep Learning Models↩︎

In the following we present five different deep learning architectures that we evaluate on the EEGEyeNet benchmark.

11.1.0.1 EEGNet.

[51] is a convolutional neural network tailored to Brain-Computer Interfaces applications. This model first performs a temporal convolution to learn frequency filters, then a depthwise convolution to learn frequency-specific spatial filters followed by a separable convolution which should learn a temporal summary for each individual feature map. Finally, a pointwise convolution is applied to mix the feature maps. For details on this architecture we refer the reader to the original work [51], which we follow in our implementation.

11.1.0.2 Convolutional Neural Network (CNN).

We experiment with a standard one-dimensional convolutional neural network with \(12\) layers and additive residual connections around blocks of three layers. Each layer consists of (1D-)convolution, batch normalization [57], ReLU activation and max pooling. In the convolutions we use \(16\) filters of size size \(64\), and for the pooling operation a kernel of size \(2\) and stride \(1\). Each residual connection performs a convolution followed by batch normalization.

11.1.0.3 Pyramidal CNN.

We consider a CNN with an inverted pyramidal shape and \(6\) layers. Each layer consists of the same modules as the CNN, with the difference that the number of filters is a multiple of \(16\) that grows with depth, i.e., \(16\) in the first layer, \(32\) in the second and so on. In this case, the kernel size is \(16\) and there are not residual connections.

11.1.0.4 InceptionTime

We re-implement the model from [58], which is an adaptation of the Inception-v4 architecture [59] to time series classification. We instantiate this model with \(12\) layers and skip connections every three layers. Each layer consists of an InceptionTime module, which takes as input \(64\) channels and performs a \(1\times 1\) bottleneck convolution that produces a feature map with \(16\) channels. Then, this feature map is passed through three different convolution operators with \(16\) filters each and kernel sizes of \(64\), \(32\), \(16\); and a max-pooling operator with a kernel size of \(3\). The four resulting feature maps are concatenated to form the layer’s output, which has \(64\) channels.This model also uses residual connections around each block of three layers.

11.1.0.5 Xception

Finally, we build a model based on the Xception architecture proposed by [60]. This model follows the same general structure as our CNN, with \(12\) layers and residual connections after three layers. Each layer contains a 1D depthwise separable convolution [60], with \(64\) filters and kernel size of \(40\), followed by batch normalization and ReLU activation.

11.2 Hyperparameter Tuning↩︎

In Table ¿tbl:tab:hyperparameters:classical? we report our tuned hyperparameters for the classical machine learning models and in Table ¿tbl:tab:hyperparameters:deep-learning? for the deep learning models.

12 Further Experiments↩︎

12.1 Experiments on the Maximally Preprocessed Data↩︎

Besides the minimally preprocessed data presented in the benchmark results at Section 4, we performed the same experiments on the maximally preprocessed data. As expected, the models perform much worse compared to the ones with minimally preprocessed data (see Table ¿tbl:tab:results:maximal?). However, the following results show that even after removing eye artifacts during preprocessing, it is still possible to infer gaze direction based on EEG data.

12.1.0.1 Left-Right.

We see that classical machine learning models achieve a better performance than the naive baseline of \(52.26\%\). In particular, tree-based models (RandomForest, GradientBoosting, AdaBoost and XGBoost) reach a performance of over \(68\%\). Although it is not a high performance, it shows that the eye movement information between left and right can also be inferred from “pure” brain activity. Furthermore, the deep learning models achieve an even higher performance of over \(86\%\), showing that one can achieve satisfactory results for this task also from the maximally preprocessed data.

12.1.0.2 Angle/Amplitude.

As in the case for minimally preprocessed data (see Table ¿tbl:tab:results?), the results in Table ¿tbl:tab:results:maximal? show that this task is more demanding than Left-Right. All classical models perform close to random. The deep learning models perform slightly better, and, interestingly, the simple CNN architecture performs best. This result is above the naive baseline but not by a big margin. Nonetheless, it shows that information about the saccade angle can be extracted from maximally preprocessed data as well. In comparison to the angle, we see that the amplitude task is strictly more difficult for maximally preprocessed data in the third column. All models except EEGNet fail at this task, with EEGNet performing slightly better than the naive baseline. This leaves the open question to what extent the angle and amplitude of saccades can be estimated from maximally preprocessed EEG data.

12.1.0.3 Absolute Position.

In the last column of Table ¿tbl:tab:results:maximal? we see that all models fail in this task, with performances very close to the naive baseline. Deep learning models (except EEGNet) perform even worse than the naive baseline. At this point, it is not clear whether the absolute position of fixations can be inferred from the maximally preprocessed EEG data.

12.2 Experiments on ZuCo 2.0 Dataset↩︎

For comparison, we performed the same experiments on one of these openly available datasets, namely the ZuCo 2.0. We have used the same pipeline from our infrastructure. First, we have synchronized the EEG-ET data and extracted the relevant events for annotation (fixation and saccades). Then we also performed feature extraction. Afterwards, we used our data preparation tool to extract the samples for our benchmarking tasks. Finally, we ran our benchmark with this dataset. Note that we didn’t use the LR-task for this dataset due to the nature of the ZuCo 2.0 experimental paradigm. Although during reading, most of the eye movements are towards the right, the dataset contains eye movements in all other directions (see Figure 9). Other referenced datasets don’t provide an interface for benchmarking purposes. Again, this is because there are differences in the recording setup, protocol and paradigms.

As expected, the models perform worse compared to the ones with minimally preprocessed data (see Table ¿tbl:tab:results?).

12.2.0.1 Angle/Amplitude.

We can see a clear difference between the results in our dataset (see Table ¿tbl:tab:results?) and the results obtained in ZuCo 2.0 dataset. The models performs worse in the ZuCo 2.0 dataset. For the angle task, the best model is the EEGNet with RMSE 1.35 radians. It performs better then the naive baseline (RMSE 1.90 radians), but not by a big margin. In comparison, in our dataset, EEGNet performs much better with an RMSE of 0.70 radians. In addition, the best performing model for our dataset is the simple CNN, which has an RMSE of 0.33 radians. We can also see a clear difference for the amplitude task as well. The best performing model for ZuCo 2.0 dataset is PyramidalCNN with an RMSE of \(61.9\) mm, while in our dataset PyramidalCNN has a better performance with an RMSE of \(30.7\) mm.

12.2.0.2 Absolute Position.

In this task, the results are not directly comparable to our dataset, since they depend on the distribution of the fixation positions. The difference of the distributions can be seen in Figure 7 and Figure 9. Furthermore, in Table ¿tbl:tab:results? we can see that the naive baseline for our dataset is 123.3, whereas for the ZuCo 2.0 dataset (Table ¿tbl:tab:results:zuco?), it is 72. In other words, in ZuCo 2.0 simply predicting the mean position leads to a better performance compared to our dataset, because the fixations are not scattered across all possible positions on the screen. In contrast, our dataset is specifically tailored for eye movement prediction. However, we can still see a clear difference in the performance. In our dataset, there is a considerable gap between the deep learning models and the naive baseline, whereas in the ZuCo dataset all models perform close to the baseline. One possible explanation for this (besides the size and the type of dataset) is that ZuCo dataset uses a different preprocessing method, whereas our dataset uses minimally preprocessing data which is more suitable for eye movement prediction.

13 Datasheets for Datasets↩︎

For the dataset documentation we used the recommended documentation framework "Datasheets for Datasets" [61].

13.1 Motivation↩︎

13.1.0.1 For what purpose was the dataset created?

The primary purpose of the project is to advance research that studies the combination of brain activities and gaze position.

13.1.0.2 Who created the dataset

The dataset was collected by the Methods of Plasticity Research Lab at the University of Zurich. All persons participating in the data collection are acknowledged on our website: www.eegeye.net

13.1.0.3 Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number

This work was supported by the Velux Stiftung Project No. 1126 and by the Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (SNF) Grant \(100014_175875\).

13.1.0.4 Any other comments?

13.2 Composition↩︎

13.2.0.1 What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)?

the EEGEyeNet dataset. contains electroencephalography and eye-tracking recordings following three different experimental paradigms. Together with the raw data, we release two sets of preprocessed data: minimally and maximally preprocessed; as well as the preprocessing code.

13.2.0.2 How many instances are there in total (of each type, if appropriate)?

The number of instances is presented in the Table ¿tbl:tab:benStats? (Benchmark statistic).

13.2.0.3 Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?

The dataset used in this project was recorded in our laboratory in the context of a larger project that aims to quantify age effects on eye movement behaviour and electroencephalography (EEG) recordings of resting-state and task-related activity. Therefore, for the EEGEyeNet dataset, we have used three paradigms that can help in the gaze position prediction.

13.2.0.4 What data does each instance consist of? “Raw” data (e.g., unprocessed text or images) or features?

For each experimental paradigm, as described in the section 3.4, we release raw data, two sets of preprocessed data: minimally and maximally preprocessed; as well as the preprocessing code.

13.2.0.5 Is there a label or target associated with each instance? If so, please provide a description

.

13.2.0.6 Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text.

13.2.0.7 Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)?

13.2.0.8 Are there recommended data splits (e.g., training, development/validation, testing)?

.

13.2.0.9 Are there any errors, sources of noise, or redundancies in the dataset?

Raw data is often contaminated by artifacts influenced by technical and environmental factors and the recorded participant’s specific.

13.2.0.10 Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)?

The dataset is self contained, there are no restrictions associated with any of the external resources that might apply to a future user.

13.2.0.11 Does the dataset contain data that might be considered confidential

.

13.2.0.12 Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?

.

13.2.0.13 Does the dataset relate to people?

13.2.0.14 Does the dataset identify any subpopulations (e.g., by age, gender)?

.

13.2.0.15 Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset?

.

13.2.0.16 Does the dataset contain data that might be considered sensitive in any way

13.2.0.17 Any other comments?

13.3 Collection Process↩︎

13.3.0.1 How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)?

We recorded ET and EEG brain activity, all procedures and mechanims are described in the Section 3.

13.3.0.2 What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated?

All procedures and mechanims are described in the Section 3.

13.3.0.3 If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?

13.3.0.4 Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?

Research Assistants and PhD Students were involved in the data collection process. All obtained fixed salary for Research Assistants/PhD Students.

13.3.0.5 Over what timeframe was the data collected?

The collection of data used in the project began in the first quarter of 2018 and lasted until May 15, 2021.

13.3.0.6 Were any ethical review processes conducted (e.g., by an institutional review board)?

This study was conducted according to the principles expressed in the Declaration of Helsinki. The study was approved by the Institutional Review Board of Canton Zurich (BASEC-Nr. 2017-00226).

13.3.0.7 Does the dataset relate to people?

13.3.0.8 Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)?

We collected the data from the individuals in question directly.

13.3.0.9 Were the individuals in question notified about the data collection?

13.3.0.10 Did the individuals in question consent to the collection and use of their data?

13.3.0.11 If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses?

13.3.0.12 Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis)been conducted?

13.3.0.13 Any other comments?

13.4 Preprocessing/cleaning/labeling↩︎

13.4.0.1 Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)?

remainder of the questions in this section.

13.4.0.2 Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)

13.4.0.3 Is the software used to preprocess/clean/label the instances available? If so, please provide a link or other access point.

13.4.0.4 Any other comments?

13.5 Uses↩︎

13.5.0.1 Has the dataset been used for any tasks already?

.

13.5.0.2 Is there a repository that links to any or all papers or systems that use the dataset?

13.5.0.3 What (other) tasks could the dataset be used for?

The dataset includes experimental paradigms assessing key cognitive functions, like inhibitory control and processing speed. Additionally, the dataset can be used for a development of new segmentation algorithms for fixations, saccades, and blinks.

13.5.0.4 Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?

13.5.0.5 Are there tasks for which the dataset should not be used?

13.5.0.6 Any other comments?

13.6 Distribution↩︎

13.6.0.1 Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created?

13.6.0.2 How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)?

The dataset will be made available along with the manuscript on our website. Raw and preprocessed EEG and eye-tracking data are available online and have the digital object identifier: DOI 10.17605/OSF.IO/KTV7M

13.6.0.3 When will the dataset be distributed?

The dataset will be distributed along with the manuscript submission.

13.6.0.4 Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?

The dataset will distributed under the terms of the Creative Commons CC BY license, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

13.6.0.5 Have any third parties imposed IP-based or other restrictions on the data associated with the instances?

13.6.0.6 Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?

13.6.0.7 Any other comments

13.7 Maintenance↩︎

13.7.0.1 Who is supporting/hosting/maintaining the dataset?

The dataset will be maintanted by the EEGEyeNet team.

13.7.0.2 How can the owner/curator/manager of the dataset be contacted (e.g., email address)?

Email addres: akastrati@ethz.ch, martyna.plomecka@uzh.ch

13.7.0.3 Is there an erratum? If so, please provide a link or other access point

13.7.0.4 Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)?

13.7.0.5 If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a fixed period of time and then deleted)?

.

13.7.0.6 Will older versions of the dataset continue to be supported/hosted/maintained?

13.7.0.7 If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?

13.7.0.8 Any other comments?

14 Risk categorisation↩︎

Our study was classified as "Risk category A" by the Institutional Review Board of Canton Zurich (BASEC-Nr. 2017-00226). Applied methods of EEG and eye-tracking are non-invasive and do not pose risk to participants. The involved tasks do not involve deception and completion of these do not pose any harm to participants.

15 Inclusion and Exclusion criteria↩︎

Inclusion criteria for participation in the study were left and right-handedness, healthy male and female participants, with age between 20 and 80 years old. Before each recording, we assured that the participant did not suffer from psychiatric symptoms, had no severe neurologic disorders, prior head injuries, a stroke, a transient circulatory disorder of the brain, diagnosis of dementia Huntington’s disease, Parkinson’s disease, sensory and/or motor problems that interfere with computer tasks. Moreover, exclusion criteria were current use of psychotropic drugs, intake of recreational synthetic or natural drugs.

16 Potential negative societal impacts of the work↩︎

In the specific case of our dataset, one risk of EEG-based eye-tracking models is that they could allow the tracking of the gaze of a subject without their consent, i.e., if the subject agrees to get their EEG data analyzed but not their gaze patterns. Furthermore, a major ethical concern is data privacy. A large-scale dataset such as the one we release in this work could be used in the future to develop methods capable of inferring sensitive information from EEG recordings, such as medical conditions. For this reason, we do not release any privacy-sensitive data. Especially, the anonymity of the participants will be guaranteed when presenting the data at scientific meetings or publishing them in scientific journals. Interested third parties can access the data (but under no circumstances the personal data) via the research repository, solely for scientific purposes (e.g. replication or further analysis). Individual participant medical information obtained from this research project is considered confidential, and disclosure to third parties is prohibited. Participant confidentiality will be further ensured by utilizing identification code numbers to correspond to medical information in the computer files.

17 Formal consent of Participants↩︎

In the following pages we present the template of the formal consent given to each participant. Every participant signed the formal consent before taking part in this study.

References↩︎