Combining EEG and eye tracking can open new possibilities for your EEG analysis. If you would like to add eye tracking to your EEG setup but are unsure how to implement this, we have great news for you: Thanks to our new cooperation with Tobii Pro, Brain Products now offers complete out-of-the-box solutions for simultaneous EEG and eye tracking!
Abstract
This article intoduces how you can combine your EEG measurements with simultaneous eye tracking. We offer a full example workflow for a specific lab-based setup, while pointing to generally important aspects for a successful combination of EEG and eye tracking. For our setup, we are using the software Tobii Pro Lab for experimental control, the Tobii Pro Spectrum for recording eye tracking data, and the actiCHamp Plus to record EEG data in combination with our Photo Sensor. In the workflow, we describe how you can design your experiment while setting up shared event markers, how to perform the combined recordings, and how to merge both data streams in BrainVision Analyzer 2.
Boost your EEG research with simultaneous eye tracking!
In the last decade, the combination of eye tracking with measures of brain activity like EEG or fMRI has increased. But why would we want to take a closer look at the eyes when investigating brain activity?
Eye tracking offers two major sources of information:
The position of a person’s gaze gives us insight into the “open” focus their attention, and this information can be highly valuable for EEG research. With this gaze information, you will be able to tell if participants are focusing their attention on a target, when exactly their focus arrives and for how long it stays until shifting elsewhere. This will let you identify trials in which the participant was not paying attention and discard them from your analyses. Most importantly though, you gain precise timing information for your EEG analysis. Event-related potentials (ERPs) can, for example, be calculated with respect to the fixation on a stimulus (i.e., Fixation-Related Potentials), instead of the mere stimulus appearance on the screen. Research addressing topics like attentional processes, visual search, reading or social perception can highly benefit from gaze information.
Changes in pupil size can inform about cognitive and emotional experiences. The pupil reacts with short dilations (in the second range) to different stimuli. These “pupil responses” are a very sensitive physiological measure, and their magnitude reflects the intensity of the undergoing cognitive/emotional processes. Thereby, stimuli that are more emotionally arousing, or that demand higher cognitive effort cause larger pupil responses. By analyzing them, you may be able to check if your experimental manipulation was successful, or to even follow cognitive or emotional processes dynamically throughout your experiment. In combination with EEG, you could, for example, use the magnitude of pupil responses to weigh or categorize different trials in your experiment.
Adding eye tracking to your EEG setup will hence open a range of new possibilities for your research!
Figure 1. Combined EEG and eye tracking setup in a laboratory setting
A workflow for your lab-based EEG & eye tracking experiment
Combining two different measures like EEG and eye tracking can be technically challenging, especially if they should be temporally aligned and analyzed together. The key here is setting up shared event markers/triggers that will appear in both the EEG and the eye tracking data at the same time. Note that not all trigger signals need to be shared. It is enough to have a few (at least two) shared event markers to align the data sets after recording. It is however crucial that an equal number of the shared events appear in both data sets, and that they mark common points in time. Therefore, we need to plan our setup and experiment with these shared trigger events in mind.
Event markers are usually generated by the software used for experimental control (like E-Prime®, Presentation®, Psychtoolbox or Tobii Pro Lab‘s Designer module). There are many ways to pass them on to your EEG and eye tracking recordings. The best setup for you will depend on your experimental software and the properties of the computer, EEG amplifier and eye tracker you are using.
Here, we want to show you one concrete example for setting up simultaneous EEG and eye tracking recordings. We are going to explain how you can record high-quality data in a lab-based setup using:
- The actiCHamp Plus EEG amplifier in combination with a Photo Sensor from Brain Products
- The Spectrum screen-based eye tracker by Tobii Pro
- Tobii Pro Lab for designing and presenting an experiment, as well as for recording and analyzing the eye tracking data
- BrainVision Analyzer 2.2.1 with the Add Channels module to integrate EEG and eye tracking data for combined analysis
Figure 2. Example workflow for simultaneous EEG and eye tracking recordings
1. Design your experiment and set up shared event markers
Naturally, the workflow needs to start with designing and planning your experiment. If you use the Tobii Pro Lab software for the experimental design, it will allow you to set up the timeline of your experiment in a very intuitive way. Make sure the timeline always starts with a calibration and validation routine to accurately map and record gaze data. Next, you can add all sorts of stimuli to the timeline, e.g., pictures, text elements, videos, or groups of stimuli. You can find an introduction video on how to create a screen-based study with Tobii Pro Lab here, and further useful information here.
When designing your experiment, you need to set up shared event markers that will allow you to temporally align EEG and eye tracking data after recording. Note that you will need at least two markers of the same type appearing at the same time in both data streams. For example, you can send the first synchronization marker a few seconds before your task begins, and the last one a few seconds after the task finishes. This way your synchronization markers span the whole experiment, and you can align the EEG and eye tracking data sets completely.
(a) Marking events in the eye tracking data
The Pro Spectrum eye tracker can receive TTL trigger signals. However, in this specific example, we are using Tobii Pro Lab not only to present stimuli, but also to record eye tracking data. Therefore, all presented stimuli will be marked automatically as “Events” in the eye tracking data and you don’t need to worry about triggers.
(b) Marking events in the EEG data
To mark stimulus events in the EEG data, TTL hardware triggers are usually the preferred solution because they offer highly accurate timing. Tobii Pro Lab can send TTL pulses to mark stimulus events if your computer has a parallel port card available. However, for this scenario we will assume that you are working with a laptop that has no parallel port.
With a small workaround, you can still precisely record the stimulus timing in your EEG data by using a Photo Sensor. This small accessory detects changes in brightness that can be recorded alongside your EEG data. Simply attach the Photo Sensor to one corner of the presentation screen and modify your stimuli in a way that they differ in brightness in this very corner (see Figure 3). This way, the photo sensor will detect a change in brightness every time the next stimulus is presented. During later analysis, you can identify the stimulus onsets from the Photo Sensor signal. The timing of this solution is very precise because stimuli are detected by the Photo Sensor exactly when they appear on the screen.
Tip: If you want to directly generate trigger events from your Photo Sensor, you can combine it with the Brain Products StimTrak! The StimTrak can convert the Photo Sensor signal into trigger pulses and pass them on to your EEG recording where they will appear as event markers. See this article for more information.
Tip: If you want to identify different kinds of stimuli from your Photo Sensor signal, you can modify your stimuli with different shades of grey (this article offers a more detailed description).
Figure 3. Using a Photo Sensor to detect stimulus onsets. In this example, two checkerboard stimuli (A and B) are shown alternatingly on the presentation screen. Only Stimulus A displays a bright square in one corner. If the Photo Sensor is attached in this corner of the presentation screen, it will detect the change in brightness at every onset and offset of Stimulus A. During later data analysis, the Photo Sensor signal can be used to derive stimulus markers with very precise timing.
Here are a few additional options for alternative setups:
Find a support article here about sending TTL trigger pulses, or read about our TriggerBox for sending trigger signals via USB port.
If your EEG amplifier and your eye tracker have trigger ports and you can send TTL pulses, you can share the exact same triggers among both devices. Either split the trigger signal with a Y-cable, or use the practical trigger mirroring function of our actiCHamp Plus: this amplifier can receive 8-bit triggers and can immediately pass them on to your eye tracker!
If you are using E-Prime® for presenting your experiment and have a screen-based Tobii eye tracker, you may be interested in the E-Prime extension for Tobii Pro Lab.
2. Prepare the eye tracking recordings
To set up your eye tracking recording, your Spectrum eye tracker needs to be connected and correctly set up in Tobii Pro Lab (find more information here). Once this is done, you will find everything you need in the “Record” tab of Tobii Pro Lab. Here, you should pay special attention to the sampling rate (or “sampling frequency”) with which you are recording the eye tracking data (click on the eye tracker symbol in the top left corner). Higher sampling rates allow you to assess not only fixations, saccades and even micro-saccades (see this article), but they also allow you to record the stimulus events with more temporal precision. Therefore, higher sampling rates are better for a more precise synchronization with the EEG data.
It is also important to set up the stimulus markers in Pro Lab with the highest temporal precision. You may encounter delays between the stimulus marker being registered in Pro Lab, and the stimulus actually appearing on the presentation screen. To reduce such delays, please make sure that the computer running Pro Lab matches the required specifications, and carefully follow these important tips to optimize your stimulus timing in Pro Lab. To find how you can determine this delay in your setup, and how you can account for it during recording, you can take a look at this Timing Guide.
Before recording data with an actual participant, you will need to run at least one test recording of your final task and make sure your current setup and the available stimulus events let you analyze everything of interest in Pro Lab’s “Analyze” tab. If all events are marked in Pro Lab and you are satisfied with their timing, you are all set for the eye tracking recordings.
To prepare your EEG recordings, you will need to set up the actiCHamp Plus with the PowerUnit, and connect the Photo Sensor to one of the amplifier’s AUX channels. When preparing your workspace in BrainVision Recorder, make sure to also set up the respective AUX channel for recording the Photo Sensor signal. For the EEG data, we can use a higher sampling rate (for example 2000 Hz) to have a high temporal precision of the signal and a good synchronization with the eye tracking data.
When everything is set up, you will need to identify the correct position for the Photo Sensor on the presentation screen. For this, briefly start a test run of your experiment and attach the Photo Sensor to the monitor with an adhesive ring. Next, run a test EEG recording to make sure you can identify all necessary stimulus events in the recorded Photo Sensor signal. Present the full experiment while recording, then load the data in BrainVision Analyzer 2. If your setup contains the Photo Sensor in combination with a StimTrak, the stimulus events should already be marked in your EEG data. Otherwise, you can now use the “Level Trigger” transformation. Here, you can identify the optimal threshold value for your Photo Sensor data and extract the stimulus events from the Photo Sensor channel (see Figure 4).
Figure 4. Identify the stimulus onsets from the photo sensor channel with the Level Trigger transformation.
Keep in mind that the shared synchronization events need to appear at the same time in both EEG and eye tracking data, and that there need to be an equal number of synchronization events present in both data sets. If necessary, you can use the “Edit Markers” transformation to rename or modify some events in your EEG data.
4. Record EEG and eye tracking data simultaneously
Now you are ready for the real data acquisition! Set up the EEG system and cap, use the prepared workspace and the Photo Sensor. To get ready for the eye tracking recordings, load the correct experiment in Tobii Pro Lab. Then have the participant sit in front of the eye tracker and presentation screen at the optimal distance. After double-checking that all settings are correct (see section “2. Prepare the eye tracking recordings” above), you can enter a name for your participant, and the “Record data” button will become available in Tobii Pro Lab.
When starting the recording in Tobii Pro Lab, follow the calibration and validation procedure until you are satisfied with accuracy and precision. Before you start the actual task, make sure to start your EEG recordings in time for the Photo Sensor to capture the first synchronization marker. Always keep an eye on the data streams in BrainVision Recorder and Tobii Pro Lab to make sure all data is recorded smoothly. When the task is finished, again make sure the Photo Sensor captured the last synchronization event before stopping the EEG recording.
Tip: Make sure you do not stop or pause the EEG recordings before the task is fully finished, so you can later align the EEG and eye tracking data sets!
5. Analyze the eye tracking data
Now it’s time to analyze your eye tracking data in Tobii Pro Lab’s “Analyze” tab. It is good practice to start with some quality control (reviewing the recording and checking for data loss). Then you will be able to perform all kinds of analyses, export metrics or create graphics from your recorded gaze data. What may be most relevant for your combined EEG and eye tracking analysis is to identify times of interest or fixations in areas of interest in your eye tracking data.
When you are done with your eye tracking analysis, you can export the gaze and pupil data together with all identified event markers and import them into your EEG data. For this, use the “Data Export” option in Pro Lab and export the data in the Pro Lab Output File (PLOF) format.
6. Identify the event markers in your EEG data
After a brief quality control, you can extract all stimulus events from the Photo Sensor channel by using the “Level Trigger” transformation with the previously tested settings (see section “3. Prepare the EEG recordings” above). If necessary, modify the resulting markers so you can clearly identify the synchronization events that should be shared with the eye tracking recording.
Tip: Be careful with segmenting your EEG data before importing the eye tracking data to make sure you don’t lose important synchronization markers!
7. Merge both data sets for combined EEG and eye tracking analysis
Finally, you can import the eye tracking data and the events you identified in your eye tracking analysis into your EEG recordings. At this time point, the sampling rates and the length of both data sets will likely be different, but BrainVision Analyzer 2 will now use the shared synchronization events to bring both data streams to the same timeline.
To merge the EEG and eye tracking data, open the EEG data containing the identified synchronization events. Next, use Analyzer’s Add Channels transform and select the previously exported eye tracking file under Import files. In the next window, you will need to select the shared synchronization markers which will be used to align both data sets. For the EEG data, they can be selected from the Markers in Active Node list, for the eye tracking data from the Markers in Import File list. If you click on the Details button, you will see if there is an equal number of synchronization markers in both data sets.
In the following dialogs, you will be able to select the specific channels and markers you would like to import. Finally, when you finish the Transformation, the eye tracking channels will appear underneath your EEG channels, and all selected event markers will be imported.
Tip: You can find a full description of how to use the Add Channels transform in the BrainVision Analyzer 2 User Manual, and more information about its latest enhanced features here. However you can also always contact our Scientific Support team if you need help with BrainVision Analyzer 2.
Now that both data streams are temporally aligned, you can start analyzing them together! As mentioned in the introduction, you can discard data during which the subject was not focusing on areas of interest. Finally, you can also segment your EEG data based on fixations or other events you identified in your eye tracking analysis, and you can calculate Fixation-Related Potentials.
Conclusion
We hope this article provided you with helpful guidelines for your lab-based EEG and eye tracking setup, and that we could walk you through the most important steps for your recordings and analysis. Keep your eyes open for more articles as well as dedicated online events about our new eye tracking solutions!
