Testing the potential of combining Functional Near-Infrared Spectroscopy with different virtual reality displays – Oculus Rift and oCtAVE

Combining Virtual Reality (VR) with brain imaging may improve controllability and ecological validity in neuroscienceentific research and clinical applications. The aim of this pilot study was to assess the pros and cons of combining mobile neuroimaging with two different styles of highly immersive displays, one that is warn on the head and the other that is entered. Specifically the combination of wearable Functional Near Infrared Spectroscopy (fNIRS) with either an Oculus Rift and surround immersive projection technology. . A comparison is drawn in terms of hemodynamic response from the prefrontal cortex, signal to noise ratio, comfort, freedom of movement and motion artefacts. Findings suggest that choice of display should depend on research question.. The potential of this work is to improve ecological validity in research and therapy.


Introduction
Recent rapid technological advances in Virtual Reality (VR) and brain imaging offer opportunities for researchers in the field of neuroscience to investigate brain response in highly controllable and repeatable settings.VR allows to create more naturalistic environments which facilitate ecological validity (Bohil, Alicea, & Biocca, 2011).Traditionally neuroscience used standard desktop displays in order to present stimuli and investigate underlying brain response.This approach however is artificial and does not resemble real life situations.Previous neuroscientific studies demonstrated that the human brain responds differently to static emotional 2D images than to interactive emotional 3D environments (Dores et al., 2014).Therefore, there is a need to develop potentially more ecologically valid combinations of VR displays and brain imaging methods.This would allow improved neuroscientific research unobscured by the limits of screens.
The most commonly used VR systems are Head-mounted Displays (HMDs) and Cave Automatic Virtual Environment (CAVE) (Cruz-Neira, Sandin, DeFanti, Kenyon, & Hart, 1992).One of the disadvantages of HMDs is that it hides others from the user.Moreover, it generally restricts natural locomotion by restraining participants with cables and limited tracking.Further, the lack of naturalistic embodimentseeing your own body in VR, may potentially impact on presence, as well as cause a cybersickness (Malik, Blake, & Suggs., 2014) in HMDs.Cave-like systems offer a solution by immersing a user into a room-sized VR simulation which supports natural locomotion and interaction in the space, without losing the sight of one's own body or others.On the other hand, Cave-like systems can be costly and expensive to maintain in clinical settings.
This chapter presents and describes testing the combination of a wearable Functional Near -Infrared Spectroscopy (fNIRS) device with two different VR displays CAVElike Immersive Projection Technology (IPT) system -Octave, and a Oculus Rift DK2.The latter was adapted to improve comfort of fit of fNIRS.In particular, this chapter focuses on combining VR and brain imaging to investigate neural activity in the prefrontal cortex related to emotional regulation in VR, as well as SNR, motion artifact (noise in a data caused by body movement), peripheral signal interference and comfort of use.The classic VR experiment -The Pit Room (Meehan, Insko, Whitton, & Brooks, 2002) was employed in order to trigger an emotional response to VR. Meehan's PIT Room experiment demonstrated that exposure to stressful VR induces emotional response, even in healthy ,participants.This was measured by heart rate, electrodermal response and self-report (Meehan et al., 2002).When combining VR with neuroimaging, maintaining quality of the signal while allowing the naturalness of the movement and response, challenging.

Literature review
Many recent studies combined neuroimaging techniques with VR in research on human cognition and performance, such as human navigation and driving behaviour (Carvalho, Pearlson, Astur, & Calhoun, 2006), social interaction (King, Blair, Mitchell, Dolan, & Burgess, 2006), spatial memory (Burgess, Maguire, Spiers, & O'Keefe, 2001), violent behaviour (Mathiak & Weber, 2006), or emotion (Baumgartner & Valko, 2006).Most of these studies employed Functional Magnetic Resonance Imaging (fMRI) or Electroencephalography (EEG).The first one offers high spatial resolution, while the second offers better temporal resolution (Ferrari & Quaresima, 2012).However, lying within a huge and noisy fMRI scanner restricts freedom of movement and could possibly evoke anxiety in some participants (Irani, Platek, Bunce, Ruocco, & Chute, 2007).Recently, improvements have been made making EEG systems more compact, wireless and portable.This allows EEGs to be combined with VR systems in which people can more freely move (Török et al., 2014).The most challenging disadvantages of EGG are its susceptibility for motion artifacts, and electronic signal interference.fNIRS provides a tool for acquiring brain scans that are between spatial resolution of fMRI and temporal resolution of EEG within VR (Irani et al., 2007).fNIRS was combined with VR display for the first time by Holper and colleagues (2010) as a tool for monitoring virtual motor rehabilitative training (Holper et al., 2010).Other recent studies showed fNIRS can be used in combination with VR in balance control (Moro, Bisconti, & Muthalib, 2014), or navigation learning (Ayaz & Shewokis, 2011).However those studies did not use fully immersive systems and did involve freedom of the movement within VR.This pilot study tested the potential of utilising technologies which would facilitate naturalness of response while measuring neural activity from prefrontal cortex.
Previous neuroimaging studies demonstrated that prefrontal cortex plays a crucial role in emotional reappraisal and cognitive regulation of emotion (Grimm et al., 2006).In particular, neuroimaging studies performed on healthy participants found increased activity in the medial prefrontal cortex (MPFC) and dorsolateral prefrontal cortex (DLPFC) during perception of fearful pictures (Lange et al., 2003), fearful faces (Nomura et al., 2004), emotional reappraisal (Ochsner & Gross, 2005), or suppressing negative mood during decision making (Beer, Knight, & D'Esposito, 2006).Comparatively, studies performed on patients with depression and anxiety disorders demonstrated hypoactivation in the MPFC and DLPFC could indicate deficits in emotional regulation (Duval, Javanbakht, & Liberzon, 2015).

Method
This pilot study tested the potential of combining the wearable fNIRS device -NIRSport, with two VR display systems: a cave -like (IPT) system -Octave, and a custom fNIRS -adapted Oculus Rift DK2.The aim of this study was to measure brain response to the evocative VR.The objective of this study was to test the feasibility of the protocol in terms of the design, integration of technology and signal to noise ratio.The approach was to combine VR display systems in which participants can move freely while neural index of emotional regulation can be measured.

Immersive Projection Technology (IPT) Octave
Octave is an octagonal IPT space approximately 32.6 m 2 , which is big enough to allow a small group of people to mingle.Immersive Projection is delivered via the surrounding walls and floor display.There are 8 surrounding wall screens 2600mm x 1969mm with resolution 1400x1050 pixels, and 96Hz refresh rate.There are 14 Christie S+3K mirage projection units of which 6 project to the floor with 8 rear projection systems utilised for the walls.Octave images are generated by a workstation with 2x Xeon E5-2650, giving 32 threads, 64GB memory, SSD and 4xNvidia K5000 with a ksync card running a single desktop through 4 mosaic instances.Parallax is provided by the optical motion tracking system Vicon MX-F40, controlled by a Dell workstation running Windows 7 and Vicon Tracker 2.0, with custom designed optical markers on the glasses.The users in Octave wear XPAND 3D Shutter Glasses Lite RF (X105-RF-X1) with a shuttering frequency 96 Hz.There are no modifications required in order to measure haemodynamic responses to IPT based virtual stimuli utilising the NIRSport system, however the infrared tracking system and electronic light emitting peripheral devices may potentially introduce noise to the fNIRS data.

Custom Oculus Rift
The Oculus Rift DK2 needed adaption to allow space for the fNIRS optodes.Securing straps were changed -it was necessary to remove the 'over-the-head' strap of the oculus in order to ensure it was not in contact with probes of the NIRSport system.The remaining straps were removed and replaced with a buckled elastic strap.This helped to avoid issues caused by disturbance of probes when the HMD was put on.Due to the reduction of weight support across the top of the head, the HMDs weight was not being distributed comfortably when worn, with a moderate level of pressure on the end of the nose.In order to increase comfort, the frames of a pair of glasses were incorporated to improve support.Part of the top section of the HMD was cut away and edges were sanded to eliminate sharpness.The section removed was slightly larger than the length of a NIRSport standard probe in order to minimise chances of contact between HMD and probe during use.The lack of support across the forehead caused by removing a section of the HMD led to it leaning back toward the user at an inappropriate angle for use.Angled foam inserts combated this.

Functional Near Infrared Spectroscopy (fNIRS)
In order to measure changes in cerebellar oxygenation this study utilised the NIRSport system.(NIRsPORT 8-8, NIRx Medizintechnik GmbH, Berlin, Germany) which is a portable, wearable, multichannel fNIRS system consisting of 8 LED illumination sources and 8 active detection sensors.Emitters were placed on positions F3, AF7, AF3, Fz, Fpz, AF4, F4, AF8, while detectors were placed on positions F5, F1, Fp1, AFz, F2, Fp2, F6.Twenty channels were set up covering the prefrontal cortex (Fig 1 .).The source -detector distance was 3cm.Optodes were placed on the participant's head using an EasyCap relative to the international 10/20 system (Jasper, 1958).The data was acquired with the NIRStar acquisition Software (v2014 NIRx Medical Technologies LLC) at two near infra-red light wavelengths of 760nm and 850nm, with a sampling rate of 7.81 Hz.

Procedure
During the first stage of the experiment, both VR displays were combined with the NIRSport and were tested in terms of raw data quality through measuring Signal -To -Noise -Ratio (SNR).Visual inspection of the raw optical densities was carried out using NIRStar Software (version 2014, NIRx Medical Technologies LLC).
After the initial tests, eleven volunteers (N=11, 4 females and 7 males, mean age 33.18, SD=4.72) were recruited to walk on the virtual plank 6 meters above the floor in Octave.During the task, we recorded brain oxygenation changes from the prefrontal cortex as a neural index of emotional regulation.
The environment consisted of two rooms: the training room, which looked like a normal room with a floor and furniture, where participants familiarised themselves with the virtual environment and trained to carry out the task; and the pit room which had no floor but the wooden ledge on which participants stood and looked down into the pit room (which looks like an ordinary room with a floor and furniture) approximately 6m below the plank.

Task
Prior to the experiment participants were allowed to familiarise themselves with the virtual environment and practice carrying out the task for about 5 minutes.During the experiment they were asked to perform a simple walking task.In the training room the participant walked on the floor, and in the pit room the participant walked on the virtual plank.

Design
The experiment employed a within-subject design.Each participant performed a task under two experimental conditionstraining room and pit room.There were 10 trials split equally between the training room and pit room, with each trial lasting 30 seconds.A 30 second baseline was recorded prior to the first stimuli onset.During this participants were instructed to step on the actual floor outside of the simulation area in the training room, stay still, close their eyes, clear their mind and relax.After the baseline, participants heard pre-recorded audio instructions generated in random order, to move either to the training room or move to the pit room.The whole experiment lasted 330 seconds.We employed an event-related design where participants spent 30 seconds in the training room and 30 seconds in the pit room.There were no breaks between conditions.

Data analysis
Raw time series were assessed in NIRStar Software (version 2014, NIRx Medical Technologies LLC) by calculating a SNR and performing visual inspection of the raw optical densities using the function 'check raw data' within the software.To calculate SNR, the relative coefficient of variation (CV) was calculated for each channel.Data with CV over 15% was removed from the analysis.Raw data was converted to average haemoglobin concentration changes using the modified Beer-Lambert law for each channel, each subject, and each condition.fNIRS data was preprocessed and analysed using NIRSLab (NIRx Medical Technologies version 2014).Oxy-(HbO) and deoxy -(HbR) haemoglobin time series were band-pass filtered with low cut-off frequency 0.01 Hz and high cut-off frequency 0.2 Hz to remove drifts and noise from the data.Many previous studies have shown that HbO correlates with BOLD signals better than the HbR (Hoshi et al., 2001), therefore this study focused on HbO for further analysis.Statistical data analysis was performed using NIRSLab -SPM (SPM8).Data was modelled with GLM.Two regressors were generated by convolving the weighted task time series with the canonical hemodynamic response function provided by SPM8 (Friston et al., 1996).Discrete cosine transform basis functions were used for temporal filtering, and precoloring HRF was used for the serial correlations.On the next level analysis T-contrasts were created for HbO changes to generate statistical parametric maps of activation for two regressors: training room, and pit room, for each channel and each subject.SPM T-maps were generated by using two contrasts: training roompit toom and pit room-training room, and thresholded at p < 0.05 (corrected).At the group analysis, SPM group ΔHbO Tstatistics were calculated to identify the channels significantly activated with a significance level threshold set at p < 0.5 (corrected) according to the false discovery rate method FDR used in fMRI studies (Singh & Dan, 2006).The estimated anatomical locations of each channel were determined using anatomical locations of international 10-10 system cortical projections of EEG sensors (Koessler et al., 2009;Okamoto & Dan, 2005).

SNR and motion artifact analysis results
Figure 6.shows example raw time series data from a single subject acquired from both the Oculus -fNIRS test, and Octave -fNIRS test.SNR tests demonstrated that there was no significant signal interference from both VR displays -Oculus Rift and Octave.
Due to motion artifacts, 5% of the data was removed from the Oculus -fNIRS test, and 50% from Octave -fNIRS test.

Discussion and lesson learned
The main concern in this pilot study was a possible interference of the near infrared lights from motion tracking systems on fNIRS system.By measuring SNR, this was proved an ungrounded assumption for both VR displays.
Although fNIRS is less susceptible to motion artifacts than EEG and fMRI, still it is sensitive to sudden excessive movement.As an objective of this study was promoting freedom of movement in VR, the level of motion impact was investigated.On the one hand Oculus Rift caused less motion artifacts, but it restricted freedom of movement to that head.Therefore motion artifacts were likely to have been caused primarily by optode displacement during putting the device on.On the other hand combining fNIRS and Octave caused more motion artifacts due to both the nature of the display system, as well as the experimental task itself.The data analysis revealed motion artifacts in the signal when people leant forward excessively.This was potentially a problem given the task encouraged this.However, such a level of bend did not arise from the experimental protocol, but rather participants wanting to experiment with the experience.Moreover, this study investigated how much movement is too much to keep data motion artifact free.While it was possible to remove such data, it is better practice to exclude all the data from the session.Initially the measurement system was highly unstable.This came from the desire to maximise freedom of movement and approach.Physiological data can be communicated wirelessly from the sensors to the computer; however several technological issues, such as the data collection laptop overheating, and software crashes, were related to the high system requirements for brain data acquisition which requires a high-end laptop.The solution was to put the brain data acquisition laptop in a mesh backpack.
The adaptations made to the Oculus Rift DK2 are would be harder with the newer commercial versions of the Oculus.This contains wiring through the headband.It is important to note that adaptation was a prototype.Further work needs to be carried out to fully integrate HMD based VR with neuroimaging methodologies.A potential solution is to utilise 3D printing techniques to incorporate an EEG/fNIR cap into a specially designed HMD.
In conclusion, combining fNIRS with both VR displays offers new opportunities for the researchers.Both VR systems have their own pros and cons (summarised in table 1).Therefore the selection of appropriate display should be determined by the experimental design and the research question.Virtual Reality offers a solution for bridging the gap between ecological validity and controllability (Rey & Alcañiz, 2010).Both ecological validity and controllability are important in both VR and neuroscience research, and applications This study proposed and tested a solution that integrates wireless brain imaging and two VR displaysa large IPT VR solution in which users can move more freely -Octave, and a custom fNIRS-adapted Oculus Rift.The results of our pilot study suggested trends that indicate the potential for this integration of technology to evoke emotional response within VR.
We have demonstrated the feasibility of the study and resolved all technical problems.
Although this pilot study did no obtain statistically significant results due to the sample size, it identified promising trends showing that VR can trigger emotional regulation response which can be measured by a wireless brain imaging device.Results of our study demonstrated trends in increased haemoglobin oxygenation (HbO) in right MPFC and right DLPFC indicating emotional regulation processes in the brain when participants were exposed to evocative virtual stimulus.These results are consistent with previous neuroimaging studies (Quirk & Beer, 2006).However, this study did not constrain the natural movement of the participant in one of its conditions.
This pilot study lead to a further developed investigation.Since the reported investigation was conducted, the data collection for the full experiment with a larger sample was completed, and analyse is underway.
Potential impacts of this work include opening the door to more ecologically valid study of neuroscience and measurement of response in virtual reality, and potentially, adaptive immersive neurofeedback techniques in mental health and neurorehabilitation.

Fig. 5 .
Fig.5.Pit Room experimentview into the pit room (left), view in the training room (right)

Figure 6
Figure 6 example time series acquired from combining fNIRS with custom Oculus Rift DK2 (left) and Octave (right).Noise appears as rapid changes (such as spikes and steps)

Figure 7 .
Figure 7. shows the results for SPM group analysis (Pit room versus Training room).SPM contrast for group analysis (Pit room versus Training room) at the significance threshold level p < 0.05 (corrected) revealed no significant results.However, results showed a trend toward increased HbO in MPFC (channel 12, t (4) =1.99, p = 0.1175, two-tailed) and DLPF (channel 15, t (4) = 1.81, p = 0.1445, two-tailed) when participants exposed to the virtual heights in the Pit Room in comparison to the Training Room.

Fig. 7 .
Fig.7.SPM T-map of ΔHbO in the Pit Room versus Training Room.The t-value (unthresholded) is indicated by a colour scale.

Table 1 .
Advantages and disadvantages in combining fNIRS with HMDs and IPT