Modeling Autonomic Pupillary Responses from External Stimuli Using Machine Learning

The human body exhibits a variety of autonomic responses. For example, changing light intensity provokes a change in the pupil dilation. In the past, formulae for pupil size based on luminance have been derived using traditional empirical approaches. In this paper, we present a different approach to a similar task by using machine learning to examine the multivariate non-linear autonomic response of pupil dilation as a function of a comprehensive suite of more than four hundred environmental parameters leading to the provision of quantitative empirical models. The objectively optimized empirical machine learning models use a multivariate non-linear non-parametric supervised regression algorithm employing an ensemble of regression trees which receive input data from both spectral and biometric data. The models for predicting the participant’s pupil diameters from the input data had a fidelity of at least 96.9% for both the training and independent validation data sets. The most important inputs were the light levels (irradiance) of the wavelengths near 562 nm. This coincides with the peak sensitivity of the long-wave photosensitive cones in the retina, which exhibit a maximum absorbance around max λ = 562.8 ± 4.7 nm.


Introduction
This study is part of a broader investigation into the role of the environment in influencing human physical and cognitive performance. The main purpose of this paper is to provide a baseline which accurately describes how changing illuminace affects pupil dilation, so that when emotional or cognitive factors are also involved, we can start to discern the relative roles of illumnance and cognitive load in affecting the pupil dilation [1][2][3]. The ranking of the importance of the predictor variables used in our empirical machine learning models provides a useful metric of which variables are the key drivers, providing us with valuable insights. The Autonomic Nervous System (ANT) is responsible for changes in pupil dilation. The changes in pupil dilation may occur due to changing light intensity, cognitive load and emotional load [4]. While the light intensity allows an immediate response at the retinal level, an emotional and especially cognitive response, require some higher level processing. So, when the visual input is sent from the eye to the visual cortex via the optic nerve, it first goes through the thalamus. If at this point an imminent threat is detected, it responds mobilizing the body for a 'fight or flight' response, which is then reflected in the changes in the pupil size. As the visual information is relayed to the visual center of the brain in the occipital lobe, it is further sent for processing via various routes to different parts of the brain. In a fast paced changing environment, executive function in the prefrontal lobes make decisions in a fraction of a second. This process also effects changes in pupil dilation. Some areas of the brain involved in the processing of cognitive and emotional load are deep seated structures and can only be observed by expensive equipment such as fMRI in an artificial lab setting. So, part of the question we are starting to address in this study is how can we tell the difference to which stimuli the pupil is responding? This study begins to answer this question using non-invasive methods that can be used in a natural setting by providing a methodology to accurately model the change in pupil size as a function of key environmental variables, so that when other changes are also occurring simultaneously (such as emotional and cognitive load) we can start to examine how these factors modify the pupil dilation response that occurs.
In addition to changes in pupil dilation, other autonomic responses include changes in heart rate variability, galvanic skin response (or sweating), and core temperature [5][6][7]. Each of these responses are influenced by variables such as cognitive load [8][9][10][11], age [12], pain level [13], and emotional state [14]. In several previous studies formulae for pupil size utilized a single variable, luminance [15][16][17][18][19]. A major shortcoming of these models is their lack of generality. This is illustrated in Figure 1, where the true pupil diameter is plotted against the estimated pupil diameter provided by each of the models enumerated in the legend. There is a clear contrast between the diffuse cloud of data points from previous model predictions and the high fidelity predictions of the machine learning model developed here, shown by the green (training points) and the red (independent validation points) in the foreground. Of the five previous models, Holladay's formula [15] performed the best, with a fidelity of 25%. The substantial error of these previous models is a likely reflection of both missing parameters and the challenge of finding the exact functional form required for predicting the pupil diameter. Later models added variables such as adaptation field, age, and monocular adaptation [2,[16][17][18][19][20][21]. All of the earlier models considered ambient light levels by way of the total luminance as opposed to the fine wavelength resolution of the UV/visible spectrum that was used in this study. The fine wavelength resolution allows one to identify the wavelengths to which the pupil dilation is most sensitive, it is noteworthy that there are some small variations from eye to eye in the key wavelengths for determining the pupil diameter. In this study we have utilized recent technological developments, the full visible spectrum and pupil size can be measured with high accuracy and in large volume combined with machine learning, this provides new opportunities for the development of much more robust higher fidelity empirical models. The true average diameter of the left and right pupils is given on the y-axis, and the estimation by each respective model on the x-axis. Luminance was computed from measured illuminance where the luminance was assumed to be isotropic and reflectance assumed to be 1. Models were evaluated based on description by Watson and Yellott [2].
In this first demonstration case study, with just one participant, This interaction produces electrical signals that are sent to the brain and interpreted as color [22]. These cones are disproportionately sensitive to particular wavelengths with absorbance peaks around 420 nm (violet), 534 nm (green), and 564 nm (yellow-green) [3].
An illustration of these sensitivities can be shown by a plot of the mean absorbance of the three classes of photo-receptors (shortwave, middle-wave, and long-wave cones) vs wavelength ( Figure 2).

Figure 2:
Normalized mean absorbance spectra for long-wave, middle-wave, and short-wave cones. Maximum absorbance values for each class of cones are 420 nm ± 4.7 nm, 534 nm ± 3.7 nm, and 564 ± 4.7 nm, 420 nm ± 4.7 nm, respectively. Dashed vertical lines represent the top 4 important predictors taken from the pupil diameter models created here. The sensitivity range of the Konica Minolta CL-500A Spectrophotometer is 360 -780 nm indicated by the gray double-sided arrow. Cone absorbances were based on a figure in the paper by Bowmaker and Dartnall [3].
New predictive empirical models of the pupil diameter can be derived using supervised multivariate non-linear non-parametric machine learning regression. The accuracy of the models can be evaluated using an independent validation (or testing) dataset whose data records were not utilized in the model training. This machine learning approach can also provide insights on the relative importance of the inputs (i.e. predictors). In this case we had a few hundred inputs, including the light intensities for every nm of wavelengths from 360-780 nm (ultra-violet to near infrared). The data preparation involved six steps:

1.
Collection -Recording of the raw data. Data was written to 6 separate files corresponding to the 2 devices for each of the 3 trials.

2.
Formatting -Converting raw data files to Matlab timetable objects. 6 timetables were created from the raw data files.

3.
Synchronizing -The sampling frequencies differed for each device. 1 record every 3 seconds for the spectral data, versus 100 records every second for the biometric data. To account for this, the 2 timetables for a particular trial were reconfigured to share the same time steps using Matlab's retime function with a linear interpolation. The timetables for each trial could then combined using the synchronize function. Resulting in 3 timetables, one for each of the 3 trials.

5.
Cleaning -Removing records with device error flags, NaN elements, and zero values for pupil diameter. The latter case is addressed below.

6.
Generating -Creating new variables such as the average pupil diameter and inter-eye pupil diameter difference.
A major challenge was introduced in step 5 (cleaning) of the data preparation due to a significant portion of the pupil diameter records taking values of 0. This was a non-physical consequence of the mechanism with which the pupil diameters were measured.
When there is a high intensity of ambient infrared light from bright sunshine the glasses can no longer readily discern the pupil diameter, this is reflected in Figure 3 where pupil diameter dropouts coincide with time intervals of high spectral irradiance.
These records were removed from the data, reducing the number of records from 380,000 to 80,000 records.  Hyperparameters option set to all). More information on this function is available in the Matlab documentation [23]. We have done many previous machine learning studies . The data was split into 2 subsets: one for training and one for the independent testing of each empirical machine learning model. With 90% of the data used for training the multivariate non-linear non-parametric regression models and 10% of the data used for independent testing of the models.

Results and Discussion
In   Predictor importance estimates for the average pupil diameter model.

The Left Pupil Diameter Model
The results for the Left Pupil Diameter (LPD) model are shown in Figure 5. The LPD was estimated using the same predictors as the APD, the spectral irradiance from 360-780 nm, the gyroscope, and the accelerometer data. with the exception of the irradiance at 668 nm [3].  Predictor importance estimates for the right pupil diameter model.

The Pupil Diameter Difference Model and Pupil Asymmetry
The results for the left and right pupil diameter models are noticeably different (see Figures 5 and 6), which may suggest an asymmetry in the behavior of each pupil. One measure of this asymmetry is the magnitude of the difference between the left and right pupil diameters. This is shown by the results of the Pupil Diameter Difference (PDD) model given in Figure 7. The same predictors were used for the PDD model as in the APD, LPD, and RPD models. This empirical model was not successful in predicting the PDD, since the correlation coefficient was 0.43 for the testing data subset, as shown in Figure 7a. Clearly the most important predictors for modeling this asymmetry were not available in the training dataset. (i.e. the input for one eye is preferred over the other) [57,58]. It has been suggested that ocular dominance is not a static phenomenon, but will vary with changing horizontal gaze angle [59].  Predictor importance estimates for the illuminance model.

Pupil Diameter and Illuminance
In a first order consideration, we can expect the pupil diameter to be inversely proportional to the illuminance. This is depicted in Figure 10  Left pupil diameter vs illuminance. c) Right pupil diameter vs illuminance.

The Environment
The normalized spectral irradiance at every time step for each trial is given in Figure 3. Temporal discontinuities in the spectra are due to those time intervals in which the participant walked in and out of shaded areas and/or away from the sun, which resulted in orders of magnitude differences in the spectral irradiance. Figure 11 depicts the normalized spectral irradiance plotted on a log scale. Time intervals colored predominately red represent outdoor spectra, while more colorful intervals are indoor.

Limitations
The high level of infrared noise caused significant drawbacks in the data analysis. Further developments may require light intensities and spectra to be within a non-disruptive range. Another solution may be to utilize an eye tracking instrument which uses visible light to estimate the pupil diameters.

Future Directions
Pupil size along with other autonomic responses such as heart rate variability, galvanic skin response, and core temperature changes have been associated with cognitive load and performance [5][6][7][8][9][10][11]. Although cognitive load is a significant contributor to the provocation of these responses, in a dynamic outdoor environment and while performing a physical activity (such as walking or cycling) it is not always clear which responses were due to external stimuli or cognitive status. Using a similar approach to the one used here, future data collection will expand the number of participants, environments, cognitive tasks, and biometric sensors.
Looking forward, multiple participants will allow for the assessment of the inter-person variability of the models, including parameters such as age and body composition. Different environments will vary in light intensity, air quality, elevation, and temperature. Environmental variables can be measured using mobile weather stations mounted on a participant or bicycle. Other environmental sensors such as a video camera, microphone, and LIDAR can indicate dynamic field situations and track events. Tasks such as walking and cycling will be performed. Cyclist performance can be assessed via bicycle speed and biometric data. Biometrics such as Electroencephalography (EEG), Heart Rate (ECG), Galvanic Skin Response (GSR), body temperature, Electromyography (EMG), blood oxygen level, and respiration will be considered and modeled. The ranking of predictor importance for these biometric models can help identify important relationships between environmental stimuli and different autonomic response.

Conclusion
Past formulae for predicting pupil diameter mainly considered total ambient light levels via luminance [2,[15][16][17][18][19][20][21], these models could not capture the fully multi-variate and non-linear dependence of pupil diameter on the environmental state, and consequently had poor generalization. When considering the spectrum of light from 360-780 nm (ultra-violet to near infrared) in lieu of the luminance, we were able to derive a very accurate empirical machine learning model which can predict pupil diameters with a minimum fidelity of 96.9%. The machine learning also allowed us to identify that the most important wavelengths in predicting the pupil diameters were around 562 nm (green), which is near the peak absorbance of the long-wave photo-receptive cones (562.8 ± 4.7 nm) [3].