info@biomedres.us   +1 (720) 414-3554
  One Westbrook Corporate Center, Suite 300, Westchester, IL 60154, USA

Biomedical Journal of Scientific & Technical Research

July, 2019, Volume 20, 3, pp 14999-15009

Research Article

Research Article

Modeling Autonomic Pupillary Responses from External Stimuli Using Machine Learning

Shawhin Talebi*, David J. Lary, Lakitha O.H. Wijerante and Tatiana Lary

Author Affiliations

William B. Hanson Center for Space Sciences, Department of Physics, University of Texas at Dallas, USA

Received: July 30, 2019 | Published: August 08, 2019

*Corresponding author: Shawhin Talebi, William B. Hanson Center for Space Sciences, Department of Physics, University of Texas at Dallas, 800 W Campbell Rd, Richardson TX 75080, USA

DOI: 10.26717/BJSTR.2019.20.003446

Abstract

The human body exhibits a variety of autonomic responses. For example, changing light intensity provokes a change in the pupil dilation. In the past, formulae for pupil size based on luminance have been derived using traditional empirical approaches. In this paper, we present a different approach to a similar task by using machine learning to examine the multivariate non-linear autonomic response of pupil dilation as a function of a comprehensive suite of more than four hundred environmental parameters leading to the provision of quantitative empirical models. The objectively optimized empirical machine learning models use a multivariate non-linear non-parametric supervised regression algorithm employing an ensemble of regression trees which receive input data from both spectral and biometric data. The models for predicting the participant’s pupil diameters from the input data had a fidelity of at least 96.9% for both the training and independent validation data sets. The most important inputs were the light levels (irradiance) of the wavelengths near 562 nm. This coincides with the peak sensitivity of the long-wave photosensitive cones in the retina, which exhibit a maximum absorbance around max λ = 562.8 ± 4.7 nm.

Introduction

This study is part of a broader investigation into the role of the environment in influencing human physical and cognitive performance. The main purpose of this paper is to provide a baseline which accurately describes how changing illuminace affects pupil dilation, so that when emotional or cognitive factors are also involved, we can start to discern the relative roles of illumnance and cognitive load in affecting the pupil dilation [1-3]. The ranking of the importance of the predictor variables used in our empirical machine learning models provides a useful metric of which variables are the key drivers, providing us with valuable insights. The Autonomic Nervous System (ANT) is responsible for changes in pupil dilation. The changes in pupil dilation may occur due to changing light intensity, cognitive load and emotional load [4]. While the light intensity allows an immediate response at the retinal level, an emotional and especially cognitive response, require some higher level processing. So, when the visual input is sent from the eye to the visual cortex via the optic nerve, it first goes through the thalamus. If at this point an imminent threat is detected, it responds mobilizing the body for a ‘fight or flight’ response, which is then reflected in the changes in the pupil size. As the visual information is relayed to the visual center of the brain in the occipital lobe, it is further sent for processing via various routes to different parts of the brain. In a fast paced changing environment, executive function in the prefrontal lobes make decisions in a fraction of a second. This process also effects changes in pupil dilation. Some areas of the brain involved in the processing of cognitive and emotional load are deep seated structures and can only be observed by expensive equipment such as fMRI in an artificial lab setting. So, part of the question we are starting to address in this study is how can we tell the difference to which stimuli the pupil is responding? This study begins to answer this question using non-invasive methods that can be used in a natural setting by providing a methodology to accurately model the change in pupil size as a function of key environmental variables, so that when other changes are also occurring simultaneously (such as emotional and cognitive load) we can start to examine how these factors modify the pupil dilation response that occurs.

In addition to changes in pupil dilation, other autonomic responses include changes in heart rate variability, galvanic skin response (or sweating), and core temperature [5-7]. Each of these responses are influenced by variables such as cognitive load [8- 11], age [12], pain level [13], and emotional state [14]. In several previous studies formulae for pupil size utilized a single variable, luminance [15-19]. A major shortcoming of these models is their lack of generality. This is illustrated in Figure 1, where the true pupil diameter is plotted against the estimated pupil diameter provided by each of the models enumerated in the legend. There is a clear contrast between the diffuse cloud of data points from previous model predictions and the high fidelity predictions of the machine learning model developed here, shown by the green (training points) and the red (independent validation points) in the foreground. Of the five previous models, Holladay’s formula [15] performed the best, with a fidelity of 25%. The substantial error of these previous models is a likely reflection of both missing parameters and the challenge of finding the exact functional form required for predicting the pupil diameter. Later models added variables such as adaptation field, age, and monocular adaptation [2,16-21]. All of the earlier models considered ambient light levels by way of the total luminance as opposed to the fine wavelength resolution of the UV/visible spectrum that was used in this study. The fine wavelength resolution allows one to identify the wavelengths to which the pupil dilation is most sensitive, it is noteworthy that there are some small variations from eye to eye in the key wavelengths for determining the pupil diameter. In this study we have utilized recent technological developments, the full visible spectrum and pupil size can be measured with high accuracy and in large volume combined with machine learning, this provides new opportunities for the development of much more robust higher fidelity empirical models.

Figure 1: Evaluation and comparison of previous pupil diameter models which utilized a single variable, luminance, showing poor fidelity contrasted with the multivariate empirical machine learning model for the average pupil diameter developed in this study showing good fidelity (foreground green training and red validation points). The true average diameter of the left and right pupils is given on the y-axis, and the estimation by each respective model on the x-axis. Luminance was computed from measured illuminance where the luminance was assumed to be isotropic and reflectance assumed to be 1. Models were evaluated based on description by Watson and Yellott [2].

In this first demonstration case study, with just one participant, we examined the effect of both light intensity and the orientation/ motion of the head on the diameter of a participant’s pupils. Different illumination environments can be characterized by their spectra. This light consisting of various wavelengths can interact with different photo-receptors (light sensitive cones) in the retina. This interaction produces electrical signals that are sent to the brain and interpreted as color [22]. These cones are disproportionately sensitive to particular wavelengths with absorbance peaks around 420 nm (violet), 534 nm (green), and 564 nm (yellow-green) [3]. An illustration of these sensitivities can be shown by a plot of the mean absorbance of the three classes of photo-receptors (shortwave, middle-wave, and long-wave cones) vs wavelength (Figure 2).

Figure 2: Normalized mean absorbance spectra for long-wave, middle-wave, and short-wave cones. Maximum absorbance values for each class of cones are 420 nm ± 4.7 nm, 534 nm ± 3.7 nm, and 564 ± 4.7 nm, 420 nm ± 4.7 nm, respectively. Dashed vertical lines represent the top 4 important predictors taken from the pupil diameter models created here. The sensitivity range of the Konica Minolta CL- 500A Spectrophotometer is 360 – 780 nm indicated by the gray double-sided arrow. Cone absorbances were based on a figure in the paper by Bowmaker and Dartnall [3].

New predictive empirical models of the pupil diameter can be derived using supervised multivariate non-linear non-parametric machine learning regression. The accuracy of the models can be evaluated using an independent validation (or testing) dataset whose data records were not utilized in the model training. This machine learning approach can also provide insights on the relative importance of the inputs (i.e. predictors). In this case we had a few hundred inputs, including the light intensities for every nm of wavelengths from 360-780 nm (ultra-violet to near infrared).

Materials and Methods

Data was collected during 3 outdoor/indoor walks where spectral and biometric data were recorded. The walks took place in the morning (8:30 AM) and late afternoons (4 PM), each lasting approximately fifteen minutes. Spectral data was measured approximately every 3 seconds using a NIST calibrated Konica Minolta CL-500A Illuminance Spectrophotometer, which measures the illuminance and spectral irradiance of wavelengths from 360-780 nm with 10.3 nm resolution. Pupil diameters, head orientation, and the proper acceleration of the head were recorded 100 times a second using Tobii Pro Glasses 2. The glasses use an infrared grid projected onto each eye to estimate the position and size of the pupils. The orientation and acceleration of the head are estimated using a Microelectromechanical System (MEMS) gyroscope and MEMS accelerometer located in the glasses. Data was prepared and analyzed using Matlab 2019a.

The data preparation involved six steps:

1. Collection - Recording of the raw data. Data was written to 6 separate files corresponding to the 2 devices for each of the 3 trials.

2. Formatting - Converting raw data files to Matlab timetable objects. 6 timetables were created from the raw data files.

3. Synchronizing - The sampling frequencies differed for each device. 1 record every 3 seconds for the spectral data, versus 100 records every second for the biometric data. To account for this, the 2 timetables for a particular trial were reconfigured to share the same time steps using Matlab’s retime function with a linear interpolation. The timetables for each trial could then combined using the synchronize function. Resulting in 3 timetables, one for each of the 3 trials.

4. Merging - Concatenating all 3 timetables into a single timetable.

5. Cleaning - Removing records with device error flags, NaN elements, and zero values for pupil diameter. The latter case is addressed below.

6. Generating - Creating new variables such as the average pupil diameter and inter-eye pupil diameter difference.

A major challenge was introduced in step 5 (cleaning) of the data preparation due to a significant portion of the pupil diameter records taking values of 0. This was a non-physical consequence of the mechanism with which the pupil diameters were measured. When there is a high intensity of ambient infrared light from bright sunshine the glasses can no longer readily discern the pupil diameter, this is reflected in Figure 3 where pupil diameter dropouts coincide with time intervals of high spectral irradiance. These records were removed from the data, reducing the number of records from 380,000 to 80,000 records.

Figure 3: The normalized spectral irradiance at every time step for all walks is plotted. The irradiance is normalized by dividing all values by the maximum spectral irradiance within each walk. Relative size of irradiance values are indicated by the colorbar. Spectral lines at 528, 563, 567, and 776 nm represent the most important predictors for the pupil diameter models. Left (yellow) and right (green) pupil diameters are plotted over time. Note the pupil diameter dropouts in time intervals where the spectral irradiance is high.

a) Walk 1 measurements during late afternoon (≈ 4PM).
b) Walk 2 measurements during morning (≈ 8:30 AM) with overcast.
c) Walk 3 measurements during late afternoon (≈ 4 PM).

From the recorded data we sought to estimate 5 different parameters, namely the: average of the left and right pupil diameters (APD), left pupil diameter (LPD), right pupil diameter(s) (RPD), magnitude of the difference between the left and right pupil diameters (PDD), and the illuminance. These parameters can be estimated by constructing objectively optimized empirical machine learning models. The hyperparameters (i.e. the parameters that define options associated with the training process) of an ensemble of regression trees able to use both boosting and bagging were optimized (the Matlab function fitrensemble with the Optimize Hyperparameters option set to all). More information on this function is available in the Matlab documentation [23]. We have done many previous machine learning studies [24-56]. The data was split into 2 subsets: one for training and one for the independent testing of each empirical machine learning model. With 90% of the data used for training the multivariate non-linear non-parametric regression models and 10% of the data used for independent testing of the models.

Results and Discussion

In the following subsections we discuss the results of the 5 different empirical machine learning models. The accuracy of each model was assessed via a scatter plot of the true vs estimated response variable values (see Figures 4a, 5a, 6a, 7a, & 9a). If the true and estimated values are identical, the resulting scatter plot will be a straight line with a slope of one and an intercept of zero, i.e. a perfect one to one plot with a correlation coefficient, r2, equal to 1. This ideal is indicated by a black line in each scatter plot. The correlation coefficients for the training (plotted as green circles) and testing (plotted as red pluses) datasets were computed using Matlab’s corrcoef function.

The relative predictor importance ranking of each model was derived using the predictorImportance function. The relative rankings are visualized as bar plots (see Figures 4b, 5b, 6b, 7b, & 9b). The importance estimates are plotted on a log scale with the most important predictors shown toward the top. In the pupil diameter models (i.e. models for the APD, LPD, RPD, and PDD), the top 20 out of 427 predictors are shown. For the illuminance model, all 7 predictors are given in the ranking. The top 3 predictors are indicated by red bars, the next 2 important predictors by yellow bars, and the remaining predictors by blue bars.

The Average Pupil Diameter Model

Figure 4 shows the results of the Average Pupil Diameter (APD) model. The APD was estimated using the spectral irradiance at every nm between 360-780 nm, the gyroscope, and the accelerometer data as predictor variables. The scatter plot of the true vs the estimated average pupil diameter values is shown in Figure 4a. The model had correlation coefficients of > 0.99 for both the training and testing data subsets. Thus, the empirical machine learning model was successful in predicting the average pupil diameter. Figure 4b shows the ranking of the relative importance of the inputs in predicting the APD, the top 3 predictors are the irradiance values at 561, 563, and 562 nm, which coincides with the maximum absorbance of the long-wave cones at around 563 nm [3]. This suggests the long-wave photo-receptors play a more significant role than the short- or middle-wave receptors in controlling the average size of the pupils for the participant.

Figure 4: Plots for the Average Pupil Diameter model.

a) True vs estimated average pupil diameter in millimeters.
b) Predictor importance estimates for the average pupil diameter model.

The Left Pupil Diameter Model

The results for the Left Pupil Diameter (LPD) model are shown in Figure 5. The LPD was estimated using the same predictors as the APD, the spectral irradiance from 360-780 nm, the gyroscope, and the accelerometer data. The model was successful in predicting the LPD with a correlation coefficient of > 0.96 for both the training and validation data subsets. The top predictor (567 nm) is again near the maximum absorbance of the long-wave photo-receptors (563 nm). The next top 6 predictors are the irradiance values at 528, 568, 564, 527, 668 and 570 nm, which seem to coincide with both the middle- and long-wave photo-receptors with maximum absorbance values near 533.8 ± 3.7 nm and 563 nm, respectively, with the exception of the irradiance at 668 nm [3].

Figure 5: Plots for the Left Pupil Diameter model.

a) True vs estimated left pupil diameter in millimeters.
b) Predictor importance estimates for the left pupil diameter model.

The Right Pupil Diameter Model

The results for the Right Pupil Diameter (RPD) model are shown in Figure 6. The RPD was estimated using the same predictors as the APD and LPD. For the RPD model there is a strong correlation between the estimated and true values, with coefficients of determination > 0.99 for both data subsets, shown in Figure 6a. The top 2 predictors are 563 nm and 562 nm, which again coincide with the maximum absorbance of the long-wave cones near 563 nm. The next most important predictor was the irradiance at 776 nm corresponding to near infrared light. This and the appearance of near infrared predictors in all the importance rankings may be a consequence of the infrared noise in the environment, resulting in the measured pupil diameters to be smaller than the actual values. An interesting result from the importance ranking in Figure 6b, is the appearance of a non-spectral predictor (Accelerometer Z) which denotes the proper acceleration in the direction in front of the glasses. This may be correlated to the participant looking down to navigate obstacles in the walking path such as stairs, inclines, rugged terrain, and other impediments. Focusing on a specific task or object may cause an increase in cognitive load, resulting in a pupillary response [10,11].

Figure 6: Plots for the Right Pupil Diameter model in millimeters.

a) True vs. estimated right pupil diameter in millimeters.
b) Predictor importance estimates for the right pupil diameter model.

The Pupil Diameter Difference Model and Pupil Asymmetry

The results for the left and right pupil diameter models are noticeably different (see Figures 5 and 6), which may suggest an asymmetry in the behavior of each pupil. One measure of this asymmetry is the magnitude of the difference between the left and right pupil diameters. This is shown by the results of the Pupil Diameter Difference (PDD) model given in Figure 7. The same predictors were used for the PDD model as in the APD, LPD, and RPD models. This empirical model was not successful in predicting the PDD, since the correlation coefficient was 0.43 for the testing data subset, as shown in Figure 7a. Clearly the most important predictors for modeling this asymmetry were not available in the training dataset.

Figure 7: Plots for the Pupil Diameter Difference.

a) True vs estimated pupil diameter differences in millimeters.
b) Predictor importance estimates for the pupil diameter difference model.

Another metric of the pupil asymmetry can be the accuracy of the LPD model in estimating the RPD and vice versa. The resulting scatter plots are given in Figure 8. Despite the differences in the importance rankings and failures of the PDD model, the estimates are fairly accurate with correlation coefficients of > 0.95 for both the testing and training datasets. This accuracy may suggest that although there is an asymmetry in the importance rankings for the left and right pupil models, the functioning of each pupil is very similar. A possible cause of this asymmetry is ocular dominance (i.e. the input for one eye is preferred over the other) [57,58]. It has been suggested that ocular dominance is not a static phenomenon, but will vary with changing horizontal gaze angle [59].

Figure 8: Plots for the pupil diameter prediction using model from opposite eye data. Pupil diameters are in millimeters.

a) True vs estimated left pupil diameter using the right pupil diameter model.
b) True vs estimated right pupil diameter using the left pupil diameter model.

The Illuminance Model

Figure 9 shows the results of the Illuminance model. We just saw above that if we know the light intensity we can accurately predict the pupil diameter, so now we `invert’ the experiment and ask the question, if we know the pupil diameter can we accurately estimate the light intensity? The model used the pupil diameters, gyroscope, and accelerometer data as the predictors. The estimates were somewhat accurate with correlation coefficients of 0.91 and 0.71 for the training and testing datasets, respectively. The top 2 predictors are the left and right pupil diameters, which agrees with first order considerations of the relationship between pupil diameters and external light levels. The next most important predictor was the acceleration in the z-direction (forward direction). Which may again be correlated with participant focus on obstacle navigation.

Figure 9: Plots for the Illuminance model.

a) True vs estimated illuminance in lux.
b) Predictor importance estimates for the illuminance model.

Pupil Diameter and Illuminance

In a first order consideration, we can expect the pupil diameter to be inversely proportional to the illuminance. This is depicted in Figure 10, which gives 3 scatter plots of the average, left, and right pupil diameters vs illuminance. At low illuminance values, the expected inverse relationship is apparent. At higher values (> 4000 lux) this expectation fails. The lack of a clear relationship between the two variables in all situations is likely the main contributor to the failure of previous models (Figure 1).

Figure 10: Log scale scatter plots of the pupil diameters vs illuminance. Data from walks 1, 2, and 3 are distinguished by the colors red, green, and blue, respectively. Data points with low opacity have illuminance values above 4000 lx. Note below the 4000 lx mark the variables tend to have an inverse relationship.

a) Average pupil diameter vs illuminance.
b) Left pupil diameter vs illuminance.
c) Right pupil diameter vs illuminance.

The Environment

The normalized spectral irradiance at every time step for each trial is given in Figure 3. Normalized values were computed by dividing all irradiance values by the largest irradiance within each trial. Spectral lines are plotted for 528, 563, 567, and 776 nm, based on the top 3 most important predictors across all pupil diameter models (see Figures 4b, 5b, 6b, & 7b). Where predictors of the spectral irradiance at 561, 562, and 568 nm were disregarded in lieu of the irradiance at 563 and 567 nm.

Temporal discontinuities in the spectra are due to those time intervals in which the participant walked in and out of shaded areas and/or away from the sun, which resulted in orders of magnitude differences in the spectral irradiance. Figure 11 depicts the normalized spectral irradiance plotted on a log scale. Time intervals colored predominately red represent outdoor spectra, while more colorful intervals are indoor.

Figure 11: The log of the normalized spectral irradiance at every time step for all walks is plotted. The irradiance is normalized prior to taking log by dividing all values by the maximum spectral irradiance within each walk. Relative sizes of irradiance values are indicated by the color bar. Spectral lines at 528, 563, 13, 567, and 776 nm represent wavelengths of the most important predictors for the pupil diameter models.

a) Walk 1 measurements during late afternoon (≈ 4PM).
b) Walk 2 measurements during morning (≈ 8:30 AM).
c) Walk 3 measurements during late afternoon (≈ 4PM).

Limitations

The high level of infrared noise caused significant drawbacks in the data analysis. Further developments may require light intensities and spectra to be within a non-disruptive range. Another solution may be to utilize an eye tracking instrument which uses visible light to estimate the pupil diameters.

Future Directions

Pupil size along with other autonomic responses such as heart rate variability, galvanic skin response, and core temperature changes have been associated with cognitive load and performance [5-11]. Although cognitive load is a significant contributor to the provocation of these responses, in a dynamic outdoor environment and while performing a physical activity (such as walking or cycling) it is not always clear which responses were due to external stimuli or cognitive status. Using a similar approach to the one used here, future data collection will expand the number of participants, environments, cognitive tasks, and biometric sensors.

Looking forward, multiple participants will allow for the assessment of the inter-person variability of the models, including parameters such as age and body composition. Different environments will vary in light intensity, air quality, elevation, and temperature. Environmental variables can be measured using mobile weather stations mounted on a participant or bicycle. Other environmental sensors such as a video camera, microphone, and LIDAR can indicate dynamic field situations and track events. Tasks such as walking and cycling will be performed. Cyclist performance can be assessed via bicycle speed and biometric data. Biometrics such as Electroencephalography (EEG), Heart Rate (ECG), Galvanic Skin Response (GSR), body temperature, Electromyography (EMG), blood oxygen level, and respiration will be considered and modeled. The ranking of predictor importance for these biometric models can help identify important relationships between environmental stimuli and different autonomic response.

Conclusion

Past formulae for predicting pupil diameter mainly considered total ambient light levels via luminance [2,15-21], these models could not capture the fully multi-variate and non-linear dependence of pupil diameter on the environmental state, and consequently had poor generalization. When considering the spectrum of light from 360-780 nm (ultra-violet to near infrared) in lieu of the luminance, we were able to derive a very accurate empirical machine learning model which can predict pupil diameters with a minimum fidelity of 96.9%. The machine learning also allowed us to identify that the most important wavelengths in predicting the pupil diameters were around 562 nm (green), which is near the peak absorbance of the long-wave photo-receptive cones (562.8 ± 4.7 nm) [3].

Funding

This research was funded by USAMRMC Award Number W81XWH-18-1-0400.

References

Research Article

Modeling Autonomic Pupillary Responses from External Stimuli Using Machine Learning

Shawhin Talebi*, David J. Lary, Lakitha O.H. Wijerante and Tatiana Lary/h3>

Author Affiliations

Received: July 30, 2019 | Published: August 08, 2019

Corresponding author:Shawhin Talebi, William B. Hanson Center for Space Sciences, Department of Physics, University of Texas at Dallas, 800 W Campbell Rd, Richardson TX 75080, USA

DOI: 10.26717/BJSTR.2019.20.003446

Abstract

Abbreviations: The human body exhibits a variety of autonomic responses. For example, changing light intensity provokes a change in the pupil dilation. In the past, formulae for pupil size based on luminance have been derived using traditional empirical approaches. In this paper, we present a different approach to a similar task by using machine learning to examine the multivariate non-linear autonomic response of pupil dilation as a function of a comprehensive suite of more than four hundred environmental parameters leading to the provision of quantitative empirical models. The objectively optimized empirical machine learning models use a multivariate non-linear non-parametric supervised regression algorithm employing an ensemble of regression trees which receive input data from both spectral and biometric data. The models for predicting the participant’s pupil diameters from the input data had a fidelity of at least 96.9% for both the training and independent validation data sets. The most important inputs were the light levels (irradiance) of the wavelengths near 562 nm. This coincides with the peak sensitivity of the long-wave photosensitive cones in the retina, which exhibit a maximum absorbance around λmax = 562.8 ± 4.7 nm.