The two major laboratory experiments on the accuracy of OCR using a single camera were completed in the final funding period. One study involved measuring how well the single-camera OCR algorithm accurately tracked emotional expressions in N= 31 healthy subjects who underwent emotional induction techniques. Preliminary analyses of the overall extent to which the initial 1-camera OCR algorithm could identify specific emotional expressions in many individuals with limited training revealed that the algorithm often failed to discriminate among emotions. That is, the algorithm had modest sensitivity and low specificity (i.e., it selected negative emotions too often and failed to discriminate among them). Although the facial expression models being used by the tracker were appropriate, it became apparent that a great deal of OCR inaccuracy was due to problems in identifying facial expressions when the face was partially out of view, which occurs frequently as people move their heads in all dimensional planes as they move about, work, etc. Thus, although we trained the OCR facial expression models with frontal images of facial expressions of emotion, the videos of subjects experiencing emotions would many times show subjects in non-frontal poses. The OCR algorithm model would then fail to correctly recognize the facial expression. To correct for this problem, the Metaxas Lab developed a sufficiently approximate warping transformation to warp the tracked face to a frontal pose (which is what the OCR algorithm expects to evaluate), as well as enhancing the algorithm with other analytic techniques that improve single-camera face tracking and extrapolation of facial expressions when the face is moving and/or partially out of view. The second validation experiment was conducted on a separate group of N=33 healthy adults randomized to either sleep deprivation (N=18) or no sleep deprivation (N=15). The total number of subjects to be studied was decreased in order to focus on improved computational detection of emotion and fatigue. The goal of this experiment was to determine the extent to which the model-based OCR algorithm validly detected and tracked ocular changes in slow eyelid closures (PERCLOS) when sleep deprived and when not sleep deprived, and the extent to which the PERCLOS measure reliably tracked lapses of attention during psychomotor vigilance test (PVT) performance.

Experiment 2 was our first attempt to track PERCLOS with a novel 1-camera OCR algorithm. The results were very promising. Subjects completed a 20-min PVT every 2 h while awake. Images of the face were recorded during each performance test. PERCLOS was manually scored frame-by-frame by 4 human scorers blinded to sleep condition and time of testing, and it was scored by the OCR algorithm (programmers blind to these factors). Coherence was calculated as the extent to which PVT lapses of performance were tracked by OCR-scored PERCLOS while subjects were and were not sleep deprived. Although OCR PERCLOS did not predict the occurrence of every PVT performance lapse, it was remarkably accurate for the vast majority of them, showing excellent coherence across a 36-h period of sleep deprivation, and a marked increase after sleep deprivation relative to before it, but no change in subjects who were resistant to the effects of sleep deprivation on PVT performance, or subjects who were not sleep deprived. Thus, the sensitivity of the current 1-camera OCR algorithm for detecting PERCLOS as an index of fatigue-related performance failures was 73.3% and the specificity was 89.1%. This is excellent sensitivity-specificity. These data confirm that the OCR PERCLOS detector rarely yielded false positives, and that it was acceptably high in sensitivity to predicting performance lapses. OCR PERCLOS will never reach 100% sensitivity when validated against PVT lapses because some performance lapses can occur when the eyelids are open. The findings suggest a single-camera OCR algorithm could detect the presence of performance-impairing fatigue from sleep loss in spaceflight.

The second validation experiment was conducted on a separate group of healthy adults randomized to either sleep deprivation or no sleep deprivation. The experiment sought to determine the extent to which the OCR algorithm detected ocular changes in slow eyelid closures (PERCLOS), and the extent to which the OCR PERCLOS measure reliably tracked lapses of attention during PVT performance. This was our first attempt to track PERCLOS with a 1-camera OCR algorithm. Subjects completed a 20-min PVT every 2 h while awake. Images of the face were recorded during each performance test. Coherence was calculated as the extent to which PVT lapses of performance were tracked by OCR-scored PERCLOS while subjects were and were not sleep deprived. The study revealed that the 1-camera OCR algorithm for PERCLOS had 73% sensitivity and 89% specificity for PVT performance lapses, thus confirming that the OCR PERCLOS detector rarely yielded false positives, and that it was acceptably high in sensitivity to fatigue-related performance risks in spaceflight.

In Experiment 1 (emotion recognition), a total of 31.2 hours of footage for facial emotional analysis has been collected, with 8.5 hours collected during year 3 alone. Subjective emotional questionnaires also were administered to all subjects. The 31.2 hours of footage (n=28, 14 females) has been subjectively scored by humans, who were blinded to the emotion induction tasks. Scorers rated each video based on the durations of emotional states, and which emotion was predominantly expressed. These results will be compared to the scores obtained from the model-based tracker algorithm and will assess and validate the accuracy of the computer-based system.

In Experiment 2 (stress and fatigue detection), a total of 166.7 hours of digitally recorded high definition footage was collected capturing the faces of subjects during performance of the Psychomotor Vigilance Task (PVT), with 49.3 hours collected during year 3 alone (In the 2009-2010 report, we incorrectly reported that we had a total of 180 hours of PVT footage). Human scorers manually scored a subset (n=16) of the 20-minute videos collected during the PVT. The state of the eyelid was scored frame-by-frame (4 states: open, closing, closed, opening) and then compared to the algorithm output. Facelab data and PVT reaction times were simultaneously recorded throughout these test bouts, which were administered every 2 hours over the course of two consecutive days. During year 3, 4.5 hours of footage was recorded for facial emotional analysis (14.1 hours total) and 25 hours of footage also was collected for stress analysis (88.7 hours total) during this protocol. In addition, data from neuropsychological tasks, personality questionnaires, and subjective emotional rating scales were obtained from all subjects.

With regard to the stress-related hormone analysis in Experiment 2, a total of 140 saliva samples were collected during stress-inducing tasks (5 saliva samples per subject), with 40 samples collected during year 3 alone. EEG/EKG data were also collected during these stress-inducing tasks.

In Experiment 1 (emotion recognition), 15 hours of footage for facial emotional analysis was collected. Subjective emotional questionnaires also were administered to all subjects.

In Experiment 2 (stress and fatigue detection), 180 hours of digitally recorded high definition footage was collected capturing the faces of subjects during performance of the Psychomotor Vigilance Task (PVT). Facelab data and PVT reaction times were simultaneously recorded throughout these test bouts, which were administered every 2 hours over the course of two consecutive days. 3 hours of footage was recorded for facial emotional analysis and 20 hours of footage also was collected for stress analysis. In addition, data from neuropsychological tasks, personality questionnaires, and subjective emotional rating scales were obtained from all subjects.

With regard to the stress-related hormone analysis in Experiment 2, a total of 45 saliva samples were collected during stress-inducing tasks (5 saliva samples per subject). EEG/EKG data were also collected during these stress-inducing tasks.

The project has four specific aims: (1) Create an OCR system capable of monitoring facial displays of specific emotions (i.e. angry, happy and sad). (2) Improve our current OCR system's ability to detect facial expressions of high versus low performance-induced stress. (3) Develop OCR algorithms to identify fatigue due to sleep loss based on slow eyelid closures (PERCLOS). (4) Test the technical feasibility of data acquisition and reliability of the advanced OCR system in spaceflight analogs that contain neurobehavioral stressors relevant to spaceflight (e.g., NEEMO). The project has primary relevance to strategic goals of the NSBRI Neurobehavioral and Psychosocial Factors (NBPF) Team. It addresses a high priority gap identified by the NASA SAT, BHP, and NSBRI NBPF area, and specifically targets questions 25d,c,g,h of Bioastronautics Roadmap Risk Area 25 (Human Performance Failure Due to Neurobehavioral Problems), and question 27d in Risk Area 27 (Human Performance Failure Due to Sleep Loss and Circadian Rhythm Problems).

We have made several other new developments to the OCR system: (1) the technique was validated with the use only one camera, where the previous method required two; (2) we improved tracking by using a manifold of faces that helped automatically track the face as the head moves; (3) we added the use of Conditional Random Fields in addition to Hidden Markov Modeling, to the algorithm which improved its computational efficiency; and (4) GABOR filtering (used for edge detection in image analysis) was incorporated into the ASM algorithm to track changes in facial texture, allowing it to identify features (e.g., furrowed brow).

In the first year of the current project we have begun expanding the algorithm to recognize facial expressions of emotion and behavioral indicators of excessive sleepiness (through slow eyelid closures). We are also continuing our work to improve the system's ability to correctly identify stress. Preliminary data confirm that the experimental procedures reliably induce stress, emotion and fatigue. In this first year we designed and implemented the two experiments we proposed (one on emotion detection and one on stress and fatigue detection. Twenty healthy subjects have completed the two experiments (N=9 in Experiment 1 and N=11 in Experiment 2). We are using these data to expand and improve the current OCR algorithm.

This optical computer recognition (OCR) system will provide feedback to them for autonomous selection of countermeasures for stress, depression and fatigue. The project will be accomplished through collaborative efforts of Dr. David Dinges (Unit for Experimental Psychiatry) at the University of Pennsylvania School of Medicine, and Dr. Dimitris Metaxas (Computational Biomedicine Imaging and Modeling Center) at Rutgers University.

Specific Aims

1) Create an OCR system capable of monitoring facial displays of specific emotions (i.e., angry, happy and sad).

2) Improve our current OCR systems ability to detect facial expressions of high-performance versus low-performance-induced stress.

3) Develop OCR algorithms to identify fatigue due to sleep loss based on slow eyelid closures.

4) Test the technical feasibility of data acquisition and reliability of the advanced OCR system in spaceflight analogs, such as NEEMO, that contain neurobehavioral stressors relevant to spaceflight.

The project has primary relevance to strategic goals of the NSBRI Neurobehavioral and Psychosocial Factors (NBPF) Team. It addresses a high-priority gap identified by the NASA Small Assessment Team, Behavioral Health and Performance, and NSBRI NBPF Team areas. and the project specifically targets questions 25d, c, f, and h of Bioastronautics Roadmap Risk Area 25 (Human Performance Failure Due to Neurobehavioral Problems), and question 27d in Risk Area 27 (Human Performance Failure Due to Sleep Loss and Circadian Rhythm Problems).