Menu

 


Download in PDF pdf     
Fiscal Year: FY 2012  Task Last Updated:  01/08/2013 
PI Name: Dinges, David F. Ph.D. 
Project Title: Optical Computer Recognition of Stress, Affect and Fatigue during Performance in Spaceflight 
   
Division Name: Human Research 
Program/Discipline--
Element/Subdiscipline:
NSBRI--Neurobehavioral and Psychosocial Factors Team 
 
Joint Agency Name:   TechPort:  Yes 
Human Research Program Elements: (1) BHP:Behavioral Health & Performance
Human Research Program Risks: (1) Bmed:Risk of Adverse Behavioral Conditions and Psychiatric Disorders
Human Research Program Gaps: (1) BMed02:We need to identify and validate measures to monitor behavioral health and performance during exploration class missions to determine acceptable thresholds for these measures (IRP Rev F)
Space Biology Element: None
Space Biology Cross-Element Discipline: None
Space Biology Special Category: None
PI Email: dinges@pennmedicine.upenn.edu  Fax:  215-573-6410 
PI Organization Type: UNIVERSITY  Phone: 215-898-9949  
Organization Name: University of Pennsylvania 
PI Address 1: Department of Psychiatry 
PI Address 2: 423 Service Dr, 1013 Blockley Hall 
PI Web Page:  
City: Philadelphia  State: PA 
Zip Code: 19104-4209  Congressional District: 
Comments:  
Project Type: GROUND  Solicitation:  2007 Crew Health NNJ07ZSA002N 
Start Date: 05/01/2008  End Date:  09/30/2012 
No. of Post Docs: No. of PhD Degrees: 
No. of PhD Candidates: No. of Master' Degrees: 
No. of Master's Candidates: No. of Bachelor's Degrees: 
No. of Bachelor's Candidates: Monitoring Center:  NSBRI 
Contact Monitor:   Contact Phone:   
Contact Email:  
Flight Program:  
Flight Assignment:

 

Key Personnel Changes/Previous PI:  
COI Name (Institution): Metaxas, Dimitris   ( Rutgers University )
Goel, Namni   ( University of Pennsylvania )
Basner, Mathias   ( University of Pennsylvania ) 
Grant/Contract No.: NCC 9-58-NBPF01601 
Performance Goal No.:  
Performance Goal Text:

 

Task Description: Astronauts must maintain high-level performance while experiencing demanding workload and work schedules, extreme environmental risks, and psychosocial stressors in space (e.g., isolation, confinement). Stress, negative emotions, and fatigue can jeopardize their cognitive performance and neurobehavioral status. The proposed research is developing and validating an objective, unobtrusive, computational model-based tracker of the human face that reliably identifies when astronauts are experiencing stress, emotion, and fatigue at levels that compromise performance in space. This optical computer recognition (OCR) system will provide feedback to them for autonomous selection of countermeasures for stress, depression, and fatigue. The project is being accomplished through collaborative efforts of Dr. David Dinges (Unit for Experimental Psychiatry) at the Perelman School of Medicine at the University of Pennsylvania, and Dr. Dimitris Metaxas (Computational Biomedicine Imaging and Modeling Center) at Rutgers University. The project has four specific aims: (1) Create an OCR system capable of monitoring facial displays of specific emotions (i.e., angry, happy, and sad). (2) Improve our current OCR system's ability to detect facial expressions of high versus low performance-induced stress. (3) Develop OCR algorithms to identify fatigue due to sleep loss based on slow eyelid closures (PERCLOS). (4) Test the technical feasibility of data acquisition and reliability of the advanced OCR system in spaceflight analogs that contain neurobehavioral stressors relevant to spaceflight. The project has primary relevance to strategic goals of the National Space Biomedical Research Institute (NSBRI) Neurobehavioral and Psychosocial Factors (NBPF) Team.

The two major laboratory experiments on the accuracy of OCR using a single camera were completed in the final funding period. One study involved measuring how well the single-camera OCR algorithm accurately tracked emotional expressions in N= 31 healthy subjects who underwent emotional induction techniques. Preliminary analyses of the overall extent to which the initial 1-camera OCR algorithm could identify specific emotional expressions in many individuals with limited training revealed that the algorithm often failed to discriminate among emotions. That is, the algorithm had modest sensitivity and low specificity (i.e., it selected negative emotions too often and failed to discriminate among them). Although the facial expression models being used by the tracker were appropriate, it became apparent that a great deal of OCR inaccuracy was due to problems in identifying facial expressions when the face was partially out of view, which occurs frequently as people move their heads in all dimensional planes as they move about, work, etc. Thus, although we trained the OCR facial expression models with frontal images of facial expressions of emotion, the videos of subjects experiencing emotions would many times show subjects in non-frontal poses. The OCR algorithm model would then fail to correctly recognize the facial expression. To correct for this problem, the Metaxas Lab developed a sufficiently approximate warping transformation to warp the tracked face to a frontal pose (which is what the OCR algorithm expects to evaluate), as well as enhancing the algorithm with other analytic techniques that improve single-camera face tracking and extrapolation of facial expressions when the face is moving and/or partially out of view. The second validation experiment was conducted on a separate group of N=33 healthy adults randomized to either sleep deprivation (N=18) or no sleep deprivation (N=15). The total number of subjects to be studied was decreased in order to focus on improved computational detection of emotion and fatigue. The goal of this experiment was to determine the extent to which the model-based OCR algorithm validly detected and tracked ocular changes in slow eyelid closures (PERCLOS) when sleep deprived and when not sleep deprived, and the extent to which the PERCLOS measure reliably tracked lapses of attention during psychomotor vigilance test (PVT) performance.

Experiment 2 was our first attempt to track PERCLOS with a novel 1-camera OCR algorithm. The results were very promising. Subjects completed a 20-min PVT every 2 h while awake. Images of the face were recorded during each performance test. PERCLOS was manually scored frame-by-frame by 4 human scorers blinded to sleep condition and time of testing, and it was scored by the OCR algorithm (programmers blind to these factors). Coherence was calculated as the extent to which PVT lapses of performance were tracked by OCR-scored PERCLOS while subjects were and were not sleep deprived. Although OCR PERCLOS did not predict the occurrence of every PVT performance lapse, it was remarkably accurate for the vast majority of them, showing excellent coherence across a 36-h period of sleep deprivation, and a marked increase after sleep deprivation relative to before it, but no change in subjects who were resistant to the effects of sleep deprivation on PVT performance, or subjects who were not sleep deprived. Thus, the sensitivity of the current 1-camera OCR algorithm for detecting PERCLOS as an index of fatigue-related performance failures was 73.3% and the specificity was 89.1%. This is excellent sensitivity-specificity. These data confirm that the OCR PERCLOS detector rarely yielded false positives, and that it was acceptably high in sensitivity to predicting performance lapses. OCR PERCLOS will never reach 100% sensitivity when validated against PVT lapses because some performance lapses can occur when the eyelids are open. The findings suggest a single-camera OCR algorithm could detect the presence of performance-impairing fatigue from sleep loss in spaceflight.

 

Rationale for HRP Directed Research:

 

Research Impact/Earth Benefits: The study focuses on the ability of an unobtrusive, automated optical computer recognition (OCR) technology to detect psychological stress, emotion, and fatigue during operational performance. The knowledge gained has the potential to identify an objective, unobtrusive, automated method for the recognition, monitoring, and management of the risks of neurobehavioral dysfunction in affect and alertness in spaceflight and in many Earth-based safety-sensitive occupations, such as NASA mission controllers, first responders, transportation workers (e.g., truck drivers, train conductors, airline pilots); operators in safety-sensitive industries (e.g., power plant control rooms); and military personnel.

 

Task Progress: The two major laboratory experiments on the accuracy of OCR using a single camera were completed in the final funding period. One study involved measuring how well the single-camera OCR algorithm accurately tracked emotional expressions in healthy subjects who underwent emotional induction techniques. Preliminary analyses of the overall extent to which the initial 1-camera OCR algorithm could identify specific emotional expressions in many individuals with limited training revealed that the algorithm often failed to discriminate among emotions. That is, the algorithm had modest sensitivity and low specificity (i.e., it selected negative emotions too often and failed to discriminate among them). Although the facial expression models being used by the tracker were appropriate, it became apparent that a great deal of OCR inaccuracy was due to problems in identifying facial expressions when the face was partially out of view, which occurs frequently as people move their heads in all dimensional planes as they move about, work, etc. Thus, although we trained the OCR facial expression models with frontal images of facial expressions of emotion, the videos of subjects experiencing emotions would many times show subjects in non-frontal poses. The OCR algorithm model would then fail to correctly recognize the facial expression. To correct for this problem, the Metaxas Lab developed a sufficiently approximate warping transformation to warp the tracked face to a frontal pose (which is what the OCR algorithm expects to evaluate), as well as enhancing the algorithm with other analytic techniques that improve single-camera face tracking and extrapolation of facial expressions when the face is moving and/or partially out of view.

The second validation experiment was conducted on a separate group of healthy adults randomized to either sleep deprivation or no sleep deprivation. The experiment sought to determine the extent to which the OCR algorithm detected ocular changes in slow eyelid closures (PERCLOS), and the extent to which the OCR PERCLOS measure reliably tracked lapses of attention during PVT performance. This was our first attempt to track PERCLOS with a 1-camera OCR algorithm. Subjects completed a 20-min PVT every 2 h while awake. Images of the face were recorded during each performance test. Coherence was calculated as the extent to which PVT lapses of performance were tracked by OCR-scored PERCLOS while subjects were and were not sleep deprived. The study revealed that the 1-camera OCR algorithm for PERCLOS had 73% sensitivity and 89% specificity for PVT performance lapses, thus confirming that the OCR PERCLOS detector rarely yielded false positives, and that it was acceptably high in sensitivity to fatigue-related performance risks in spaceflight.

 

Bibliography Type: Description: (Last Updated: 07/03/2019) Show Cumulative Bibliography Listing
 
Abstracts for Journals and Proceedings Di Antonio A, Moreta M, Michael N, Muto J, Minkel JD, Yang F, Htaik O, Metaxas DN, Dinges DF. "An Objective Unobtrusive Model-Based Tracker of Facial Expressions of Negative and Positive Emotions in Space Flight." 2012 NASA Human Research Program Investigators’ Workshop, Houston, TX, February 14-16, 2012.

2012 NASA Human Research Program Investigators’ Workshop, Houston, TX, February 14-16, 2012. , Feb-2012

Abstracts for Journals and Proceedings Jones CW, Basner M, Goel N, Metaxas D, Dinges DF. "Unobtrusive Tracking of Slow Eyelid Closures as a Measure of Fatigue From Sleep Loss." 25th Annual Meeting of the Associated Professional Sleep Societies, LLC 2011, Minneapolis, MN, June 11-15, 2011.

Sleep 2011;34 Suppl:A114. , Jun-2011

Abstracts for Journals and Proceedings Jones CW, Basner M, Yu X, Yang F, Goel N, Metaxas D, Dinges DF. "Unobtrusive Tracking of Slow Eyelid Closures as a Measure of Fatigue from Sleep Loss." 26th Annual Meeting of the Associated Professional Sleep Societies, Boston, MA, June 9-13, 2012.

Sleep. 2012;35 Suppl:A110. , Jun-2012

Abstracts for Journals and Proceedings Jones CW, Yang F, Basner M, Goel N, Abe T, Metaxas D, Dinges DF. "Unobtrusive Tracking of PERCLOS as a Measure of Sleepiness-Related Fatigue in Space Flight." 2012 NASA Human Research Program Investigators’ Workshop, Houston, TX, February 14-16, 2012.

2012 NASA Human Research Program Investigators’ Workshop, Houston, TX, February 14-16, 2012. , Feb-2012

Articles in Peer-reviewed Journals Balkin TJ, Horrey WJ, Graeber RC, Czeisler CA, Dinges DF. "The challenges and opportunities of technological approaches to fatigue management." Accident Analysis and Prevention. 2011 Mar;43(2):565-72. Review. http://dx.doi.org/10.1016/j.aap.2009.12.006 ; PubMed PMID: 21130217 , Mar-2011
Articles in Peer-reviewed Journals Minkel JD, Banks S, Htaik O, Moreta MC, Jones CW, McGlinchey EL, Simpson NS, Dinges DF. "Sleep deprivation and stressors: Evidence for elevated negative affect in response to mild stressors when sleep deprived." Emotion. 2012 Oct;12(5):1015-20. Epub 2012 Feb 6. http://dx.doi.org/10.1037/a0026871 ; PubMed PMID: 22309720 , Oct-2012
Papers from Meeting Proceedings Hu Q, Yang F, Yang P, Peng X, Zhong L, Zhang S, Metaxas DN. "Facial Expression Recognition under Large Pose Variation." 10th IEEE International Conference on Automatic Face and Gesture Recognition 2013, Shanghai, China, April 22-26, 2013.

10th IEEE International Conference on Automatic Face and Gesture Recognition 2013, Shanghai, China, April 22-26, 2013. , Apr-2013

Papers from Meeting Proceedings Yang F, Huang J, Metaxas DM. "Sparse Shape Registration for Occluded Facial Feature Localization." 2011 IEEE International Conference on Automatic Face and Gesture Recognition, Santa Barbara, California, March 21-25, 2011.

2011 IEEE International Conference on Automatic Face and Gesture Recognition, Santa Barbara, California, March 21-25, 2011. p. 272-277. http://dx.doi.org/10.1109/FG.2011.5771410 , Mar-2011

Papers from Meeting Proceedings Yang F, Huang J, Yang P, Metaxas DM. "Eye Localization through Multiscale Sparse Dictionaries." 2011 IEEE International Conference on Automatic Face and Gesture Recognition, Santa Barbara, California, March 21-25, 2011.

2011 IEEE International Conference on Automatic Face and Gesture Recognition, Santa Barbara, California, March 21-25, 2011. p. 514-518. http://dx.doi.org/10.1109/FG.2011.5771450 , Mar-2011

Download in PDF pdf     
Fiscal Year: FY 2011  Task Last Updated:  05/04/2011 
PI Name: Dinges, David F. Ph.D. 
Project Title: Optical Computer Recognition of Stress, Affect and Fatigue during Performance in Spaceflight 
   
Division Name: Human Research 
Program/Discipline--
Element/Subdiscipline:
NSBRI--Neurobehavioral and Psychosocial Factors Team 
 
Joint Agency Name:   TechPort:  Yes 
Human Research Program Elements: (1) BHP:Behavioral Health & Performance
Human Research Program Risks: (1) Bmed:Risk of Adverse Behavioral Conditions and Psychiatric Disorders
Human Research Program Gaps: (1) BMed02:We need to identify and validate measures to monitor behavioral health and performance during exploration class missions to determine acceptable thresholds for these measures (IRP Rev F)
Space Biology Element: None
Space Biology Cross-Element Discipline: None
Space Biology Special Category: None
PI Email: dinges@pennmedicine.upenn.edu  Fax:  215-573-6410 
PI Organization Type: UNIVERSITY  Phone: 215-898-9949  
Organization Name: University of Pennsylvania 
PI Address 1: Department of Psychiatry 
PI Address 2: 423 Service Dr, 1013 Blockley Hall 
PI Web Page:  
City: Philadelphia  State: PA 
Zip Code: 19104-4209  Congressional District: 
Comments:  
Project Type: GROUND  Solicitation:  2007 Crew Health NNJ07ZSA002N 
Start Date: 05/01/2008  End Date:  04/30/2012 
No. of Post Docs: No. of PhD Degrees: 
No. of PhD Candidates: No. of Master' Degrees: 
No. of Master's Candidates: No. of Bachelor's Degrees: 
No. of Bachelor's Candidates: 37  Monitoring Center:  NSBRI 
Contact Monitor:   Contact Phone:   
Contact Email:  
Flight Program:  
Flight Assignment:

 

Key Personnel Changes/Previous PI:  
COI Name (Institution): Metaxas, Dimitris   ( Rutgers University )
Goel, Namni   ( University of Pennsylvania )
Basner, Mathias   ( University of Pennsylvania ) 
Grant/Contract No.: NCC 9-58-NBPF01601 
Performance Goal No.:  
Performance Goal Text:

 

Task Description: Astronauts must maintain high-level performance while experiencing demanding workload and work schedules, extreme environmental risks, and psychosocial stressors in space (e.g., isolation, confinement). Stress, negative emotions and fatigue can jeopardize their cognitive performance and neurobehavioral status. The proposed research is developing and validating an objective, unobtrusive, computational model-based tracker of the human face that reliably identifies when astronauts are experiencing stress, emotion and fatigue at levels that compromise performance in space. This optical computer recognition (OCR) system will provide feedback to them for autonomous selection of countermeasures for stress, depression and fatigue. The project is being accomplished through collaborative efforts of Dr. David Dinges (Unit for Experimental Psychiatry) at the University of Pennsylvania School of Medicine, and Dr. Dimitris Metaxas (Computational Biomedicine Imaging and Modeling Center) at Rutgers University. The project has four specific aims: (1) Create an OCR system capable of monitoring facial displays of specific emotions (i.e. angry, happy and sad). (2) Improve our current OCR system's ability to detect facial expressions of high versus low performance-induced stress. (3) Develop OCR algorithms to identify fatigue due to sleep loss based on slow eyelid closures (PERCLOS). (4) Test the technical feasibility of data acquisition and reliability of the advanced OCR system in spaceflight analogs that contain neurobehavioral stressors relevant to spaceflight. The project has primary relevance to strategic goals of the NSBRI Neurobehavioral and Psychosocial Factors (NBPF) Team. Two major laboratory experiments for OCR development and validation are underway. The project is 75% complete. To date, half of the total number of subjects required to complete the two experiments have been studied.

 

Rationale for HRP Directed Research:

 

Research Impact/Earth Benefits: The study focuses on the ability of an unobtrusive, automated optical technology to detect psychological stress, emotion and fatigue during operational performance. The knowledge gained has the potential to identify an objective, unobtrusive, automated method for the recognition, monitoring, and management of the risks of neurobehavioral dysfunction in affect and alertness in space flight and in many Earth-based safety-sensitive occupations, such as transportation workers (e.g., truck drivers, train conductors, airline pilots); operators in safety-sensitive industries (e.g., power plant control rooms); and military personnel.

 

Task Progress: In the third year of the current project we have continued to expand the OCR algorithm to recognize facial expressions of emotion and behavioral indicators of excessive sleepiness (through slow eyelid closures). We are also continuing our work to improve the system's ability to correctly identify stress. Preliminary data confirm that the experimental procedures reliably induce stress, emotion and fatigue. During the third year, we have continued collecting data for the two experiments we proposed (one on emotion recognition and one on stress and fatigue detection). 56 healthy subjects have completed the two experiments (n=28 in Experiment 1 and n=28 in Experiment 2). We are using these data to expand and improve the current OCR algorithm. Together with data acquired in year 1 and year 2, a total of n=56 subjects have been studied to date (i.e., n=28 in each of the 2 experiments).

In Experiment 1 (emotion recognition), a total of 31.2 hours of footage for facial emotional analysis has been collected, with 8.5 hours collected during year 3 alone. Subjective emotional questionnaires also were administered to all subjects. The 31.2 hours of footage (n=28, 14 females) has been subjectively scored by humans, who were blinded to the emotion induction tasks. Scorers rated each video based on the durations of emotional states, and which emotion was predominantly expressed. These results will be compared to the scores obtained from the model-based tracker algorithm and will assess and validate the accuracy of the computer-based system.

In Experiment 2 (stress and fatigue detection), a total of 166.7 hours of digitally recorded high definition footage was collected capturing the faces of subjects during performance of the Psychomotor Vigilance Task (PVT), with 49.3 hours collected during year 3 alone (In the 2009-2010 report, we incorrectly reported that we had a total of 180 hours of PVT footage). Human scorers manually scored a subset (n=16) of the 20-minute videos collected during the PVT. The state of the eyelid was scored frame-by-frame (4 states: open, closing, closed, opening) and then compared to the algorithm output. Facelab data and PVT reaction times were simultaneously recorded throughout these test bouts, which were administered every 2 hours over the course of two consecutive days. During year 3, 4.5 hours of footage was recorded for facial emotional analysis (14.1 hours total) and 25 hours of footage also was collected for stress analysis (88.7 hours total) during this protocol. In addition, data from neuropsychological tasks, personality questionnaires, and subjective emotional rating scales were obtained from all subjects.

With regard to the stress-related hormone analysis in Experiment 2, a total of 140 saliva samples were collected during stress-inducing tasks (5 saliva samples per subject), with 40 samples collected during year 3 alone. EEG/EKG data were also collected during these stress-inducing tasks.

 

Bibliography Type: Description: (Last Updated: 07/03/2019) Show Cumulative Bibliography Listing
 
Abstracts for Journals and Proceedings Jones CW, Basner M, Goel N, Metaxas D, Dinges DF. "The Validity of Unobtrusive Tracking of Slow Eyelid Closures as a Measure of Space Flight Fatigue." 18th IAA Humans in Space Symposium, Houston, TX, April 11-15, 2011.

18th IAA Humans in Space Symposium, Houston, TX, April 11-15, 2011. , Apr-2011

Abstracts for Journals and Proceedings Dinges DF, Metaxas D, Banks S, Goel N, Ecker A, Basner M. "Overview of NSBRI neurobehavioral and psychosocial projects involving detection and mitigation stress, fatigue and conflict in space." NASA Human Research Program Investigators' Workshop, Houston, TX, February 3-5, 2010.

NASA Human Research Program Investigators' Workshop, Houston, TX, Abstract Book, February 2010. , Feb-2010

Abstracts for Journals and Proceedings Yang F, Michael N, Metaxas D, Dinges DF. "Development of Optical Computer Recognition (OCR) for Monitoring Fatigue in Space." 18th IAA Humans in Space Symposium, Houston, TX, April 11-15, 2011.

18th IAA Humans in Space Symposium, Houston, TX, April 11-15, 2011. , Apr-2011

Abstracts for Journals and Proceedings Minkel JD, Moreta M, Muto J, Jones C, Goel N, Dinges DF. "Behavioral Risks in Space Flight: Impact of Stressors When Sleep Deprived." 18th IAA Humans in Space Symposium, Houston, TX, April 11-15, 2011.

18th IAA Humans in Space Symposium, Houston, TX, April 11-15, 2011. , Apr-2011

Abstracts for Journals and Proceedings Minkel JD, Moreta MC, Muto JA, Jones CW, Goel N, Dinges DF. "Elevated Stress Responses Following Sleep Deprivation in Healthy Adults." 25th Annual Meeting of the Associated Professional Sleep Societies, LLC 2011, Minneapolis, MN, June 11-15, 2011.

Sleep 2011;34 Suppl:A104-5. , Apr-2011

Abstracts for Journals and Proceedings Moreta MC, Muto J, Minkel JD, Di Antonio AR, Htaik O, Metaxas DN, Dinges DF. "Monitoring Emotions in Space by Tracking Facial Expressions." 18th IAA Humans in Space Symposium, Houston, TX, April 11-15, 2011.

18th IAA Humans in Space Symposium, Houston, TX, April 11-15, 2011. , Apr-2011

Abstracts for Journals and Proceedings Michael N, Yang F, Metaxas D, Dinges DF. "Development of Optical Computer Recognition (OCR) for Monitoring Stress and Emotions in Space." 18th IAA Humans in Space Symposium, Houston, TX, April 11-15, 2011.

18th IAA Humans in Space Symposium, Houston, TX, April 11-15, 2011. , Apr-2011

Articles in Peer-reviewed Journals Minkel J, Htaik O, Banks S, Dinges D. "Emotional expressiveness in sleep-deprived healthy adults." Behav Sleep Med. 2011 Jan;9(1):5-14. http://dx.doi.org/10.1080/15402002.2011.533987 ; PMID: 21218289 , Jan-2011
Download in PDF pdf     
Fiscal Year: FY 2010  Task Last Updated:  05/21/2010 
PI Name: Dinges, David F. Ph.D. 
Project Title: Optical Computer Recognition of Stress, Affect and Fatigue during Performance in Spaceflight 
   
Division Name: Human Research 
Program/Discipline--
Element/Subdiscipline:
NSBRI--Neurobehavioral and Psychosocial Factors Team 
 
Joint Agency Name:   TechPort:  Yes 
Human Research Program Elements: (1) BHP:Behavioral Health & Performance
Human Research Program Risks: (1) Bmed:Risk of Adverse Behavioral Conditions and Psychiatric Disorders
Human Research Program Gaps: (1) BMed02:We need to identify and validate measures to monitor behavioral health and performance during exploration class missions to determine acceptable thresholds for these measures (IRP Rev F)
Space Biology Element: None
Space Biology Cross-Element Discipline: None
Space Biology Special Category: None
PI Email: dinges@pennmedicine.upenn.edu  Fax:  215-573-6410 
PI Organization Type: UNIVERSITY  Phone: 215-898-9949  
Organization Name: University of Pennsylvania 
PI Address 1: Department of Psychiatry 
PI Address 2: 423 Service Dr, 1013 Blockley Hall 
PI Web Page:  
City: Philadelphia  State: PA 
Zip Code: 19104-4209  Congressional District: 
Comments:  
Project Type: GROUND  Solicitation:  2007 Crew Health NNJ07ZSA002N 
Start Date: 05/01/2008  End Date:  04/30/2012 
No. of Post Docs: No. of PhD Degrees: 
No. of PhD Candidates: No. of Master' Degrees: 
No. of Master's Candidates: No. of Bachelor's Degrees:  21 
No. of Bachelor's Candidates: 24  Monitoring Center:  NSBRI 
Contact Monitor:   Contact Phone:   
Contact Email:  
Flight Program:  
Flight Assignment:

 

Key Personnel Changes/Previous PI:  
COI Name (Institution): Metaxas, Dimitris   ( Rutgers University )
Goel, Namni   ( University of Pennsylvania ) 
Grant/Contract No.: NCC 9-58-NBPF01601 
Performance Goal No.:  
Performance Goal Text:

 

Task Description: Astronauts must maintain high-level performance while experiencing demanding workload and work schedules, extreme environmental risks, and psychosocial stressors in space (e.g., isolation, confinement). Stress, negative emotions and fatigue can jeopardize their cognitive performance and neurobehavioral status. The proposed research is developing and validating an objective, unobtrusive, computational model-based tracker of the human face that reliably identifies when astronauts are experiencing stress, emotion and fatigue at levels that compromise performance in space. This optical computer recognition (OCR) system will provide feedback to them for autonomous selection of countermeasures for stress, depression and fatigue. The project is being accomplished through collaborative efforts of Dr. David Dinges (Unit for Experimental Psychiatry) at the University of Pennsylvania School of Medicine, and Dr. Dimitris Metaxas (Computational Biomedicine Imaging and Modeling Center) at Rutgers University. The project has four specific aims: (1) Create an OCR system capable of monitoring facial displays of specific emotions (i.e. angry, happy and sad). (2) Improve our current OCR system's ability to detect facial expressions of high versus low performance-induced stress. (3) Develop OCR algorithms to identify fatigue due to sleep loss based on slow eyelid closures (PERCLOS). (4) Test the technical feasibility of data acquisition and reliability of the advanced OCR system in spaceflight analogs that contain neurobehavioral stressors relevant to spaceflight (e.g., NEEMO). The project has primary relevance to strategic goals of the NSBRI Neurobehavioral and Psychosocial Factors (NBPF) Team. Two major laboratory experiments for OCR development and validation are underway. The project is 50% complete. To date, half of the total number of subjects required to complete the two experiments have been studied.

 

Rationale for HRP Directed Research:

 

Research Impact/Earth Benefits: The study focuses on the ability of an unobtrusive, automated optical technology to detect psychological stress, emotion and fatigue during operational performance. The knowledge gained has the potential to identify an objective, unobtrusive, automated method for the recognition, monitoring, and management of the risks of neurobehavioral dysfunction in affect and alertness in space flight and in many Earth-based safety-sensitive occupations, such as transportation workers (e.g., truck drivers, train conductors, airline pilots); operators in safety-sensitive industries (e.g., power plant control rooms); and military personnel.

 

Task Progress: In the second year of the current project we have continued to expand the OCR algorithm to recognize facial expressions of emotion and behavioral indicators of excessive sleepiness (through slow eyelid closures). We are also continuing our work to improve the system's ability to correctly identify stress. Preliminary data confirm that the experimental procedures reliably induce stress, emotion and fatigue. During the second year, we have continued collecting data for the two experiments we proposed (one on emotion recognition and one on stress and fatigue detection). Twenty healthy subjects have completed the two experiments (n=11 in Experiment 1 and n=9 in Experiment 2). We are using these data to expand and improve the current OCR algorithm. Together with data acquired in year 1 on n=20 other subjects, a total of n=40 subjects have been studied to date (i.e., n=20 in each of the 2 experiments).

In Experiment 1 (emotion recognition), 15 hours of footage for facial emotional analysis was collected. Subjective emotional questionnaires also were administered to all subjects.

In Experiment 2 (stress and fatigue detection), 180 hours of digitally recorded high definition footage was collected capturing the faces of subjects during performance of the Psychomotor Vigilance Task (PVT). Facelab data and PVT reaction times were simultaneously recorded throughout these test bouts, which were administered every 2 hours over the course of two consecutive days. 3 hours of footage was recorded for facial emotional analysis and 20 hours of footage also was collected for stress analysis. In addition, data from neuropsychological tasks, personality questionnaires, and subjective emotional rating scales were obtained from all subjects.

With regard to the stress-related hormone analysis in Experiment 2, a total of 45 saliva samples were collected during stress-inducing tasks (5 saliva samples per subject). EEG/EKG data were also collected during these stress-inducing tasks.

 

Bibliography Type: Description: (Last Updated: 07/03/2019) Show Cumulative Bibliography Listing
 
Abstracts for Journals and Proceedings Dinges DF, Metaxas D, Minkel JD, Lee C, Caruso H, Banks S, McGlinchey EL. "Optical computer recognition of the face to monitor stress, emotion and fatigue in space flight." 80th Annual Aerospace Medical Association Scientific Meeting, Los Angeles, CA, May 4, 2009.

Aviation, Space, and Environmental Medicine. 2009 Mar;80(3):224. , Mar-2009

Abstracts for Journals and Proceedings Dinges DF, Metaxas DN, Minkel JD, Banks S, Michael N. "Optical computer recognition of behavioral stress in space flight." NASA Human Research Program Investigators' Workshop, League City, TX, February 3, 2009.

NASA Human Research Program Investigators' Workshop, Abstract Book, February 2009. , Feb-2009

Articles in Peer-reviewed Journals Lim J, Wu WC, Wang J, Detre JA, Dinges DF, Rao H. "Imaging brain fatigue from sustained mental workload: an ASL perfusion study of the time-on-task effect." Neuroimage. 2010 Feb 15;49(4):3426-35. http://dx.doi.org/10.1016/j.neuroimage.2009.11.020 ; PMID: 19925871 , Feb-2010
Awards Dinges D. "Raymond F. Longacre Award for Outstanding Accomplishment in the Psychological and Psychiatric Aspects of Aerospace Medicine, Aerospace Medical Association, May 2009." May-2009
Books/Book Chapters Rathjen T, Whitmore M, McGuire K, Goel N, Dinges DF, Tvaryanas AP, Zehner G, Hudson J, Dismukes RK, Musson DM. "An introduction to human factors in aerospace." in "Fundamentals of Aerospace Medicine (4th edition)." Ed. J.R. Davis et al. Philadelphia : Lippincott Williams & Wilkins, 2008. p. 491-515., Dec-2008
Dissertations and Theses Minkel J. "Affective consequences of sleep deprivation." Dissertation, University of Pennsylvania, May 2009. , May-2009
Download in PDF pdf     
Fiscal Year: FY 2009  Task Last Updated:  06/05/2009 
PI Name: Dinges, David F. Ph.D. 
Project Title: Optical Computer Recognition of Stress, Affect and Fatigue during Performance in Spaceflight 
   
Division Name: Human Research 
Program/Discipline--
Element/Subdiscipline:
NSBRI--Neurobehavioral and Psychosocial Factors Team 
 
Joint Agency Name:   TechPort:  Yes 
Human Research Program Elements: (1) BHP:Behavioral Health & Performance
Human Research Program Risks: (1) Bmed:Risk of Adverse Behavioral Conditions and Psychiatric Disorders
Human Research Program Gaps: (1) BMed02:We need to identify and validate measures to monitor behavioral health and performance during exploration class missions to determine acceptable thresholds for these measures (IRP Rev F)
Space Biology Element: None
Space Biology Cross-Element Discipline: None
Space Biology Special Category: None
PI Email: dinges@pennmedicine.upenn.edu  Fax:  215-573-6410 
PI Organization Type: UNIVERSITY  Phone: 215-898-9949  
Organization Name: University of Pennsylvania 
PI Address 1: Department of Psychiatry 
PI Address 2: 423 Service Dr, 1013 Blockley Hall 
PI Web Page:  
City: Philadelphia  State: PA 
Zip Code: 19104-4209  Congressional District: 
Comments:  
Project Type: GROUND  Solicitation:  2007 Crew Health NNJ07ZSA002N 
Start Date: 05/01/2008  End Date:  04/30/2012 
No. of Post Docs: No. of PhD Degrees: 
No. of PhD Candidates: No. of Master' Degrees: 
No. of Master's Candidates: No. of Bachelor's Degrees: 
No. of Bachelor's Candidates: 26  Monitoring Center:  NSBRI 
Contact Monitor:   Contact Phone:   
Contact Email:  
Flight Program:  
Flight Assignment:

 

Key Personnel Changes/Previous PI:  
COI Name (Institution): Banks, Siobhan   ( University of Pennsylvania Health System )
Metaxas, Dimitri   ( Rutgers University  ) 
Grant/Contract No.: NCC 9-58-NBPF01601 
Performance Goal No.:  
Performance Goal Text:

 

Task Description: Astronauts must maintain high-level performance while experiencing demanding workload and work schedules, extreme environmental risks, and psychosocial stressors in space (e.g., isolation, confinement). Stress, negative emotions and fatigue can jeopardize their cognitive performance and neurobehavioral status. The proposed research will deliver an objective, unobtrusive, computational model-based tracker of the human face that validly and reliably identifies when astronauts are experiencing stress, emotion and fatigue at levels that compromise performance in space. This optical computer recognition (OCR) system will provide feedback to them for autonomous selection of countermeasures for stress, depression and fatigue. The project will be accomplished through collaborative efforts of Dr. David Dinges (Unit for Experimental Psychiatry) at the University of Pennsylvania School of Medicine, and Dr. Dimitris Metaxas (Computational Biomedicine Imaging and Modeling Center) at Rutgers University.

The project has four specific aims: (1) Create an OCR system capable of monitoring facial displays of specific emotions (i.e. angry, happy and sad). (2) Improve our current OCR system's ability to detect facial expressions of high versus low performance-induced stress. (3) Develop OCR algorithms to identify fatigue due to sleep loss based on slow eyelid closures (PERCLOS). (4) Test the technical feasibility of data acquisition and reliability of the advanced OCR system in spaceflight analogs that contain neurobehavioral stressors relevant to spaceflight (e.g., NEEMO). The project has primary relevance to strategic goals of the NSBRI Neurobehavioral and Psychosocial Factors (NBPF) Team. It addresses a high priority gap identified by the NASA SAT, BHP, and NSBRI NBPF area, and specifically targets questions 25d,c,g,h of Bioastronautics Roadmap Risk Area 25 (Human Performance Failure Due to Neurobehavioral Problems), and question 27d in Risk Area 27 (Human Performance Failure Due to Sleep Loss and Circadian Rhythm Problems).

 

Rationale for HRP Directed Research:

 

Research Impact/Earth Benefits: The study focuses on the ability of an unobtrusive, automated optical technology to detect psychological stress, emotion and fatigue during operational performance. The knowledge gained has the potential to identify an objective, unobtrusive, automated method for the recognition, monitoring, and management of the risks of neurobehavioral dysfunction in affect and alertness in space flight and in many Earth-based safety-sensitive occupations, such as transportation workers (e.g., truck drivers, train conductors, airline pilots); operators in safety-sensitive industries (e.g., power plant control rooms); and military personnel.

 

Task Progress: The overarching goal of the project is to further develop and validate computerized system that unobtrusively detects stress, emotion and fatigue during space flight. We previously completed development of an optical algorithm for real-time dynamic tracking of the face using a deformable model-based tracker and Active Shape Modeling that reliably discriminates low from high stress by monitoring facial expressions. To overcome the limitations of previous optical tracking techniques, Metaxas and colleagues developed a formal framework for the integration of edge detection and optical flow into a deformable model framework and applied it to facial shape and motion estimation. This method used a single camera to track the shape of the face and its movement in 3-dimensional space, and it created a deformable model, incorporating optical flow (an approximation of the motion of objects within a visual representation) into the model as a constraint.

We have made several other new developments to the OCR system: (1) the technique was validated with the use only one camera, where the previous method required two; (2) we improved tracking by using a manifold of faces that helped automatically track the face as the head moves; (3) we added the use of Conditional Random Fields in addition to Hidden Markov Modeling, to the algorithm which improved its computational efficiency; and (4) GABOR filtering (used for edge detection in image analysis) was incorporated into the ASM algorithm to track changes in facial texture, allowing it to identify features (e.g., furrowed brow).

In the first year of the current project we have begun expanding the algorithm to recognize facial expressions of emotion and behavioral indicators of excessive sleepiness (through slow eyelid closures). We are also continuing our work to improve the system's ability to correctly identify stress. Preliminary data confirm that the experimental procedures reliably induce stress, emotion and fatigue. In this first year we designed and implemented the two experiments we proposed (one on emotion detection and one on stress and fatigue detection. Twenty healthy subjects have completed the two experiments (N=9 in Experiment 1 and N=11 in Experiment 2). We are using these data to expand and improve the current OCR algorithm.

 

Bibliography Type: Description: (Last Updated: 07/03/2019) Show Cumulative Bibliography Listing
 
Download in PDF pdf     
Fiscal Year: FY 2008  Task Last Updated:  06/02/2008 
PI Name: Dinges, David F. Ph.D. 
Project Title: Optical Computer Recognition of Stress, Affect and Fatigue during Performance in Spaceflight 
   
Division Name: Human Research 
Program/Discipline--
Element/Subdiscipline:
NSBRI--Neurobehavioral and Psychosocial Factors Team 
 
Joint Agency Name:   TechPort:  Yes 
Human Research Program Elements: (1) BHP:Behavioral Health & Performance
Human Research Program Risks: (1) Bmed:Risk of Adverse Behavioral Conditions and Psychiatric Disorders
Human Research Program Gaps: (1) BMed02:We need to identify and validate measures to monitor behavioral health and performance during exploration class missions to determine acceptable thresholds for these measures (IRP Rev F)
Space Biology Element: None
Space Biology Cross-Element Discipline: None
Space Biology Special Category: None
PI Email: dinges@pennmedicine.upenn.edu  Fax:  215-573-6410 
PI Organization Type: UNIVERSITY  Phone: 215-898-9949  
Organization Name: University of Pennsylvania 
PI Address 1: Department of Psychiatry 
PI Address 2: 423 Service Dr, 1013 Blockley Hall 
PI Web Page:  
City: Philadelphia  State: PA 
Zip Code: 19104-4209  Congressional District: 
Comments:  
Project Type: GROUND  Solicitation:  2007 Crew Health NNJ07ZSA002N 
Start Date: 05/01/2008  End Date:  04/30/2012 
No. of Post Docs:   No. of PhD Degrees:   
No. of PhD Candidates:   No. of Master' Degrees:   
No. of Master's Candidates:   No. of Bachelor's Degrees:   
No. of Bachelor's Candidates:   Monitoring Center:  NSBRI 
Contact Monitor:   Contact Phone:   
Contact Email:  
Flight Program:  
Flight Assignment:

 

Key Personnel Changes/Previous PI:  
COI Name (Institution): Banks, Siobhan    ( University of Pennsylvania School of Medicine )
Metaxas, Dimitri    ( Rutgers University ) 
Grant/Contract No.: NCC 9-58-NBPF01601 
Performance Goal No.:  
Performance Goal Text:

 

Task Description: Astronauts must maintain high-level performance while experiencing demanding workload and work schedules, extreme environmental risks, and psychosocial stressors in space (for example, isolation and confinement). Stress, negative emotion and fatigue can jeopardize their cognitive performance and neurobehavioral status. The project will deliver an objective, unobtrusive, computational model-based tracker of the human face that validly and reliably identifies when astronauts are experiencing stress, emotion and fatigue at levels that compromise performance in space.

This optical computer recognition (OCR) system will provide feedback to them for autonomous selection of countermeasures for stress, depression and fatigue. The project will be accomplished through collaborative efforts of Dr. David Dinges (Unit for Experimental Psychiatry) at the University of Pennsylvania School of Medicine, and Dr. Dimitris Metaxas (Computational Biomedicine Imaging and Modeling Center) at Rutgers University.

Specific Aims

1) Create an OCR system capable of monitoring facial displays of specific emotions (i.e., angry, happy and sad).

2) Improve our current OCR systems ability to detect facial expressions of high-performance versus low-performance-induced stress.

3) Develop OCR algorithms to identify fatigue due to sleep loss based on slow eyelid closures.

4) Test the technical feasibility of data acquisition and reliability of the advanced OCR system in spaceflight analogs, such as NEEMO, that contain neurobehavioral stressors relevant to spaceflight.

The project has primary relevance to strategic goals of the NSBRI Neurobehavioral and Psychosocial Factors (NBPF) Team. It addresses a high-priority gap identified by the NASA Small Assessment Team, Behavioral Health and Performance, and NSBRI NBPF Team areas. and the project specifically targets questions 25d, c, f, and h of Bioastronautics Roadmap Risk Area 25 (Human Performance Failure Due to Neurobehavioral Problems), and question 27d in Risk Area 27 (Human Performance Failure Due to Sleep Loss and Circadian Rhythm Problems).

 

Rationale for HRP Directed Research:

 

Research Impact/Earth Benefits: 0

 

Task Progress: New project for FY2008.

 

Bibliography Type: Description: (Last Updated: 07/03/2019) Show Cumulative Bibliography Listing