|
Task Progress:
|
To achieve the two research aims of this project, we initiated an effort to design a framework to generate mappings between the ocular structure and its function, by developing a computational framework inspired by deep Convolutional Neural Networks (CNNs). We then initiated these novel mappings, trained during research task (RT-1.1). Unlike current classification and segmentation algorithms that merely label test results, the proposed mappings are able to directly connect one domain (function) to the other (physiology). This functionality is significant, as it will enable us to predict progression of changes by cross-validating test results from one domain (function) with the other (structure). Moreover, these deep network models enable the design of cohort studies as a part of research task (RT-1.2) in order to uncover model similarities and differences between Spaceflight Associated Neuro-ocular Syndrome (SANS) and its terrestrial analogs. Below, additional details about the accomplishments in each research task are presented.
1) Mapping Across Domains In order to establish a comprehensive mapping between different ophthalmic domains, we started by designing a conditional generative adversarial network (GAN) to map across the publicly available data we had at our disposal, i.e., fluorescein angiography (FA) and fundus photographs. The GAN comprises a vision-transformer-based generative adversarial network (GAN) consisting of residual, spatial feature fusion, up sampling and down sampling blocks for generators, and transformer encoder blocks for discriminators. We incorporate multiple losses for generating vivid fluorescein angiography images from normal and abnormal fundus photographs for training. Multi-scale Generators: To capture large and fine-scale features to produce realistic vascular images, we combine multi-scale coarse and fine generators. We adopt two generators (fine and coarse). The fine generator synthesizes local features such as arteries and venules. Conversely, the coarse generator translates global features such as large blood vessels, optic disc, and overall contrast and illumination. The generators consist of multiple down sampling, up sampling, spatial feature fusion, residual blocks, and a multi-scale feature summation block between the two generators.
Down Sampling and Up Sampling Blocks: We use, as generators, auto-encoders comprising of multiple down sampling and up sampling blocks for feature extraction. A single down sampling block contains a convolution layer, a batch-norm layer, and a Leaky-ReLU activation function successively. In contrast, an up-sampling block consists of a transposed convolution layer, batch-norm, and Leaky-ReLU activation layer consecutively. We use the down sampling block twice in the fine generator, followed by nine successive residual identity blocks. Finally, the up-sampling blocks are used again to make the spatial output the same as the input. For the coarse generator, we utilize the down sampling once, and after three consecutive residual blocks, a single up sampling block is employed to get the same spatial output as the input. Spatial Feature Fusion Block: The spatial feature fusion (SFF) block consists of two residual units with Convolution, Batch-Norm, Leaky-ReLU layers successively. There are two skip connections, one going from the input and element-wise platform, summed to the first residual unit’s output, and one coming from the input layer and added with the last residual unit’s output. We use spatial feature fusion blocks for combining spatial features from the bottom layers with the topmost layers of the architecture. The fine generator comprises two SFF blocks that connect each of the two down sampling blocks with the two up sampling blocks successively. In contrast, the coarse generator has only one SFF block between the single down sampling and up sampling block. The reason behind incorporating the SFF block is to extract and retain spatial information that is otherwise lost due to consecutive down sampling and up sampling. As a result, we can combine these features with the learned features of the later layers of the network to get an accurate approximation.
VisionTransformers as Discriminators: GAN discriminators require adapting to local and global information changes for differentiating real and fake images. To alleviate this inherent problem, we need a heavy architecture with many parameters. In contrast, convolution with a large receptive field can be employed for obtaining multi-scale features, but can cause overfitting on training data. To resolve this problem, we propose a new Vision Transformer-based Markovian discriminator. We use eight Vision Transformer encoders, consisting of a multi-headed attention layer and multi-layer perceptron (MLP) block. The Layer Normalization layer precedes each block, and a residual skip connection is added to the output from the input. To handle 2D images of 512 x 512, we reshape the images into a sequence of flattened 2D patches with resolution 64 x 64. By doing so, we end up having 64 patches in total. The Transformer uses a constant latent vector size of 64 through all its layers, so we flatten the patches and map to 64 dimensions with a trainable linear projection. The output of this projection is called the patch embedding. Position embeddings are added to the patch embeddings to preserve positional information. We use regular learnable 1D position embeddings. For multi-headed attention, we use 4 heads. For MLP blocks, we use two dense layers with features sized at 128 x 64, each succeeded by a GeLU activation and a dropout of 0.1. Contrarily, our Vision Transformer has two outputs, an MLP head, and a Convolutional layer. The MLP head has two outputs, with hidden units for FA image classification (Abnormal and Normal). In contrast, the convolution layer outputs a feature map of 64 x 64 for classifying each patch in the original image. We use two Vision Transformer-based discriminators that incorporate identical structures but operate at two different scales. The coarse angiograms and fundus are resized to 256 x 256 by a factor of 2 using the Lanczos filter. Both discriminators have identical transformer encoder and output layers. Consequently, we fuse learnable elements from both generators, while training them with their paired Vision Transformer-based discriminators.
2) Non-Intrusive Diagnostics We present a methodology that comprises a calibration step, four different visual function tests that measure different aspects of user perception, and then a composite pipeline that simulates the modeled deficits for validation.
VR Calibration: In order to properly utilize the virtual assessment, the environment would need to be calibrated at the beginning of each session. Simple calibrations, such as adjusting lens distance, interpupillary distance, and headset adjustments, are done at the start. Additional system specific calibrations, such as color gamut calibration, is done once per each VR device. After these adjustments, the fixation and tracking capabilities of the eyes are tested, first binocularly and then monocularly. These performance metrics are saved alongside the user demography information.
VR Assessment: After the calibration phase, the user’s visual assessment can commence. Visual acuity (VA), contrast sensitivity (CS), and visual distortions are assessed through a variety of procedures. For VA, binocular distant VA as well as dynamic VA is measured under mesopic (natural light) conditions. Instead of using images of conventional charts, we render individual characters in front of the user at predetermined distances and scale it based on user response. The results are reported in logMAR scale, among others. The contrast sensitivity is measured using Gabor patches as stimuli. In this test, the user gaze follows a Gabor patch that alters its contrast and spatial frequency based on user performance. At the end, the contrast sensitivity expressed in logCS among other contrast sensitivity units. The Amsler grid test is adapted to VR to measure the perceptual distortions in age-related macular degeneration (AMD) patients. At the start of the exam, the Amsler grid is displayed in front of both eyes. While looking at a fixation point in the grid, if the straight grid lines appear to be distorted the user emulates the metamorphopsia of the deficient eye on the healthy eye. This grid manipulation is modeled as a Gaussian mixture of different scotoma parameters. The results are reported as the image of the altered Amsler grid.
VR Simulation: The collected results for each of the visual assessments are then used to create a simulation of the perception of the user. This pipeline combines results from all of the tests to offer a single visualization. For example, lower visual acuity values would lead to the scene appearing blurry and the existence of scotomas would create distortions in the scene. The saved parameters can be pulled up at any time so that others can experience the perceptual loss measured by all three tests individually and collectively.
2-1) Objectives of Visual Function Assessments Currently, on board the International Space Station (ISS), astronauts undergo many routine functional visual assessments (e.g., visual acuity, Amsler grid test). Contrast sensitivity testing is also available. For optimal monitoring, these visual assessments may benefit from consistent distancing and illumination calibration to reduce the subjectivity of the tests. We achieve these objectives through virtual reality (VR) head-mounted systems. A laptop screen-based test is repurposed for an immersive experience with this technology. Additionally, by delivering all visual function tests using one VR device, it will be possible to make inferences on other tests once a session is recorded. Specifically, for SANS monitoring, it is important to identify any subtle perceptual impact so that countermeasures can be designed. Intelligent delivery of stimuli under various conditions would help identify subtle perceptual loss. Optic disc edema, globe flattening, nerve fiber layer thickening, and choroidal folds are common imaging findings in SANS. While it is important to monitor SANS, frequently repeating these imaging tests would consume a significant portion of mission time. Therefore, quick sessions of different visual function tests are being considered to continually track the different aspects of SANS symptoms. This can be achieved by mapping visual functional data with imaging data using pre-existing astronaut data as well as head-down tilt bed rest, an analog for SANS.
We have conducted several primary tests on this system, including visual acuity, contrast sensitivity, Amsler grid, and visual fields. These assessments can be linked to specific SANS findings that parallel terrestrial ocular relationships, such as contrast sensitivity and retinal nerve fiber layer thickening. In addition, these visual function tests may be able to further characterize any deficiencies in SANS by providing additional visual assessment tests.
|
|
|
Abstracts for Journals and Proceedings
|
Zaman N, Ong J, Tavakkoli A, Zuckerbrod S, Webster M. "Adaptation to prominent monocular metamorphopsia using binocular suppression." Optica Fall Vision Meeting, Virtual, September 20-October 3, 2021. Abstract Issue. 2021 Optica Fall Vision Meeting. J Vis. 2022 Feb 1;22(3):11. https://doi.org/10.1167/jov.22.3.11 , Feb-2022
|
|
|
Abstracts for Journals and Proceedings
|
Ong J, Zaman N, Kamran SA, Waisberg E, Tavakkoli A, Lee AG, Webster M. "A multi-modal visual assessment system for monitoring Spaceflight Associated Neuro-Ocular Syndrome (SANS) during long duration spaceflight." 2021 Optica Fall Vision Meeting, Virtual, September 20-October 3, 2021. Abstracts. 2021 Optica Fall Vision Meeting, Virtual, September 20-October 3, 2021. , Sep-2021
|
|
|
Abstracts for Journals and Proceedings
|
Kamran SA, Hossain KF, Tavakkoli A, Ong J, Zuckerbrod SL. "A generative adversarial deep neural network to translate between ocular imaging modalities while maintaining anatomical fidelity." 2021 Optica Fall Vision Meeting, Virtual, September 20-October 3, 2021. Abstract Issue. 2021 Optica Fall Vision Meeting. J Vis. 2022 Feb 1;22(3):3. https://doi.org/10.1167/jov.22.3.3 , Feb-2022
|
|
|
Articles in Peer-reviewed Journals
|
Waisberg E, Ong J, Zaman N, Kamran SA, Lee AG, Tavakkoli A. "A non-invasive approach to monitor anemia during long-duration spaceflight with retinal fundus images and deep learning." Life Sci Space Res (Amst). 2022 May;33:69-71. https://doi.org/10.1016/j.lssr.2022.04.004 ; PMID: 35491031 , May-2022
|
|
|
Articles in Peer-reviewed Journals
|
Ong J, Tavakkoli A, Strangman G, Zaman N, Kamran SA, Zhang Q, Ivkovic V, Lee AG. "Neuro-ophthalmic imaging and visual assessment technology for spaceflight associated neuro-ocular syndrome (SANS)." Surv Ophthalmol. 2022 Apr 21;S0039-6257(22)00048-0. Review. https://doi.org/10.1016/j.survophthal.2022.04.004 ; PMID: 35461882 , Apr-2022
|
|
|
Articles in Peer-reviewed Journals
|
Ong J, Zaman N, Kamran SA, Waisberg E, Tavakkoli A, Lee AG, Webster M. "A multi-modal visual assessment system for monitoring Spaceflight Associated Neuro-Ocular Syndrome (SANS) during long duration spaceflight." J Vis. 2022 Feb 1;22(3):6. https://doi.org/10.1167/jov.22.3.6 , Feb-2022
|
|
|
Papers from Meeting Proceedings
|
Kamran SA, Hossain KF, Tavakkoli A, Zuckerbrod SL, Baker SA. "Vtgan: Semi-supervised retinal image synthesis and disease prediction using vision transformers." 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Montreal, Canada, October 11-17, 2021. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 2021 Nov 24:3228-3238, http://dx.doi.org/10.1109/ICCVW54120.2021.00362 , Nov-2021
|
|
|
Papers from Meeting Proceedings
|
Kamran SA, Hossain KF, Tavakkoli A, Zuckerbrod SL, Sanders KM, Baker SA. "RV-GAN: Segmenting retinal vascular structure in fundus photographs using a novel multi-scale generative adversarial network." 24th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2021), Virtual, September 27-October 1, 2021. Proceedings of MICCAI 2021. Lecture Notes in Computer Science, vol 12908. Springer, Cham. https://doi.org/10.1007/978-3-030-87237-3_4 , Sep-2021
|
|
|
Papers from Meeting Proceedings
|
Kamran, S. A., Hossain, K. F., Tavakkoli, A., & Zuckerbrod, S. L. "Attention2angiogan: Synthesizing fluorescein angiography from retinal fundus images using generative adversarial networks." 25th International Conference on Pattern Recognition (ICPR), Virtual, January 12-15, 2021. Proceedings of the 25th International Conference on Pattern Recognition (ICPR), 2021 Jan 12:9122-9129. https://doi.ieeecomputersociety.org/10.1109/ICPR48806.2021.9412428 , Jan-2021
|
|
Home
Developed and operated by: NASA Research and Education Support Services