Workshop on

Brian A. Wandell

The number and type of imaging systems has grown enormously over the last several decades; these systems are an essential component in mobile communication, medicine, automotive and drone applications. Imaging systems are also increasingly used with deep learning systems that require large amounts of training data. For these reasons software prototyping has become an essential tool for the design, evaluation and training of modern image systems. I will describe three closely related open-source and freely available image systems toolboxes - ISETCam, ISETBio, and ISET3d - that support design and evaluation of whole image systems. The presentation will include examples of how we model the three-dimensional scene spectral radiance, retinal encoding (physiological optics and cone sampling), and image systems hardware (multi-element lenses, image sensors). I will also provide two examples of how we used end-to-end image system simulation to aid understanding and design.

1. ISETAuto: Detecting vehicles with depth and radiance information (2021) . Zhenyi Liu, Joyece Farrell, Brian Wandell, IEEE Access ACCESS.2021.3063692
2. A system for generating complex physically accurate sensor images for automotive applications (2019). Zhenyi Liu, Minghao Shen, Jiaqi Zhang, Shuangting Liu, Henryk Blasinski, Trisha Lian, Brian Wandell. IS&T Electronic Imaging Conference, San Francisco.
3. Ray tracing 3D spectral scenes through human optics models (2019). Trish Lian, Kevin McKenzie, David Brainard, Nicolas Cottaris, Brian Wandell. Journal of Vision October 2019, Vol.19, 23. doi:https://doi.org/10.1167/19.12.23
4. A computational observer model of spatial contrast sensitivity: Effects of photocurrent encoding, fixational eye movements and inference engine (2020). Nicolas P. Cottaris, Brian A. Wandell, Fred Rieke, David H. Brainard Journal of Vision doi:https://doi.org/10.1167/jov.20.7.17

See the wiki pages of the repositories at: https://github.com/ISET

Min H. Kim

Existing time-resolved non-line-of-sight (NLOS) imaging methods reconstruct hidden scenes by inverting the optical paths of indirect illumination measured at visible relay surfaces. These methods are prone to reconstruction artifacts due to inversion ambiguities and capture noise, which are typically mitigated through the manual selection of filtering functions and parameters. This talk introduces a fully-differentiable end-to-end NLOS inverse rendering pipeline that self-calibrates the imaging parameters during the reconstruction of hidden scenes, using as input only the measured illumination while working both in the time and frequency domains. The proposed pipeline extracts a geometric representation of the hidden scene from NLOS volumetric intensities and estimates the time-resolved illumination at the relay wall produced by such geometric information using differentiable transient rendering. It then uses gradient descent to optimize imaging parameters by minimizing the error between our simulated time-resolved illumination and the measured illumination. This end-to-end differentiable pipeline couples diffraction-based volumetric NLOS reconstruction with path-space light transport and a simple ray marching technique to extract detailed, dense sets of surface points and normals of hidden scenes. This talk demonstrates the robustness of the proposed method to consistently reconstruct geometry and albedo, even under significant noise levels.

Boxin Shi

Compared with conventional frame-based cameras, the event camera has unique advantages, especially in their ability to perceive high-speed moving objects and scenes with high dynamic range. Existing research has fully demonstrated the advantages of event cameras in computer vision tasks such as image deblurring, high dynamic range imaging, and high-speed object detection and recognition. However, the photometric image formation model of event cameras has not been fully analyzed. This talk will share a series of research progress on modeling and analyzing the photometric image formation model of an event camera that records and responses to high-speed radiance changes: obtaining high-fidelity scene radiance estimation via analyzing the transient event frequency, constructing real-time photometric stereo with fast-moving light sources to estimate surface normals, and conducting direct-global illumination separation by capturing shadows of a line occlude swiftly sweeping over the scene.

Kede Ma

Contemporary no-reference image quality assessment (NR-IQA) models can effectively quantify the perceived image quality, with high correlations between model predictions and human perceptual scores on fixed test sets. However, little progress has been made in comparing NR-IQA models from a perceptual optimization perspective. However, little progress has been made in leveraging NR-IQA models as natural image priors for use in real-world image enhancement. Here, for the first time, we demonstrate that NR-IQA models can be plugged into the maximum a posteriori (MAP) estimation framework for image enhancement. This is achieved by taking the gradients differentiable and bijective diffusion latents rather than in the raw pixel domain. Different NR-IQA models are likely to induce different enhanced images, which are ultimately subject to psychophysical testing. This leads to a new computational method for comparing NR-IQA models within the analysis-by-synthesis framework. Compared to conventional correlation-based metrics, our method provides complementary insights into the relative strengths and weaknesses of the competing NR-IQA models in the context of perceptual optimization. Taking a step further, we finetune the best-performing NR-IQA model on the combination of previously trained datasets and newly MAP-enhanced images, showcasing the value of MAP-enhanced images for NR-IQA model rectification.

Nicholas X. Fang

In recent years, remarkable progress has been made in the fields of spatial wave modulation and nanomaterial fabrication, unlocking numerous possibilities for engineering wave-matter interactions. Through systematic exploration of the design space, promising applications have been found in opto-mechanics, nanophotonics, and 3D imaging. Consequently, large-scale computational optimization has become an invaluable tool for discovering novel optical structures. This raises the question: can we harness the power of inverse design to develop acoustic metamaterials with tailored wave-filtering properties for next-generation ultrasound imaging devices?

This talk presents the recent exploration of metasurfaces for acoustic waves enabled by advanced design and micro/nanofabrication techniques. An example is the recent effort of ultrasonic metalenses over broadband frequencies, ranging from 38 kHz to 0.5 MHz. Through simulations and experimental validation, we show that ultrasonic metalens offer remarkable advantages in intensity transmission and enhancement in spatial resolution. We will also discuss the potential impact of user defined metamaterials for emerging applications such as underwater acoustic navigation, non-destructive evaluation, and high-resolution medical ultrasound imaging.