HKU-KAUST Joint Postgraduate Workshop on

This report presents recent advances in learned optics for RGB-D imaging, focusing on expanding information bandwidth and enhancing optical modulation through hardware–software co-design. By jointly optimizing optical elements and neural reconstruction networks, the proposed systems break the conventional separation between lens design and computational processing. Two main directions are explored: learned stereo imaging and learned wide field-of-view (FoV) imaging. For stereo RGB-D, a rank-2 coded diffractive optical element (DOE) is introduced to encode complementary stereo and focus cues, enabling high-fidelity depth estimation and RGB reconstruction with improved robustness and flexibility. For wide-FoV imaging, off-aperture optical encoding provides additional degrees of freedom and localized wavefront control, overcoming angular trade-offs while maintaining compact system structures. These approaches are validated through differentiable image formation modeling and real-world camera prototypes, demonstrating superior performance in depth accuracy, image quality, and system efficiency. The work highlights the potential of intelligent optical systems for next-generation imaging in applications such as robotics, augmented reality, and computational photography, paving the way toward fully end-to-end learned camera systems.

Lipids are among the most ubiquitous components in living organisms, and changes in lipid levels can indicate a wide range of physiological and pathological conditions, including non-alcoholic fatty liver disease (NAFLD), atherosclerosis, and neurodegenerative disorders. Photoacoustic imaging (PAI) in the 1.7-μm absorption window offers strong lipid contrast, but its quantitative performance has been limited by the capabilities of existing laser sources. We introduce a 1725-nm hybrid optical parametric oscillator emitter (HOPE) tailored for lipid-based PAI, which outperforms existing sources in this wavelength band for photoacoustic applications by virtue of a narrow emission bandwidth, high optical signal-to-noise ratio, and enhanced spectral energy density parameters. By substantially increasing the efficiency of photoacoustic signal generation, the HOPE enables high-contrast lipid imaging that requires no pre-processing before segmentation. This allows us to implement a streamlined, custom segmentation pipeline for quantifying lipid infiltration in human liver tissues imaged with the 1725-nm PAM platform. The resulting steatosis measurements show strong agreement with clinical assessments, underscoring the potential of this approach to complement existing diagnostic procedures through accurate, label-free lipid evaluation. In addition, we also investigate a physics-informed deconvolution framework to mitigate depth-dependent resolution loss caused by optical scattering. This approach can potentially enable significant resolution recovery in the diffuse optical regime and extend the effective imaging depth of our 1725-nm lipid-specific PAM platform, supporting high-fidelity visualisation of lipid-rich tissues across greater depths.

Recent efforts in data-driven spectral reconstruction aim at extracting spectral information from RGB images. Published work reports exceedingly high numerical scores for this reconstruction task, yet real-world performance lags substantially behind. In this paper, we systematically analyze the performance of such methods with three groups of dedicated experiments. First, we evaluate the practical overfitting limitations with respect to current datasets. Second, we reveal fundamental limitations in the ability of RGB to spectral methods to deal with metameric or near-metameric conditions, which have so far gone largely unnoticed due to the insufficiencies of existing datasets. Finally, we analyze the potential for modifying the problem setting to achieve better performance by exploiting optical encoding and metameric augmentation. Our experiments show that the overall performance of existing methods is primarily limited by the dataset issues. Future progress on snapshot spectral imaging will heavily depend on the generation of improved datasets for the design of effective optical encoding strategies.

We present an inverse-designed acoustic metalens for holographic wavefront control, enabled by a holographic iterative angular spectrum algorithm that derives spatially en-coded phase profiles. By unifying the high-frequency eikonal equation with long-wavelength homogenization theory, we establish a frequency-independent, one-to-one correspondence between refractive index distributions and transmission phases—bridging microscale particle design and macroscale wave engineering. Experimentally, the fabricated metalens achieves diffraction-limited acoustic focusing, demonstrating broadband performance with minimal aberration.

High-throughput sensing in manufacturing and optical systems requires the ability to recover fine structural or wavefront details under rapid motion and challenging illumination, where conventional frame-based imaging and classical Shack–Hartmann wavefront sensing often fail due to motion blur, limited dynamic range, and restricted tolerance to large-gradient variations. This seminar introduces a unified approach based on Computational Neuromorphic Imaging (CNI), which leverages the asynchronous, high-dynamic-range nature of event-based sensors. For industrial inspection, CNI enables reliable detection of dynamic surface defects with strong robustness to vibration and rapidly changing conditions. For wavefront analysis, a neuromorphic wave-normal inference framework directly reconstructs high-dimensional wave normals and overcomes the spot-overlapping constraints faced by traditional angle-based sensors, pushing the wavefront sensor dynamic range to Type III. Together, these developments illustrate how CNI provides a coherent foundation for ultrafast optical field inspection beyond the limits of conventional imaging systems.

Differentiable rendering is a key technique for inverse rendering, but existing methods have some limitations: Conventional color derivatives fail to fully capture how scene parameters affect rendering results; traditional loss functions suffer from locality and gradient sparsity, which hinder optimization; and overall pipelines lack robustness in complex scenarios. To solve these problems, this presentation proposes novel path geometry derivatives. These derivatives establish a direct, constraint-driven relationship between scene parameters and light path geometry based on the implicit function theorem, moving beyond over-reliance on color derivatives. Starting with simple paths in rasterization, the method extends to complex paths in path tracing and uses tailored constraints to ensure smooth path contributions and valid derivative computation. It further applies this framework to differentiable photon mapping and combines it with a matching loss function based on optimal transport to achieve global, dense gradients. Experiments show the method outperforms other differentiable rendering method, providing smoother gradients and more robust optimization.

Differentiable optics enables innovative designs using diffractive optical elements, yet a critical gap between simulation and manufacturing limits their practical application. We propose a fabrication-aware design pipeline tailored for direct-write grayscale lithography and nano-imprinting to enable inexpensive, large-area mass production. We introduce a super-resolved neural lithography model that predicts fabricated 3D geometry, enabling fabrication-aware, end-to-end optimization. To address computational challenges, we implement a tensor-parallel framework distributing FFTs across multiple GPUs, facilitating designs up to 32.16×21.44 mm, simulated on grids of up to 128,640 by 85,760 feature points. Our fabricated prototypes demonstrate strong agreement with simulations in holography and PSF engineering. Additionally, we achieve high-quality imaging using a single DOE paired with Wiener filtering. These contributions effectively lower the technical barriers to the real-world deployment of computational diffractive optics.

The work Enhanced Velocity Field Modeling for Gaussian Video Reconstruction improves 3D Gaussian-splatting video reconstruction for dynamic scenes. The authors identify that deformation-field based approaches often overfit in cases of complex motion or scale variation, producing irregular Gaussian trajectories and sub-optimal visual quality. They propose FlowGaussian-VR, which introduces a velocity-field rendering (VFR) pipeline to optimize per-Gaussian velocities under optical-flow supervision, and a flow-assisted adaptive densification (FAD) strategy to increase Gaussian density in dynamic regions. The result is significantly improved temporal coherence, reduced blur on challenging multi-view dynamic datasets, with physically plausible, trackable Gaussian motions. Building on these high-fidelity dynamic reconstructions, EventTracer takes the reconstructed 3D scenes and renders high-temporal-resolution event streams. By combining low-sample-per-pixel path tracing with a lightweight spiking network using BiLIF units and trained via bidirectional EMD loss, EventTracer efficiently generates realistic event sequences that preserve fine spatiotemporal details, more closely matching real event-camera data than prior simulators, while running at roughly 4 minutes per second of 720p video. Together, these works present an end-to-end, physics-informed pipeline: first reconstructing dynamic 3D scenes with motion-aware Gaussian modeling, then producing realistic, high-frequency event data — which enhances dynamic scene modeling, rendering, and event-based vision for VR/AR, robotics, and beyond.

Traditional cameras require complex, heavy lens stacks to ensure corner-to-corner sharpness. Nature, however, found a more efficient solution: the Fovea. The human eye doesn't try to be sharp everywhere at once; it directs high-resolution focus exactly where it's needed. We have engineered a camera system that adopts this biological strategy. Using a Deformable Phase Plate (DPP), our system dynamically performs localized aberration correction. Just as your eye darts around a scene to build a picture, our system captures multiple "foveated" images—sharp at the fixation point, softer at the periphery. By stacking these dynamic glances, we achieve a composite image with full-field clarity, eliminating the need for bulky corrective optics. This offers a glimpse into a future of miniaturized, dynamic optics for VR and tracking applications. To efficiently cover the entire field of view, we propose joint optimization of DPP deformations under imaging budget constraints. Due to the DPP device's non-linear behavior, we introduce a neural network-based control model for improved alignment between simulation-hardware performance. Further applications on extended depth-of-field imaging and object tracking is discussed.

Artificial intelligence-driven materials discovery offers rapid design of novel material compositions, yet synthesis and characterization lag behind. Characterization, in particular, remains bottlenecked by labor-intensive experiments using expert-operated instruments that typically rely on electromagnetic spectroscopy. We introduce SpectroGen, a generative AI model for transmodality spectral generation, designed to accelerate materials characterization. SpectroGen generates high-resolution, high-signal-to-noise ratio spectra with 99% correlation to ground truth and a root-mean-square error of 0.01 a.u. Its performance is driven by two key innovations: (1) a novel distribution-based physical prior and (2) a variational autoencoder (VAE) architecture. The prior simplifies complex structural inputs into interpretable Gaussian or Lorentzian distributions, while the VAE maps them into a physically grounded latent space for accurate spectral transformation. SpectroGen generalizes across spectral domains and promises rapid, accurate spectral predictions, potentially transforming high-throughput discovery in domains such as battery materials, catalysts, superconductors, and pharmaceuticals.

Holographic techniques are increasingly recognized for their potential in the development of display systems for virtual reality (VR) and augmented reality (AR). The generation of realistic 3D visual representations is crucial for facilitating user immersion. However, current computer-generated holography (CGH) methods utilized in these applications are often constrained by limitations in computational efficiency, substantial data processing demands, and limitations in display form factor. This manuscript explores recent holographic display technologies employing system-aware CGH algorithms designed to mitigate these inherent challenges.