Workshop on

I will discuss three mysteries at the heart of the current vision agenda: What do image generators "know" and not "know"? What should be represented explicitly? How can we tell how well an image generator works?

I will show strong evidence that depth, normal and albedo can be extracted from two kinds of image generator, with minimal inconvenience or training data. This suggests that image generators might "know" more about scene appearance than we realize. I will show that there are important scene properties that image generators very reliably get wrong. These include shadow geometry and perspective geometry. Similarly, video generators get object constancy and properties like momentum conservation wrong.

If generators know intrinsics, why extract them from images? Perhaps because some other downstream process -- rendering, movement, whatever -- will use them. But rendering seems to work much better if you use latents, and numerous systems move using entirely latent representations. The only good argument in favor of an exposed image representation is that it might be easy to interact or compute with. This suggests paying more attention to representations that suppress detail in favor of capturing major effects in a compact form. I will show a method capable of computing such representations very efficiently for vast numbers of images.

Fixing image generators requires an evaluation procedure that will tell you if the generator got better. Image generators are evaluated using entirely irrational procedures (doesn't this look good; FID; and so on). But, looked at closely, most image generators and conditional image generators don't work very well. Part of the problem is we simply don't know how to do evaluate them in a sensible way. I will discuss some progress in this respect.

Diffusion models excel in solving imaging inverse problems due to their exceptional ability to model complex image priors. When integrated with image formation models, they enable a physics-guided diffusion process—termed diffusion posterior sampling (DPS)—that effectively retrieves high-dimensional posterior distributions of images from corrupted observations. However, DPS’s dependence on clean diffusion priors and accurate imaging physics limits its practical utility, especially when clean data is scarce for pre-training diffusion models or when the underlying physical models are inaccurate for inference. However, DPS requires clean diffusion priors and accurate imaging physics for inference, limiting its practical use when clean data is scarce for pre-training diffusion models or when the underlying physical models are inaccurate or unknown. In this talk, Prof. Sun will introduce an expectation-maximization (EM) framework that adapts DPS for scenarios with inaccurate priors or physics. This method alternates between an E-step, in which clean images are reconstructed from corrupted data assuming known diffusion prior and physical model, and an M-step, where these reconstructions are employed to refine and recalibrate the prior or physical model. This iterative process progressively aligns the learned image prior with the true clean distribution and adapts the physical model for enhanced accuracy. They validate their approach through extensive experiments across diverse computational imaging tasks—including inpainting, deblurring, denoising, and realistic microscopic imaging—demonstrating new state-of-the-art performance.

Humans navigate a rich 3D world, continuously acquiring diverse skills and engaging in various activities through perception, understanding, and interaction. The long-term research goal of Prof. Qi's group is to simulate the dynamic 3D world and equip AI systems with 3D spatial understanding. A fundamental challenge in achieving this lies in how to effectively represent 3D structures. In this talk, Prof. Qi will present a series of research efforts from her group focused on learning 3D representations from videos for surface reconstruction, novel view synthesis, and 4D modeling of dynamic scenes, with applications in video processing. She will also discuss future directions toward generating editable 4D worlds from casually captured videos.

AI algorithms for computer-based visual understanding have advanced significantly, due to the prevalence of deep learning and large-scale visual datasets in the RGB domain, which have also been proven vulnerable to digital and physical adversarial attacks. To deal with complex scenarios, many other imaging modalities beyond the visibility scope of human eyes, such as near infrared (NIR), short wavelength infrared (SWIR), thermal infrared (TIR), polarization, neuromorphic pulse, have been introduced, yet the vulnerabilities of visual AI based on these non-RGB modalities have not received due attention. In this talk, Prof. Zheng will show that typical AI algorithms, like object detection and segmentation, can be more fragile than in the RGB domain, and the properly crafted attackers can hardly be noticed by naked human eyes. They showcase a serial of physics-based attackers in the NIR, SWIR, and TIR domain, a printable attacker for event-based human detection, and a projection-based attacker on polarization-based reconstruction and segmentation.

High-quality, efficient acquisition of complex appearance and geometry is a classic problem in computer graphics and vision. Differentiable acquisition maps both physical acquisition and computational reconstruction to a differentiable pipeline, enabling a fully automatic, joint optimization of hardware and software. This significantly improves modeling efficiency and quality over existing work. In this talk, Prof. Wu will briefly introduce his recent research on differential acquisition, including OpenSVBRDF, the first large-scale measured SVBRDF database (SIGAsia 2023a), the first dynamic visible-light tomography system (SIGAsia 2023b), as well as a system for real-time acquisition and reconstruction of dynamic 3D volumes (CVPR 2024). He will also describe on-going research on building a dedicated high-performance device for acquiring the appearance of cultural relics in the National Museum of China.

Computational photography is widely used in our daily lives, from the convenience of smartphone photography, image enhancement, and short video filters to more advanced applications like autonomous driving, medical imaging, and scientific observation. With the rapid development of generative models and neural rendering fields, image generation has made significant strides, raising a challenging question: can traditional computational photography be fully replaced by large models?

In this talk, we will explore how to combine traditional computational photography techniques with foundational and generative models. Specifically, we will address the following four aspects: first, how to use generative models to enhance existing image processing, such as UltraFusion HDR, where we will demonstrate achieving high dynamic range fusion of up to 9 stops for the first time using generative models; second, how to leverage generative model priors to improve the imaging quality of novel camera systems (e.g., lensless cameras and event cameras); third, how foundational models can automate camera system development, including automated evaluation (DepictQA) and automated debugging (RLPixTuner); and finally, the application of large models in 3D reconstruction and video frame interpolation.

Conventional cameras can only acquire light intensity in two dimensions. In order to further obtain the depth, polarization, and spectral information of the target object or to achieve imaging resolution beyond the diffraction limit, it is often required to use bulky and expensive instruments. Here, Prof. Yang will present his group’s recent effort to replace conventional camera lenses with flat diffractive optical elements or metasurfaces. By leveraging the unique capability of metasurface to tailor the vectorial field of light, in combination with an advanced image retrieval algorithm, they aim to build compact camera systems that can capture multi-dimensional light field information of a target scene in a single shot under ambient illumination conditions. Specifically, he will show how they use flat-optics to build a monocular camera that can capture a 4D image, including 2D all-in-focus intensity, depth, and polarization of a target scene in a single shot over extended scenes. He would also like to present their effort to construct flat-optics-based monocular 3D camera modules for real-world applications.

Simulating rain in a realistic and physically accurate manner has long been a challenge—especially in complex, uncontrolled outdoor environments. Traditional physics-based simulations produce high-quality rain effects like splashes and droplets, but require meticulous scene setup and lack scalability. On the other hand, recent advances in 3D reconstruction, such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), enable flexible scene modeling and novel view synthesis, yet fall short when it comes to dynamic scene editing like rain simulation.

In this talk, I will present RainyGS, a novel method that bridges this gap by combining physically-based rain simulation with the efficiency and flexibility of the 3DGS framework. RainyGS allows for real-time, photorealistic rendering of rain effects—from light drizzles to heavy storms—on in-the-wild scenes, all without manual scene setup. By integrating shallow water dynamics, raindrop motion, and realistic reflections into 3DGS, our method achieves over 30 fps rendering and supports diverse open-world scenarios, including urban driving scenes. I'll showcase how RainyGS opens up new possibilities for dynamic environmental simulation in computer vision, virtual reality, and autonomous driving research.

3D Gaussian Splatting (GS) excels in real-time novel view synthesis but faces challenges in geometric reconstruction fidelity and volumetric rendering accuracy. This talk explores two advancements addressing these limitations: RaDe-GS and Volumetric Gaussian Splatting. RaDe-GS introduces rasterized depth and surface normal rendering for unstructured 3D Gaussians, optimizing shape reconstruction through depth consistency to achieve state-of-the-art geometric accuracy (e.g., NeuraLangelo-level Chamfer distance on DTU) while retaining the original framework’s computational efficiency. Complementing this, Volumetric Gaussian Splatting redefines GS primitives as stochastic solids, enabling volumetric rendering with an attenuation function that eliminates visual artifacts (e.g., popping effects) and ensures continuity in parameter optimization. This approach precisely models light interactions in scattering media and overlapping primitives, outperforming existing methods in both generic and volumetric scenes. Together, these works advance GS’s capabilities—RaDe-GS bridges its rendering quality with precise 3D reconstruction, while Volumetric GS unlocks artifact-free, physically grounded rendering for complex media—demonstrating GS’s potential as a unified framework for high-fidelity 3D scene modeling.

In this talk, Prof. Baek will discuss his recent work on designing, controlling, and utilizing computational imaging systems for inverse rendering, robotic vision in extreme conditions, transparent and metameric material analysis, and advanced VR/AR systems. The core idea is to take a holistic approach—considering rendering, the transformation of multi-dimensional light properties as they travel from the source through a scene, their interaction with imaging systems, and the computational processes for reconstruction and restoration.