USTC-HKU Joint Research Seminar on

High-precision reconstruction of neuronal connectivity maps at the synaptic level is a key way to understand how the brain works. With the development of serial electron microscopy imaging, large-scale connectomics research has entered the era of petabyte-scale data. However, even a small volume of brain has tens of thousands of densely distributed neurons with complex shapes, and individual neurons can span large areas, making it extremely challenging to fully reconstruct their 3D fine structures from large-scale EM images. In this talk, I will share our work and latest progress on fine 3D reconstruction of neurons and their morphological analysis in large-scale EM images.

Accurate microscale material characterization requires simultaneous sensitivity to chemical composition and structural anisotropy. We present a polarimetric-hyperspectral microscopy framework that co-registers wavelength-resolved imaging with polarization measurements from the same field of view. Hyperspectral data provide chemical fingerprints, while Stokes-derived parameters, including the degree and angle of linear polarization, reveal orientation, birefringence, and microstructural organization. The system was evaluated using polyethylene terephthalate (PET) and polymethyl methacrylate (PMMA) films and nanoplastic fragments. Conventional hyperspectral imaging showed intra-class spectral variability caused by scattering, deformation, and orientation, which led to partial overlap between materials. Polarimetric descriptors captured structural differences that were not visible in spectral contrast alone. Joint analysis of the two modalities reduced morphology-induced spectral ambiguity and enabled clearer separation of PET and PMMA across both films and fragments. This physics-informed approach provides chemically specific and structurally sensitive characterization, offering an interpretable solution for microscale material identification beyond conventional hyperspectral imaging.

Although 3D reconstruction methods based on RGB images have rapidly advanced and matured over recent years, it is still challenging to capture complete scene information and optimize for high-quality 3D/4D models with unimodal visual inputs under unusual environments. To this end, we set out to explore the application of unconventional visual modalities, represented by event streams and polarization images, on the tasks of high-speed dynamic scene reconstruction and reflective object reconstruction & relighting, respectively. For the former task, we have proposed ERF-GS, an event-RGB fused 4DGS methods targeting real-world scenarios, and EventTracer, a path tracing-based, efficient and differentiable event stream rendering framework. For the latter task, we have developed PhyGaP, a physically-grounded Gaussian Splatting pipeline that realizes realistic reconstruction, inverse rendering, and relighting of glossy objects.

Unifying multimodal understanding and generation is a compelling frontier that is beginning to emerge in the medical field. However, the limited existing unified medical models typically treat understanding and generation as disjoint objectives, lacking a meaningful functional synergy. In this work, we identify and address a critical question in unified medical modeling: what form of “understanding” truly benefits generation. We present SynerMedGen, a unified framework built on the proposed principle of generation-aligned understanding, which synergizes understanding objectives with generation tasks via task alignment. SynerMedGen introduces three generation-aligned understanding tasks and a two-stage training strategy that transfers generation-beneficial representations learned during understanding training to medical image synthesis. Remarkably, even with understanding training alone, our SynerMedGen achieves strong zero-shot performance across 22 medical image synthesis tasks and demonstrates robust generalization. When combined with generation training, SynerMedGen consistently outperforms state-of-the-art specialized medical image synthesis models as well as recent unified medical models. We also release SynerMed, a large-scale dataset of 1M paired synthesis samples and 2M understanding instances for studying understanding–generation synergy.

Reconstructing neuronal connectivity at synaptic resolution is essential for understanding how the neural circuit function. Volume electron microscopy now enables connectomic analysis at unprecedented scale, but automated segmentation still struggles with fine neuronal structures. Dendritic spines—the primary sites of excitatory synaptic input—are particularly challenging: their thin necks and highly variable morphologies make them prone to fragmentation or disconnection from parent dendrites, leaving significant gaps in the reconstructed connectivity. In this talk, I will present a geometry-aware learning framework that combines EM image features with local surface geometry to identify spine fragments, and then applies graph neural networks to infer their connectivity to parent dendrites. I will conclude with experimental results on EM volumes from mouse cortex, examining how geometric and topological cues interact in resolving spine–dendrite associations and where current limitations remain for morphologically complex cases.

Segmenting microbial communities from fluorescence microscopy images is challenging due to severe cell overlap, fluorescence crosstalk, annotation noise, and the scarcity of fully labelled mixed-species data. To overcome these obstacles, we introduce BacUNet, a multi-label deep learning framework that employs a strategic shift in training: it first learns fundamental features from single-species data, and then progressively adapts to complex, overlapping communities using synthetically generated mixed-species images, effectively simulating and solving the core challenges of this task. This strategy effectively decouples overlapping cells without requiring manual multi-label annotations. Evaluated on a 11-class bacterial dataset, BacUNet achieved robust performance, successfully correcting noisy labels and outperforming standard training approaches. Our framework offers a practical solution for high-throughput, accurate decoding of complex spatial microbial structures.

Large-scale electron microscopy (EM) datasets enable neural-circuit reconstruction at synaptic resolution, but recovering complete neurons remains challenging because automated segmentation often produces fragmented and over-segmented structures. This talk presents our research on neuron reconstruction and automated tracing from large-scale EM segmentation data, focusing on associating neuronal fragments and mapping multiple segment labels to complete neurons. We explore a multimodal approach to identifying downstream candidate fragments and estimating their connection probabilities by integrating image features with point cloud morphological features. The goal is to reduce manual proofreading and support scalable connectomic reconstruction and analysis.