MMLab@NTU

StoryMem: Multi-shot Long Video Storytelling with Memory
K. Zhang, L. Jiang, A. Wang, J. Z. Fang, T. Zhi, Q. Yan, H. Kang, X. Lu, X. Pan
Technical report, arXiv:2512.19539, 2026
[arXiv] [Project Page]

StoryMem is a long-video storytelling framework that turns pre-trained single-shot video diffusion models into multi-shot storytellers by maintaining an explicit memory bank of keyframes from previously generated shots, enabling stronger cross-shot consistency, persistent character and scene coherence, and cinematic quality across minute-long stories; it also introduces ST-Bench to evaluate multi-shot visual storytelling.

4RC: 4D Reconstruction via Conditional Querying Anytime and Anywhere
Y. Luo, S. Zhou, Y. Lan, X. Pan, C. C. Loy
Technical report, arXiv:2602.10094, 2026
[arXiv] [Project Page]

4RC is a unified feed-forward framework for 4D reconstruction from monocular videos. It learns a compact spatio-temporal latent representation that jointly models scene geometry and motion, enabling an encode-once, query-anytime paradigm to recover dense 3D structure and motion between arbitrary frames and timestamps efficiently.

Thinking with Camera: A Unified Multimodal Model for Camera-Centric Understanding and Generation
K. Liao, S. Wu, Z. Wu, L. Jin, C. Wang, Y. Wang, F. Wang, W. Li, C. C. Loy
International Conference on Learning Representations, 2026 (ICLR)
[PDF] [arXiv] [Project Page] [Demo]

Puffin is a camera-centric multimodal model that unifies camera understanding and controllable image generation. By treating camera parameters as language tokens, it aligns geometric reasoning with vision–language models, enabling spatially consistent cross-view generation, camera reasoning, and scene exploration. The model is trained on Puffin-4M, a large dataset of vision–language–camera triplets.

STream3R: Scalable Sequential 3D Reconstruction with Causal Transformer
Y. Lan, Y. Luo, F. Hong, S. Zhou, H. Chen, Z. Lyu, S. Yang, B. Dai, C. C. Loy, X. Pan
International Conference on Learning Representations, 2026 (ICLR)
[arXiv] [Project Page]

STream3R reformulates multi-view 3D reconstruction as a streaming Transformer problem. Using causal attention and feature caching across frames, it incrementally reconstructs dense scene geometry from image streams, enabling scalable and efficient online 3D perception. The model learns geometric priors from large-scale 3D data and achieves strong performance on both static and dynamic scenes.

Light-X : Generative 4D Video Rendering with Camera and Illumination Control
T. Liu, Z. Chen, Z. Huang, S. Xu, S. Zhang, C. Ye, B. Li, Z. Cao, W. Li, H. Zhao, Z. Liu
International Conference on Learning Representations, 2026 (ICLR)
[arXiv] [Project Page]

Light-X is a controllable 4D video generation framework that renders videos from monocular input with joint control over camera trajectory and illumination, supporting effects such as bullet time, dolly zoom, and text-guided relighting; by disentangling geometry and lighting cues, it produces more temporally consistent and realistic results than prior video relighting and novel-view methods.

PhysX-Anything: Simulation-Ready Physical 3D Assets from Single Image
Z. Cao, F. Hong, Z. Chen, L. Pan, Z. Liu
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2026 (CVPR)
[arXiv] [Project Page]

PhysX-Anything generates simulation-ready physical 3D assets from a single image, producing not just shape but also explicit geometry, articulation, and physical attributes so the assets can be used directly in simulation and embodied AI; it also introduces a more efficient geometry representation and the PhysX-Mobility dataset with 2K+ real-world objects to support this task.

Scaling Video Matting via a Learned Quality Evaluator
P. Yang, S. Zhou, K. Hao, Q. Tao
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2026 (CVPR)
[arXiv] [Project Page] [Demo]

MatAnyone 2 is a practical video matting framework that improves robustness in real-world scenes while preserving fine boundary details such as hair and motion blur. It introduces a learned Matting Quality Evaluator (MQE) to estimate matte quality without ground truth, enabling both better training supervision and large-scale data curation. Using this idea, we build VMReal, a large real-world video matting dataset with 28K clips and 2.4M frames, and further improve temporal robustness with a reference-frame training strategy for long videos.