[ECCV 2026] MoGe4D: Geometry-Aware Single-Image 4D Synthesis via Dense Trajectory Generation

Yanran Zhang^*,1, Ziyi Wang^*,1, Wenzhao Zheng^†,1, Zheng Zhu², Jie Zhou¹, Jiwen Lu¹

¹Department of Automation, Tsinghua University     ²GigaAI

^*Equal Contribution    ^†Corresponding Author

📄 Paper Summary

Generating interactive and dynamic 4D scenes from a single static image is a core challenge. Existing methods decouple geometry from motion — either generate-then-reconstruct (geometric inconsistency) or reconstruct-then-generate (limited, externally-constrained motion) — causing spatiotemporal inconsistency and poor generalization.

MoGe4D (Motion and Geometry-aware image-to-4D synthesis) is a geometry-conditioned framework that models a scene as dense 4D point trajectories. Starting from an initial geometric prior of the input image, it predicts future time-varying trajectories through a diffusion process, tightly coupling geometric modeling with motion generation. This yields 4D scenes with strong temporal coherence, geometry-aware consistency, and compelling novel-view synthesis.

Contributions:

• TrajScene-60K — 60K videos with dense 4D point trajectories (3M+ frames, ~12B 3D points).
• 4D-STraG — a diffusion trajectory generator with depth-guided motion normalization and a Motion Perception Module (MPM).
• 4D-ViSM — a view-synthesis module rendering the 4D representation under arbitrary camera trajectories.

🧱 Model Structure

This repository releases the three trained components of MoGe4D:

Path	Size	Description
4D-STraG/diffusion_pytorch_model.safetensors	~31.9 GiB	4D Scene Trajectory Generator (diffusion model, built on Wan2.1-14B)
4D-ViSM/lora_diffusion_pytorch_model.safetensors	~1.36 GiB	4D View Synthesis Module (LoRA adapter)
VAE/vae/pytorch_model.bin	~484 MiB	Motion-sensitive VAE for trajectory signals
VAE/{encoder,decoder}_prompt/pytorch_model.bin	~1–2 MiB	VAE prompt encoder/decoder
VAE/{optimizer.bin, scheduler.bin, random_states_0.pkl}	—	Training states for the VAE (optional, for resuming training)

🛠️ Usage

1. Set up the environment

git clone https://github.com/Zhangyr2022/MoGe4D.git
cd MoGe4D
conda create -n MoGe4D python=3.10 && conda activate MoGe4D
conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia
pip install -r requirements.txt

Install third-party deps: UniDepth and diff-gaussian-rasterization.

2. Download the checkpoints

huggingface-cli download Yanran21/MoGe4D --local-dir ./models --resume-download

(Also place the base backbones Wan2.1-Fun-V1.1-14B-Control/InP, OmniMAE, and UniDepth under ./models.)

3. Inference

bash scripts/inference/infer.sh        # whole pipeline: image → 4D scene → multi-view videos

See the GitHub README for training scripts and details.

📊 Results

MoGe4D delivers superior geometric consistency, dynamic realism, and visual fidelity over decoupled approaches (e.g., generate-then-reconstruct with VGGT). Please refer to the paper for quantitative metrics and qualitative comparisons.

📖 Citation

@inproceedings{zhang2026moge4d,
  title={Geometry-Aware Single-Image 4D Synthesis via Dense Trajectory Generation},
  author={Zhang, Yanran and Wang, Ziyi and Zheng, Wenzhao and Zhu, Zheng and Zhou, Jie and Lu, Jiwen},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2026}
}

📧 Contact

• Yanran Zhang — zhangyr21@mails.tsinghua.edu.cn