ReWorld-5B

ReWorld: An Interactive World Model with Long-Horizon Memory 🌍

Model Description

ReWorld is a real-time interactive world model with long-horizon memory. Given a start image, a text prompt, and a stream of keyboard/camera actions, it generates an explorable world as $704\times1280$ streaming video — and when the camera leaves a place and later returns, the scene is still there, instead of being re-invented.

ReWorld-5B is built on Wan2.2-TI2V-5B, turned into a chunk-wise causal autoregressive generator with KV caching. Two ideas carry the design:

• Control–memory decoupled training. Mixed per-head attention windows train action following under short windows and long-range recall under long ones, with random head routing keeping the two capabilities separable, and random chunk dropping making sparse histories in-distribution.
• Memory consolidation at inference. The entire past lives under a fixed budget: a bounded KV cache (12 chunks) backed by a pose-indexed landmark bank, from which the model retrieves the landmarks nearest the current camera pose. GPU memory stays constant no matter how long the rollout runs — minute-long out-and-back rollouts still regenerate the starting view.

The model is trained on a metric-scale-aligned data engine: eight sources (Unreal-rendered fly-throughs, game roaming, and real-world footage, 220K+ pose-annotated clips) placed on one physical action scale, so the same key press moves the camera the same distance in every source.

Key Features

• Long-horizon memory: revisit consistency over minute-long rollouts under a fixed KV budget (bounded cache + landmark bank; no growth with rollout length).
• Keyboard-level camera control: WASD + yaw/pitch actions with metric scale — the same command produces the same physical displacement across scenes.
• Real-time streaming mode: a rank-128 DMD LoRA compresses sampling to 4 steps; attach it for real-time interaction, detach it to recover the high-fidelity multi-step mode from the same backbone.
• One backbone, two operating points: the released AR weights serve the multi-step mode directly; AR weights + LoRA serve the real-time mode.
• Diverse worlds: photorealistic, game-style, and stylized scenes.

Model Files

File	Size	Description
reworld_5b_ar_ema.pt	22.3 GB	EMA weights of the autoregressive backbone (fp32). Used by both operating modes.
reworld_5b_dmd_lora.pt	1.3 GB	Rank-128 DMD-distilled LoRA. Attach for 4-step real-time streaming; the control path carries no LoRA.
configs/plucker720p_dmd_infer.yaml	—	Reference inference config (704×1280, 4-step, bounded-KV defaults).

Quick Start

1. Install

git clone https://github.com/zhifeichen097/ReWorld.git
cd ReWorld

conda create -n reworld python=3.10 -y
conda activate reworld

pip install -r requirements.txt
pip install flash-attn --no-build-isolation
pip install peft

2. Download the base components

ReWorld-5B uses Wan2.2-TI2V-5B components for the text encoder and VAE:

pip install "huggingface_hub[cli]"
huggingface-cli download Wan-AI/Wan2.2-TI2V-5B --local-dir ./Wan2.2-TI2V-5B

3. Download ReWorld weights

pip install modelscope
modelscope download zhifeichen097/ReWorld-5B --local_dir ./ReWorld-5B

4. Run

Interactive session (type actions, watch the world stream back):

python inference_action_v2.py \
    --config_path configs/plucker720p_dmd_infer.yaml \
    --mode interactive \
    --prompt "A cinematic Minecraft village at sunset" \
    --num_inference_steps 4 \
    --output_folder outputs/interactive

Image-to-world generation with the bounded-memory cache:

python inference_i2v_v2.py \
    --config_path configs/plucker720p_dmd_infer.yaml \
    --mode dataset \
    --init_image path/to/start_image.png \
    --num_inference_steps 4 \
    --kv_policy v15b \
    --kv_budget_chunks 12 \
    --kv_n_sink 1 \
    --kv_recent_w 5 \
    --output_folder outputs/run

Point model_ckpt / lora_ckpt in the config (or --checkpoint_path / --lora_checkpoint_path) at the two downloaded files. See docs/INFERENCE.md for multi-GPU evaluation, KV-cache policies, and the full argument reference.

Citation

@article{chen2026reworld,
  title   = {ReWorld: An Interactive World Model with Long-Horizon Memory},
  author  = {Chen, Zhifei and Wang, Luozhou and Shen, Guibao and Yan, Dongyu and Yang, Shuai and Xu, Tianshuo and Du, Yihua and Wang, Wei and Gui, Tianyi and Huang, Lianghua and Chen, Yingcong},
  journal = {arXiv preprint arXiv:2608.23565},
  year    = {2026}
}

License

Released under the Apache License 2.0, consistent with the Wan2.2 base model.

Acknowledgements

Built on Wan2.2-TI2V-5B. The real-time route follows the AR-train-then-distill recipe of the LongLive series and the DMD/Self-Forcing line of work. Thanks to the teams behind these open efforts.