diffrhythm-1.2-p100a

DiffRhythm 1.2 Full, ported to one Tenstorrent Blackhole p100a. Weights: ASLP-lab/DiffRhythm-1_2-full · Upstream code: ASLP-lab/DiffRhythm · Port: SunandBean/tt-diffrhythm

Runs on p100a (mesh 1×1). The CFM transformer and the stable-audio VAE decoder run on the card; the style encoder, the sampler and the post-processing stay on the CPU.

What this port adds — the CFM transformer and the stable-audio-tools Oobleck decoder on the card, sharing the worker, transport and host helpers with the sibling tt-acestep port. What it builds on — DiffRhythm's own pipeline (Apache-2.0), which runs unchanged on the CPU; only those two stages move.

A 96-second song in 10.0 s end to end — the CFM transformer 7.7 s across 62 calls, the audio VAE decode 2.0 s. The same song takes 9.7–10.1 s on an RTX 5070 Ti.

Listen

Exactly what the container returned on run 1 of the measurement below — POST /predict with {"prompt": "warm lo-fi hip hop, mellow rhodes piano, soft vinyl crackle, relaxed drums", "instrumental": true, "duration": 96, "seed": 42}, generated in 10.05 s on the card. The file is the response's WAV bytes, unmodified: 96.0 s stereo 44.1 kHz 16-bit, sha256 7e89a573e6d94e4f.

media/diffrhythm_96s_instrumental.wav (16 MB)

How it is wired

Two stages move to the card and nothing else changes: remote.py replaces cfm.transformer with a RemoteDiffRhythmDiT and decodes through a RemoteVae, so the upstream CFM.sample loop runs unmodified.

Both shims reach the card through exactly one method, worker.call(msg), and there are two things that answer it.

Two processes (TTWorker), when DiffRhythm and tt-metal cannot share an interpreter — which is the deployment this port came from, where DiffRhythm resolved torch 2.10.0+cu128 and numpy>=2 against ttnn's torch 2.11.0+cpu and numpy 1.x:

  DiffRhythm venv (CPU)                    TT image (p100a)
  ─────────────────────                    ────────────────
  style encoder                ──socket──► worker.py
  CFM sampler loop                         DiffRhythmDiT  (on the card)
  post-processing                          OobleckTT      (on the card)

Tensors cross as raw little-endian bytes with a dtype and a shape (wire.py), because the two sides cannot share a pickle of a torch tensor.

One process (LocalWorker), when one environment satisfies both — which is what the container below is. LocalWorker answers the same call(msg) in-process, so the shims, their window arithmetic and their drop-in signatures are reused unchanged rather than reimplemented.

Messages still carry wire-encoded tensors in that case even though both ends are one process. That is deliberate: it is what lets the shims run against either worker, and the encoding is a float32 copy of a few MB against card work measured in seconds.

Quickstart

Prerequisite: a tt-metal / ttnn environment for the worker, and DiffRhythm installed in its own environment for the host side.

hf download sunandbean/diffrhythm-1.2-p100a --include "code/*" --local-dir diffrhythm-1.2-p100a && cd diffrhythm-1.2-p100a
pip install -e code/

Host side, inside the DiffRhythm environment:

from tt_diffrhythm import TTWorker, RemoteDiffRhythmDiT, RemoteVae

worker = TTWorker(cfm_model_pt, model="diffrhythm", vae=vae_model_pt).start()
worker.wait_ready()
cfm.transformer = RemoteDiffRhythmDiT(worker, cfm.transformer)
...                                          # upstream CFM.sample runs unchanged
audio = RemoteVae(worker, upsample=2048)(latents)
worker.close()

Worker side, inside the tt-metal image:

python -m tt_diffrhythm.worker --socket /ipc/worker.sock \
  --checkpoint /checkpoint/cfm_model.pt --vae /vae/vae_model.pt
python -m tt_diffrhythm.warm_cache     # compile every sequence bucket ahead of time

Duration	96–285 seconds
Lyrics	Plain lyrics with estimated line timing, or a timestamped LRC
Guidance	CFG: pred + 4 (pred − null) — every step runs two DiT calls
Shapes	Song lengths fall into sequence buckets (diffrhythm_host.seq_buckets), both CFG branches, plus the VAE windows. warm_cache compiles all of them.

Performance

Measured on an idle p100a. A 96-second song is 2067 latent frames, padded to a sequence of 2560.

DiT, whole 96-second song	8.9 s (7.7 s at HiFi2)
One DiT call at sequence 2560	121 ms
Per call during the device check	0.143–0.186 s

Where a 121 ms call goes (dit_profile.py, 156.8 ms profiled):

Op	Time	Calls
matmul	66.6 ms	66
SDPA	21.9 ms	16
add	12.0 ms	47
RMS norm	7.7 ms	32
slice	7.3 ms	62
rotary embedding	5.2 ms	32

The audio VAE decoder, against a CPU decode of the same latents:

Latent frames	CPU	p100a (first / warm)
512	10.67 s	33.94 / 0.33 s
1024	—	0.81 s

The first decode at a shape pays the kernel compilation; afterwards it is 30× faster than the CPU.

From the published container

The same image anyone can tt-model pull, reached over its own HTTP server on an idle card (2026-10-09). Three 96-second instrumentals, seed 42, from a cold start:

	run 1	run 2	run 3
Generation	10.05 s	10.06 s	10.04 s
CFM transformer, 62 calls	7.68 s	7.68 s	7.69 s
VAE decode	2.03 s	2.05 s	2.01 s

All three returned the same 96.0 s stereo 44.1 kHz WAV. Load to ready was 50 s, of which the VAE warm-up is 3.1 s; device DRAM after load is 2.9 GB of 29.9 — the card is nowhere near full, and the port holds that headroom deliberately so the sequence buckets can grow to 285 seconds.

The whole request takes about 30 s of wall time, because the response carries the song: 17 MB of WAV, base64-encoded. Generation is the 10 s above; the rest is serialising and shipping it.

Against an RTX 5070 Ti

DiffRhythm's own GPU path with the deployment's stage-by-stage placement (one model in VRAM at a time, which is what keeps it inside 16 GB), same prompt and duration, a fresh seed per run:

	p100a (container)	RTX 5070 Ti
96-second instrumental	10.04–10.06 s	9.73–10.05 s
First run	10.05 s	12.35 s
Peak device memory	2.9 GB of 29.9	2.5 GB of 15.5

The same, within noise. The card reaches it with the CFM transformer and the audio VAE decoder in TTNN and everything else on the CPU; the GPU reaches it by moving three models in and out of VRAM.

Accuracy

The torch reference in this port against upstream DiffRhythm (float32, per CFM call): PCC 0.9999997–0.9999999. For scale, upstream float32 against the GPU fp16 run is 0.999995–0.999999 — so the reimplementation is closer to upstream than upstream is to its own GPU run.

The card against this port's reference: minimum PCC 0.99994 across the whole 96-second song.

Final latents against the GPU fp16 run, which is what actually matters after CFG amplification:

	Final latent PCC	Relative RMS
CPU float32	0.99997	0.008
p100a (HiFi4, the default)	0.9954	0.096
p100a (HiFi2)	0.9931	0.118
CPU bf16	0.99129	0.132

The card beats a CPU bf16 run. Decoded audio against the GPU: PCC 0.9721 and 2.06 dB spectral distance, against 0.9691 and 3.07 dB for CPU bf16 (CPU float32 is 0.9996 / 0.51 dB). The audio VAE decoder itself is PCC 0.99963 with 0.93 dB spectral distance against the CPU.

CFG is why the margin matters: pred + 4 (pred − null) multiplies the difference between the two branches by four, so a small per-call error is amplified before it reaches the audio. DRPrecision therefore keeps the activations, the residual and the output in fp32 and defaults to HiFi4 matmuls — HiFi2 is about 14% faster and measurably worse.

Caveats

• Single card only. Mesh 1×1.
• Seeds do not reproduce across backends. A song made on the p100a cannot be reproduced bit-for-bit on a GPU with the same seed.
• Two environments, or one. Where DiffRhythm's environment and the tt-metal runtime cannot be merged, the host side runs inside DiffRhythm's own environment and the worker inside a tt-metal image. Where they can — which the container below is — LocalWorker runs both in one process. Which one you get is a property of the environment, not of the port.
• CFG doubles the work. Every sampler step is two DiT calls, one conditional and one null.
• The first song of each shape pays the kernel compilation, and so does the first VAE decode at a shape (33.9 s against 0.33 s warm). Run warm_cache after changing the code or the image.
• A render holds the card for the length of the song. On a shared card that blocks everything else.

Licensing

• Weights: ASLP-lab/DiffRhythm-1_2-full, Apache-2.0. Not redistributed here.
• Port code (code/tt_diffrhythm/): Apache-2.0, from SunandBean/tt-diffrhythm.
• code/{model,infer,g2p,config,dataset}/ are DiffRhythm itself (Apache-2.0), at 28ad63c, unmodified. They ship here because the container runs them: upstream is a research checkout with no packaging, so it is vendored as source rather than resolved by pip. code/models/ is tt-metal's models/common and models/tt_dit (Apache-2.0), for the same reason.
• code/tt_diffrhythm/common.py and the worker, transport and audio VAE modules are shared with the sibling tt-acestep port (Apache-2.0); oobleck_vae.py implements both DiffRhythm's stable-audio-tools decoder and ACE-Step's diffusers Oobleck decoder.

Run it as a container

Everything below is generated by tt-model package from the manifest this image was built from.

tt-model pull  sunandbean/diffrhythm-1.2-p100a --with-weights
tt-model serve sunandbean/diffrhythm-1.2-p100a

pull --with-weights downloads the Docker image and the ASLP-lab/DiffRhythm-1_2-full weights (into your HF cache; they are not in the image). serve starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs Application startup complete.

Request shape

POST /predict takes {"prompt": str, "lyrics": str, "instrumental": bool, "duration": int, "seed": int} and returns a base64 WAV with timing_s and the lrc the song was actually sung to.

prompt is the style description. Vocals need lyrics: real LRC is used as given, and plain lines get approximate starts derived from their word counts -- a convenience, not learned alignment. duration is 96-285 seconds; use the ACE-Step port for longer songs. The schedule is fixed at 32 steps; any other inference_steps is a 422.

Provenance

The exact sources the image was built from — the model code under code/ is byte-identical to what is inside the image (pyproject.toml, PYTHON.md and examples/ are packaging files this repo adds, and are not in the image):

component	built from
tt-metal	58648d44443223041148125843ca7c7d4780463e
code/ digest	6fe96af2d7e7a1b8 (sha256, first 16 hex digits)
built	2026-10-09T14:54:37+00:00 by tt-model 0.1.0