README-PCD.md

LFM2.5-2.6B-RLCD

Fast finite-choice structured inference using unchanged LiquidAI/LFM2.5-2.6B weights. This package implements Parallel Constrained Decoding (PCD): prefill shared context once, branch the model's attention and convolution state, evaluate allowed answers in parallel, and serialize a typed JSON object in Python.

Inference-only, experimental, and uncalibrated. No training, LoRA, reinforcement learning, quantization, or saved weight modification was performed. The RLCD name follows the community inference examples; it does not mean we reproduced TypeSafe/Jev's Reinforcement Learning for Calibrated Decisions. We target the fast finite-choice interaction pattern, not proprietary training, calibration, API parity, or an unmeasured speed claim.

Results and release status

See measured results, wrong answers, and limitations, with raw JSON evidence under results/pcd/. Only the measurements for the released source revision belong to this release. This is a small development/diagnostic evaluation, not held-out production validation.

On the final single-L40S run, fast token PCD averaged 56.24 ms on 12 development cases versus 557.41 ms for direct-answer AR JSON (9.9x). On a single synthetic 28-boolean configuration, it took **106.98 ms versus 4,875.30 ms (45.6x)**, with all fields correct for both methods. These are warm GPU request times, not network latency.

Accuracy is the important limitation: token PCD reached 88.9% development field accuracy but only 72.2% on six fresh audit cases, versus 94.4% for AR JSON. It also failed both high-cardinality token-mode probes. The production accuracy gate was not met. Sequence scoring correctly resolved those high-cardinality examples, but the 255-choice case was slower than AR. See the report rather than extrapolating the favorable 28-field speedup to every task.

Schema validity does not mean a correct decision. Do not use the returned probabilities as validated automation thresholds. Test your own labeled workload; use human review or a stronger reasoning model where mistakes matter. No comparison against Jev's service was performed.

Two inference modes

token: fast parallel choices

Each field receives an unambiguous atomic option code. Codes are verified against the actual tokenizer. One cached branch per field produces a decision hidden state; the output head is projected only onto the relevant option-token rows, not the entire 128,000-token vocabulary. The selected code is mapped back to the original enum string or actual Python boolean.

This is a classifier-style next-token decision, not arbitrary free-text generation. It does not collapse multi-token enum values onto an ambiguous first token: opaque codes distinguish them. The output includes the distribution over allowed codes, explicitly marked uncalibrated. Code/label wording and ordering can affect predictions.

sequence: full-sequence scoring reference

Each candidate is a complete JSON value with a newline terminator. The whole JSON member is canonically tokenized before factoring its shared token prefix, preserving space/quote merges. All remaining candidate tokens are scored with teacher forcing and full-vocabulary normalized log-likelihood; the score is not a first-token proxy. Shared-prefix choices, escaped strings, and Unicode are supported. The scoring convention still has length and wording bias.

Both paths reuse the shared prompt once and fork isolated hybrid caches. Token mode normally uses two backbone calls for up to 32 fields. Sequence mode uses 1 + ceil(total_candidates / branch_batch_size) calls. Parallelism reduces sequential steps; it does not make memory, FLOPs, or latency constant in the input size.

Install and use

Get the code without immediately downloading duplicate weight files:

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/monotykamary/LFM2.5-2.6B-RLCD
cd LFM2.5-2.6B-RLCD
uv venv --python 3.11
uv pip install --python .venv/bin/python -r requirements-pcd.txt
source .venv/bin/activate

The default engine loads the pinned original LiquidAI model. On a CUDA GPU:

from pcd import Engine

schema = {
    "type": "object",
    "properties": {
        "topic": {
            "type": "string",
            "enum": ["billing", "technical", "shipping"],
            "description": "The main issue in the message",
        },
        "refund": {
            "type": "boolean",
            "description": "Whether the customer explicitly requests a refund",
        },
    },
    "required": ["topic", "refund"],
    "additionalProperties": False,
}
engine = Engine(device="cuda", dtype="float16", attention="sdpa")
result = engine.constrained(
    "I was charged twice. Please refund the duplicate charge.", schema, mode="token"
)
print(result["object"])          # Typed values; field decisions may still be wrong.
print(result["fields"])          # Scores, candidate distributions, and margins.
print(result["elapsed_ms"])
assert result["calibrated"] is False

reference = engine.constrained("Please refund my duplicate charge.", schema, mode="sequence")

To use the bundled weights instead, pass model_id="monotykamary/LFM2.5-2.6B-RLCD" and revision="main" to Engine; pin the public commit SHA for reproducibility. The original BF16 tensors are bundled unchanged, even though the measured engine uses an FP16 runtime cast. Plain AutoModelForCausalLM.from_pretrained(...) loads the original generative model; it does not enable parallel constrained inference. No trust_remote_code=True is needed.

Supported schema and limits

• Closed, flat object, with every property required and additionalProperties: false.
• Fields are booleans or nonempty unique string enums; descriptions are optional.
• No arbitrary numbers, free-text strings, arrays, nested/optional fields, or relational constraints.
• Defaults: 32 fields, 256 total candidates, 4,096 shared-prompt tokens (schema included), 64 tokens per serialized value, and 32 branches per microbatch.
• Limits and unsupported schema keywords are rejected, not silently ignored.
• Successful calls guarantee syntax, types and enum membership through programmatic assembly. They do not guarantee truth, completeness of the answer space, or cross-field consistency.

Return values live in result["object"]; telemetry is separate and does not alter your schema. token probabilities are a restricted-code softmax; sequence probabilities are normalized candidate likelihoods. Neither is a calibration guarantee or evidence that all candidates include the correct answer. Include an explicit unknown/other option where appropriate.

LFM2.5-specific details

The pinned model has 8 full-attention layers, 22 short-convolution layers and 2,697,198,592 parameters. Branches copy both KV tensors and convolution state; mutable expanded views are not shared. Request caches are discarded, not retained across users. GPU operations are serialized within one engine instance to bound memory and avoid state races.

The native chat template always opens <think>. The PCD prompts explicitly supply an empty closed reasoning span; this is not an officially supported enable_thinking=False mode. Skipping reasoning can hurt accuracy. The original native reasoning behavior is preserved in the unchanged model/tokenizer and is available through normal generation.

FP32 CUDA and local tiny-model checks establish cache/scoring equivalence; measured FP16 rounding tolerances and errors are in the results. The initial BF16 path failed the tighter hidden-state comparison and is not the recommended validated serving precision.

Run on Modal, frugally

The scripts use the existing huggingface-cache Volume, a separate results Volume, one L40S, zero minimum containers, a maximum of one container per parameterization, and short idle shutdown. No H100, multi-GPU job or training run is required.

modal run lfm25_pcd_modal.py --task prepare
modal run lfm25_pcd_modal.py --task validate --precision float32
modal run lfm25_pcd_modal.py --task benchmark --suite diagnostic --repeats 3
modal run lfm25_pcd_modal.py --task benchmark --suite stress --repeats 3

Preparation/publication use a Modal Secret named huggingface (HF_TOKEN preferred). GPU inference only needs public cached weights; it is not given the write credential. Benchmarks have a time budget and persist raw results. They do not perform automatic retries or switch to a larger GPU on failure. Pricing and actual billed time depend on your workspace.

PCDModel.infer.remote(context, schema, mode) supports on-demand calls. An authenticated POST Web Function is included (PCDModel.extract); it accepts context, schema, and optional mode. It requires Modal proxy authentication (Modal-Key and Modal-Secret headers). Deploy with modal deploy lfm25_pcd_modal.py only after validating your workload. This release does not claim a production-ready always-on endpoint or measured network/cold-start latency.

Tests and reproducibility

python -m pytest tests/pcd -q

Local tests use a small randomly initialized LFM2 model: no GPU and no full-model download. They test cached/full scoring, branch isolation, both convolution cache paths, microbatch invariance, schema rejection, token shifts and typed output. Modal validation repeats critical checks on the real pinned model. Benchmarks retain outputs, errors, source hashes, dependency versions, actual accelerator, precision, prompt sizes and memory measurements.

Upstream identity: LiquidAI/LFM2.5-2.6B at 654f9463ce32b05d0429d76fe1f580b27d4c1ac0. The Hub release includes BASE_MODEL_MANIFEST.json and ARTIFACTS.json for model and code provenance. The original model card is preserved as UPSTREAM_README.md and appended below the constrained-inference documentation in the published README.

Attribution and license

The 350M reference's hybrid-cache/full-sequence design informed this implementation: notnotsamuel/LFM2.5-350M-RLCD. See THIRD_PARTY_NOTICES.md and LICENSE-CODE.

LiquidAI weights, tokenizer, configuration and original model documentation retain the LFM Open License v1.0, including redistribution requirements and commercial revenue conditions. The inference-code MIT license does not replace the model license.

TypeSafe describes Jev's actual RLCD training in its AI primer. This is an independent PCD implementation, not an affiliated or equivalent Jev release.