CMF Decision and Laya: a 77-option stress test

BANKING77 · 3,080 test requests · no oracle · 28 September 2026

This compares a trained CMF skill with Laya's base English checkpoint given all 77 label names in one request, without fine-tuning. Both return structured decisions without generating answer tokens.

Scope: this is a high-cardinality stress test, not a measurement of the best Laya configuration. Its upstream documentation recommends narrowing large option sets with predict_shortlist, and a multilingual checkpoint for long inputs. Neither was measured here. Preparation also differs: CMF was trained on this task; Laya was not. The results do not isolate architecture from training. Upstream guidance at the tested source revision.

Primary, all 3,080 rows	CMF Decision / Metal	Laya 0.3.21 / MPS
Correct / all	2875 / 3080	1098 / 3080
Top-1 accuracy, before abstention	93.34%	35.65%
95% Wilson interval	92.41%–94.17%	33.98%–37.36%
Macro-F1	0.9334	0.3612
Local request p50	3.10 ms	917.59 ms
Local request p95	7.58 ms	1359.36 ms
Peak process RSS, OS high-water mark	555 MiB	2786 MiB

The measured p50 ratio is 296.5× for these paths and this 77-way task. CMF already stores the trained task; Laya reads the rubric with each request. This is not a universal speed or quality ranking. Laya is not a generative-LLM straw man, and no remote Jev timing is mixed into this chart.

More examples: a failed long-rubric configuration

A subset of 231 test rows (three per class) was fixed by text hash before evaluation. The longer rubric supplies two training examples per class, as in the earlier Jev experiment. Laya caps each criterion at 48 tokens. The same subset is shown for all three conditions; it is not compared with CMF's full-set score.

The longer English-checkpoint run collapsed toward one label: 206 of 231 predictions were visa_or_mastercard. Treat its 1.73% as a failed configuration, not as Laya's general accuracy. Increasing a token limit prevents truncation; it does not establish that this checkpoint can use the longer context reliably. We retain this prespecified result rather than discarding an unfavorable run.

Same 231 rows	Accuracy	Macro-F1	p50 / p95, ms
CMF, unchanged trained skill	92.21%	0.9173	3.09 / 7.69
Laya, label names only	35.50%	0.3391	934.61 / 1444.52
Laya, two examples per label	1.73%	0.0069	4905.75 / 5958.05

Long- and short-rubric latency must not be treated as the same workload. The first measured Laya input contains 939 tokens in the short condition and 3,199 in the longer condition. These include the rubric, not only the user text.

A post-hoc sanity check used two simple three-option controls and one training input on both MPS/FP16 and CPU/FP32. Both devices returned the same labels and rounded confidence values in all four configurations: the simple controls worked, while the long-rubric training input again selected visa_or_mastercard. This limited check found no device-specific explanation; it does not diagnose the exact cause or replace a full CPU evaluation. No held-out examples were used to tune a replacement configuration. Diagnostic results.

Protocol and resource interpretation

Apple M4, 24 GiB unified memory, macOS 27.2. Runtimes ran sequentially, with 50 training requests for warm-up. These are local text-to-decision request times, excluding startup, HTTP and file serialization.
CMF: released Cortiq 0.8.0 binary, unchanged published model, Metal explicitly selected, nonzero GPU timings on every test row. Gate coverage: 90.62%; accuracy among accepted requests: 97.24%. The primary top-1 score includes rejections.
Laya: source 9d955671415fc19f069b9cc998928075c1f255ec, version 0.3.21, English checkpoint at 55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851, PyTorch 2.14.0, runtime dtype torch.float16, four CPU threads, MPS synchronized around each call, zero CPU fallbacks. No compilation, batching, fine-tuning or oracle. Upstream source.
All 77 option markers are checked. Short rubric: head limit 1024 / total 1536; long rubric: 4096 / 4608. Using Laya's smaller defaults would squeeze the 77-label question. This configuration is explicit, not a hidden default.
Laya warns that this checkpoint's 11+ choice temperature is outside its validated range and clamps it to 0.5. We do not compare its confidence values or claim calibrated selective accuracy; temperature does not change top-1 order.
Peak RSS includes loading and warm-up. Laya MPS driver allocation after the short run is 2058 MiB. Do not add GPU allocations to RSS on unified memory or mistake RSS for total GPU memory. The CMF file is 304,520,292 bytes; Laya's weights alone are 842,609,210 bytes, plus tokenizer/configuration. Storage size is not working memory.
The Mac remained a normal desktop: lightweight editing, Git, ADB and HTTP clients for a separate phone ran during the long test. No other local neural evaluation or Flutter build/test ran concurrently. This is not an isolated laboratory timing guarantee. Mobile builds ran on separate GitHub runners.
Public BANKING77 tests had already been used in development. Not a new blind holdout. CMF was fitted on train + dev; Laya's base checkpoint was not fitted on this dataset in this experiment. Different preparation remains a limitation even on identical hardware. BANKING77: PolyAI, CC-BY-4.0.

Evidence

Summary and hashes · Frozen protocol · Frozen questions/subset · Warm-up implementation note · Environment and checkpoint hashes · Reproduction guide

Runner source, CMF runner and statistics, chart generator. The scripts retain the source repository layout and explicit release/data paths; replace those paths when reproducing elsewhere. Machine-local paths in the public metadata are redacted; per-row files are unchanged. BANKING77 inputs in the evidence bundle are attributed to PolyAI under CC-BY-4.0; see its data note.

Per-row decisions and timings: CMF, Laya short rubric, Laya examples rubric. The model SHA-256 is ed9b8ec2bbfe9e9fd30f14a5eaf82314f38bc7e7510a39772baa2de3801d79b1. The test-file SHA-256 is 649607fa7b124293c80b455d2effe5e41da713e1836ee936073fe34ab9dc6fba.