| # CMF Decision and Laya: a 77-option stress test |
|
|
| **BANKING77 · 3,080 test requests · no oracle · 28 September 2026** |
|
|
| This compares a **trained CMF skill** with Laya's **base English checkpoint** |
| given all 77 label names in one request, without fine-tuning. Both return |
| structured decisions without generating answer tokens. |
|
|
| **Scope:** this is a high-cardinality stress test, not a measurement of the best |
| Laya configuration. Its upstream documentation recommends narrowing large option |
| sets with `predict_shortlist`, and a multilingual checkpoint for long inputs. |
| Neither was measured here. Preparation also differs: CMF was trained on this |
| task; Laya was not. The results do not isolate architecture from training. |
| [Upstream guidance at the tested source revision](https://github.com/NandhaKishorM/laya/blob/9d955671415fc19f069b9cc998928075c1f255ec/README.md#honest-limits). |
|
|
|  |
|
|
| | Primary, all 3,080 rows | CMF Decision / Metal | Laya 0.3.21 / MPS | |
| |---|---:|---:| |
| | Correct / all | 2875 / 3080 | 1098 / 3080 | |
| | Top-1 accuracy, before abstention | 93.34% | 35.65% | |
| | 95% Wilson interval | 92.41%–94.17% | 33.98%–37.36% | |
| | Macro-F1 | 0.9334 | 0.3612 | |
| | Local request p50 | 3.10 ms | 917.59 ms | |
| | Local request p95 | 7.58 ms | 1359.36 ms | |
| | Peak process RSS, OS high-water mark | 555 MiB | 2786 MiB | |
|
|
| The measured p50 ratio is **296.5×** for these paths |
| and this 77-way task. CMF already stores the trained task; Laya reads the rubric |
| with each request. This is not a universal speed or quality ranking. Laya is not |
| a generative-LLM straw man, and no remote Jev timing is mixed into this chart. |
|
|
| ## More examples: a failed long-rubric configuration |
|
|
| A subset of 231 test rows (three per class) was fixed by text hash **before |
| evaluation**. The longer rubric supplies two training examples per class, as in |
| the earlier Jev experiment. Laya caps each criterion at 48 tokens. The same subset |
| is shown for all three conditions; it is not compared with CMF's full-set score. |
|
|
| The longer English-checkpoint run **collapsed toward one label**: |
| 206 of 231 predictions were `visa_or_mastercard`. Treat its 1.73% |
| as a failed configuration, **not as Laya's general accuracy**. Increasing a token |
| limit prevents truncation; it does not establish that this checkpoint can use |
| the longer context reliably. We retain this prespecified result rather than |
| discarding an unfavorable run. |
|
|
| | Same 231 rows | Accuracy | Macro-F1 | p50 / p95, ms | |
| |---|---:|---:|---:| |
| | CMF, unchanged trained skill | 92.21% | 0.9173 | 3.09 / 7.69 | |
| | Laya, label names only | 35.50% | 0.3391 | 934.61 / 1444.52 | |
| | Laya, two examples per label | 1.73% | 0.0069 | 4905.75 / 5958.05 | |
|
|
| Long- and short-rubric latency must not be treated as the same workload. |
| The first measured Laya input contains 939 tokens in the short condition and |
| 3,199 in the longer condition. These include the rubric, not only the user text. |
|
|
| A post-hoc sanity check used two simple three-option controls and one training |
| input on both MPS/FP16 and CPU/FP32. Both devices returned the same labels and |
| rounded confidence values in all four configurations: the simple controls |
| worked, while the long-rubric training input again selected `visa_or_mastercard`. |
| This limited check found no device-specific explanation; it does not diagnose |
| the exact cause or replace a full CPU evaluation. No held-out examples were used |
| to tune a replacement configuration. [Diagnostic results](evidence/laya/train-device-diagnostic.json). |
|
|
| ## Protocol and resource interpretation |
|
|
| - Apple M4, 24 GiB unified memory, macOS 27.2. Runtimes ran **sequentially**, |
| with 50 training requests for warm-up. These are local text-to-decision |
| request times, excluding startup, HTTP and file serialization. |
| - CMF: released Cortiq 0.8.0 binary, unchanged published model, Metal explicitly |
| selected, nonzero GPU timings on every test row. Gate coverage: |
| 90.62%; accuracy among accepted requests: |
| 97.24%. The primary top-1 score includes rejections. |
| - Laya: source `9d955671415fc19f069b9cc998928075c1f255ec`, version 0.3.21, |
| [English checkpoint](https://huggingface.co/convaiinnovations/laya) at |
| `55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851`, PyTorch 2.14.0, |
| runtime dtype **torch.float16**, four CPU threads, MPS synchronized |
| around each call, **zero CPU fallbacks**. No compilation, batching, fine-tuning |
| or oracle. [Upstream source](https://github.com/NandhaKishorM/laya). |
| - All 77 option markers are checked. Short rubric: head limit 1024 / total 1536; |
| long rubric: 4096 / 4608. Using Laya's smaller defaults would squeeze the |
| 77-label question. This configuration is explicit, not a hidden default. |
| - Laya warns that this checkpoint's 11+ choice temperature is outside its |
| validated range and clamps it to 0.5. We do not compare its confidence values |
| or claim calibrated selective accuracy; temperature does not change top-1 order. |
| - Peak RSS includes loading and warm-up. Laya MPS driver allocation after the |
| short run is 2058 MiB. **Do not add GPU |
| allocations to RSS** on unified memory or mistake RSS for total GPU memory. |
| The CMF file is 304,520,292 bytes; Laya's weights alone are 842,609,210 bytes, |
| plus tokenizer/configuration. Storage size is not working memory. |
| - The Mac remained a normal desktop: lightweight editing, Git, ADB and HTTP |
| clients for a separate phone ran during the long test. No other local neural |
| evaluation or Flutter build/test ran concurrently. This is not an isolated |
| laboratory timing guarantee. Mobile builds ran on separate GitHub runners. |
| - Public BANKING77 tests had already been used in development. **Not a new |
| blind holdout.** CMF was fitted on train + dev; Laya's base checkpoint was |
| not fitted on this dataset in this experiment. Different preparation remains |
| a limitation even on identical hardware. BANKING77: PolyAI, CC-BY-4.0. |
|
|
| ## Evidence |
|
|
| [Summary and hashes](evidence/laya/comparison.json) · |
| [Frozen protocol](evidence/laya/PROTOCOL.md) · |
| [Frozen questions/subset](evidence/laya/freeze.json) · |
| [Warm-up implementation note](evidence/laya/IMPLEMENTATION.md) · |
| [Environment and checkpoint hashes](evidence/laya/environment.json) · |
| [Reproduction guide](evidence/laya/REPRODUCE.md) |
|
|
| [Runner source](evidence/laya/code/laya_compare.py), [CMF runner and statistics](evidence/laya/code/laya_cmf_finish.py), [chart generator](evidence/laya/code/laya_figures.py). |
| The scripts retain the source repository layout and explicit release/data paths; |
| replace those paths when reproducing elsewhere. Machine-local paths in the |
| public metadata are redacted; per-row files are unchanged. BANKING77 inputs in |
| the evidence bundle are attributed to PolyAI under CC-BY-4.0; see its data note. |
|
|
| Per-row decisions and timings: [CMF](evidence/laya/cmf-metal.jsonl), |
| [Laya short rubric](evidence/laya/laya-short.jsonl), |
| [Laya examples rubric](evidence/laya/laya-full.jsonl). |
| The model SHA-256 is `ed9b8ec2bbfe9e9fd30f14a5eaf82314f38bc7e7510a39772baa2de3801d79b1`. |
| The test-file SHA-256 is `649607fa7b124293c80b455d2effe5e41da713e1836ee936073fe34ab9dc6fba`. |
|
|
