docs: align the status note with the card's fallback counts
Browse filesThe note quoted a rounded zero for byte fallback and claimed the README
matched model_card.json field for field. The README now reports counts
where the card reports rates, so the note says that instead of asserting
an identity that no longer holds.
- NEXT_STEPS.md +4 -3
NEXT_STEPS.md
CHANGED
|
@@ -19,9 +19,10 @@ tokenizer.json (here) sha256 34d181532eee7e6754bf…
|
|
| 19 |
mon_tokenizer/src/mon_tokenizer/data/…json sha256 34d181532eee7e6754bf…
|
| 20 |
```
|
| 21 |
|
| 22 |
-
**Byte-identical.** Every metric in `README.md` also
|
| 23 |
-
|
| 24 |
-
|
|
|
|
| 25 |
|
| 26 |
That is the check that matters, because this repository has failed it before. The
|
| 27 |
history records a release where `tokenizer.json` was silently the 4,000-piece
|
|
|
|
| 19 |
mon_tokenizer/src/mon_tokenizer/data/…json sha256 34d181532eee7e6754bf…
|
| 20 |
```
|
| 21 |
|
| 22 |
+
**Byte-identical.** Every metric in `README.md` also traces to `model_card.json`:
|
| 23 |
+
Mon at 4.686 chars/token over 29,600 lines, 100% round-trip, 20 byte-fallback
|
| 24 |
+
tokens in 486,631. The card records fallback as a rate and the README as a count,
|
| 25 |
+
because Mon's rate rounds to a two-decimal zero and reads as *never*.
|
| 26 |
|
| 27 |
That is the check that matters, because this repository has failed it before. The
|
| 28 |
history records a release where `tokenizer.json` was silently the 4,000-piece
|