janakhpon commited on
Commit
0e5c937
·
1 Parent(s): dcdec7f

docs: align the status note with the card's fallback counts

Browse files

The note quoted a rounded zero for byte fallback and claimed the README
matched model_card.json field for field. The README now reports counts
where the card reports rates, so the note says that instead of asserting
an identity that no longer holds.

Files changed (1) hide show
  1. NEXT_STEPS.md +4 -3
NEXT_STEPS.md CHANGED
@@ -19,9 +19,10 @@ tokenizer.json (here) sha256 34d181532eee7e6754bf…
19
  mon_tokenizer/src/mon_tokenizer/data/…json sha256 34d181532eee7e6754bf…
20
  ```
21
 
22
- **Byte-identical.** Every metric in `README.md` also matches `model_card.json`
23
- field for field — Mon at 4.686 chars/token over 29,600 lines, 100% round-trip,
24
- 0.00% byte fallback.
 
25
 
26
  That is the check that matters, because this repository has failed it before. The
27
  history records a release where `tokenizer.json` was silently the 4,000-piece
 
19
  mon_tokenizer/src/mon_tokenizer/data/…json sha256 34d181532eee7e6754bf…
20
  ```
21
 
22
+ **Byte-identical.** Every metric in `README.md` also traces to `model_card.json`:
23
+ Mon at 4.686 chars/token over 29,600 lines, 100% round-trip, 20 byte-fallback
24
+ tokens in 486,631. The card records fallback as a rate and the README as a count,
25
+ because Mon's rate rounds to a two-decimal zero and reads as *never*.
26
 
27
  That is the check that matters, because this repository has failed it before. The
28
  history records a release where `tokenizer.json` was silently the 4,000-piece