NeTSlab/emg_mts1_mopbf_en_nl_zh_equal

eMG-MTS S1 — Multi-Timescale eMG (English / Dutch / Chinese)

emg_mts1_mopbf_en_nl_zh_equal is a small, multilingual, linguistically motivated causal language model trained for the BabyLM 2026 multilingual track (English, Dutch, Mandarin Chinese) on a developmentally plausible word budget. It is a research model whose purpose is to test whether a cognitively grounded memory mechanism can match a strong opaque baseline on syntactic benchmarks, not a general-purpose chat or production model.

S1 is the fast-only member of the eMG-MTS family (one retention tier). Despite that, it already recovers long-range filler–gap dependencies at a level at or above the Hawk baseline (see Evaluation).

What it is

eMG-MTS realizes memory as multi-timescale, content-addressable retrieval. Each "selective state" is a decayed linear-attention retention tier: the read at position t is a competition over cue·key similarity weighted by a base-level recency term γ^(t−i) —

o_t = Σ_{i≤t} γ^(t−i) (q_t · k_i) v_i  /  Σ_{i≤t} γ^(t−i) (q_t · k_i)

which is the Lewis & Vasishth (2005) retrieval equation made architectural. The per-tier decay γ is fixed, not learned, so the timescale is a structural constant rather than something the model must discover. Because the underlying recurrence S_t = γ S_{t−1} + k_t v_tᵀ is linear with a scalar decay, it has an exact parallel form (loop-free, torch.compile-friendly), and the state is a bounded d_k × d_v fast-weight matrix (this is a linear RNN with a content-addressable state, not softmax attention: no KV-cache, no lookback, no softmax). Long-range information travels only through retention reads: the residual stream carries a strictly local causal-conv path, so depth composes range without an escape hatch.

For numerical stability the queries/keys are RMS-normalized before the feature map and the read is computed in fp32 (a normalized linear-attention read is otherwise prone to divergence).

• Architecture: 14 blocks, hidden size 704, one fast retention tier (γ = 0.85), depthwise causal conv (kernel 4) + gated MLP local path.
• Parameters: ≈103M total (≈75M compute core; the rest is the tied embedding over a ~39.9K vocab).
• Tokenizer: native MorPiece (MoP-16K, byte-fallback), shipped with the model as morpiece_native.json + tokenization_morpiece.py + tokenizer_MorPiece.py. The byte-fallback keeps rare Chinese characters distinct, which is why ZhoBLiMP does not collapse.
• Heads: causal-LM plus sequence- and token-classification heads (for GLUE-style and UD-style fine-tuning) share one backbone.

How to use

The model and tokenizer use custom code, so trust_remote_code=True is required.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

name = "NeTSlab/emg_mts1_mopbf_en_nl_zh_equal"
tok = AutoTokenizer.from_pretrained(name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(name, trust_remote_code=True).eval()

# minimal-pair scoring (lower loss = more grammatical)
def sentence_logprob(text):
    ids = tok(text, return_tensors="pt").input_ids
    with torch.no_grad():
        out = model(input_ids=ids, labels=ids)
    return -out.loss.item() * ids.shape[1]

print(sentence_logprob("The keys to the cabinet are on the table."))
print(sentence_logprob("The keys to the cabinet is on the table."))

For fine-tuning (e.g. the BabyLM classification / UD tasks), the same checkpoint loads under the classification auto-classes with a freshly initialized head:

from transformers import AutoModelForSequenceClassification, AutoModelForTokenClassification
seq = AutoModelForSequenceClassification.from_pretrained(name, num_labels=3, trust_remote_code=True)
tok_clf = AutoModelForTokenClassification.from_pretrained(name, num_labels=17, trust_remote_code=True)

Training data & procedure

• Data: the BabyLM 2026 multilingual selection (English / Dutch / Chinese), with per-language exposure balanced by byte-premium-adjusted English-equivalent words (_equal). The three languages contribute roughly equal English-equivalent word counts.
• Objective: plain next-token cross-entropy (no auxiliary or supervision losses in this S1 model).
• Optimizer: AdamW, bf16 mixed precision, cosine LR schedule with warmup, gradient clipping 1.0, weight decay 0.01.
• Tokenizer: native MorPiece MoP-16K (byte-fallback), loaded and re-emitted natively so training and evaluation segmentation match exactly.

Evaluation

Zero-shot minimal-pair accuracy on the BabyLM 2026 suite. Aggregate group scores: English 0.6425 · Dutch 0.5667 · Chinese 0.6497.

Benchmark	Metric	S1
English — BLiMP (BabyLM filtered)	acc	0.7029
English — MultiBLiMP	acc	0.8792
Dutch — BLiMP-NL	acc	0.7975
Dutch — MultiBLiMP	acc	0.9086
Chinese — ZhoBLiMP	acc	0.8090
Chinese — xcomps_zh	acc	0.5676
Dutch — xcomps_nl	acc	0.5704

Selected syntactic phenomena (English BLiMP):

Phenomenon	acc
wh_vs_that_with_gap	0.343
wh_vs_that_with_gap (long distance)	0.135
wh_vs_that_no_gap	0.961
wh_questions_subject_gap	0.891
wh_questions_subject_gap_long_distance	0.876
wh_questions_object_gap	0.754
wh_island	0.596
adjunct_island	0.767
determiner_noun_agreement (mean)	~0.90
anaphor_number_agreement	0.969
principle_A_case_1	1.000
distractor_agreement (mean)	0.453

Two findings are worth highlighting because they are the point of the model:

• Filler–gap is genuinely acquired. wh_vs_that_with_gap reaches 0.343 (with no_gap at 0.96), a level comparable to or above the Griffin/Hawk baseline, and a large jump over cue-based predecessors that were mechanistically inert on this dependency. The residual short-vs-long gap (0.343 → 0.135) is the expected limitation of a single fast tier and is the target of the slow-tier sibling (S2).
• Chinese is not degraded by tokenization. ZhoBLiMP is 0.809, with no wholesale collapse — a direct benefit of the native byte-fallback MorPiece tokenizer.

Model family

Model	Tiers	Switchboard	Note
emg_mts1 (this repo)	fast (γ=0.85)	off	the fast-only baseline
emg_mts2	fast + slow (γ=0.995)	off	adds the slow tier for long-distance preservation
emg_mts1par / emg_mts2par	as above	on (film)	+ typological switchboard (Roberts-grounded FiLM)

Limitations & intended use

• Research model. Intended for syntactic evaluation and analysis, not for text generation quality, factual reliability, or downstream deployment. Commonsense/NLI-style tasks (HellaSwag, WinoGrande, XStoryCloze) sit near chance, as expected at this scale and budget.
• Rare constructions are under-learned. Dutch parasitic_gaps (0.251) and a few Chinese subtasks are low — these are constructions that are rare in the training diet; the limitation is one of recruitment (data frequency), not of the architecture's capacity.
• Long-distance object extraction is the known weak point of the fast-only S1 (see the 0.343 → 0.135 drop); use S2 where long-range preservation matters.
• Trained on ~3 languages only; any typological interpretation of internal representations is correspondingly limited.

References

• De et al. (2024), Griffin / Hawk: Mixing Gated Linear Recurrences with Local Attention (baseline family).
• Lewis & Vasishth (2005), An Activation-Based Model of Sentence Processing as Skilled Memory Retrieval (the retrieval equation realized here).
• Roberts (2019), typological parameters (used by the optional switchboard variants).
• BabyLM 2026 multilingual shared task.

Developed at IUSS Pavia / NeTS-lab with crucial optimizations and bug-fixes by Claude Opus 4.8