HaHaScore — Honest Evaluation Report

All numbers in this card are 5×3 repeated CV verified (15 measurements, 10,000-resample bootstrap 95% CI).

Honest Results Summary

Model	Task	Metric	5×3 honest	CI
Bridge 4 (BiGRU 791d)	Humor detection (StandUp4AI, binary)	AUC = 0.9285 ± 0.0084	[0.9246, 0.9331]	✅ PUBLISHABLE
v10 (Cascade Gate + CORAL)	Multimodal humor detection (12-video gold)	humor ρ = 0.6924 ± 0.0359, gold ρ = 0.5453 ± 0.1313	—	✅ Honest
v5 (DeBERTa + WavLM bilinear)	Multimodal humor detection	ρ = 0.632 ± 0.007	—	✅ Honest baseline
v1 (RoBERTa-base + Linear)	Text-only humor-strength on Jester	ρ = 0.2920 ± 0.0272	[0.2783, 0.3057]	⚠️ GATE NOT MET
v2 LaughO (sister)	Audio-only laughter detection on AudioSet	ρ = 0.7501 ± 0.1817 (1 shard)	—	⚠️ Proof-of-concept

Bridge 4 — 5×3 Honest CV (the headline result)

Bridge 4 (BiGRU 791d over 20 segments) achieves 5×3 honest AUC = 0.9285 ± 0.0084 (95% CI [0.9246, 0.9331], n=15).

Architecture: BiGRU(128d, 2-layer, bidirectional) over 20 sequential segments
Features: 639 videos × 20 segments × 791d (768d WavLM-base-plus + 23d prosody)
Loss: BCE on per-video mean humor score (binarized at 0.5 for AUC)
CV: 3 seeds × 5 folds = 15 measurements, joke-disjoint
Pre-registered gates: ALL PASSED (AUC lower ≥ 0.80, n ≥ 15, per-fold std < 0.05)

The single-fold number (0.8422) was actually conservative — the 5×3 mean is 0.086 HIGHER. This is the opposite of the +0.163 pattern (where single-fold was inflated by 0.111).

v10 Multimodal — 5×3 Honest CV

Metric	Honest 5-fold speaker-disjoint	Notes
v10 humor AUC	0.6924 ± 0.0359	primary metric
v10 gold AUC	0.5453 ± 0.1313	σ across folds = 0.13

v1 Text-only Jester — 5×3 Honest CV

ρ = 0.2920 ± 0.0272 (95% CI [0.2783, 0.3057], 15 measurements)

Architecture: RoBERTa-base + Linear(768, 1) regression head
Data: SeppeV/rated_jokes_dataset_from_jester (50K subsample from 1.76M)
Pre-registered gate: ρ ≥ 0.35 AND CI excludes 0.30 — GATE NOT MET
Contribution: methodology paper (5×3 framework applied to continuous-regression humor tasks)

§1 Honest Disclosure (READ FIRST)

This card reflects a self-audit completed 2026-10-04 across three independent council reviews and two falsification experiments. Single-fold AUC 0.860 (v7) and 0.823 (v10) are speaker-identity leakage artifacts. Random-split CV inflated humor AUC by ~0.17 vs speaker-disjoint CV.

Other claims that were investigated and dropped:

❌ "AUC 0.860 (v7)" — single-fold leakage
❌ "AUC 0.823 / 0.613 (v10 single-fold)" — single-fold leakage
❌ "+0.163 gold AUC from per-language normalization" — FALSIFIED by 5×3 repeated CV; true effect: −0.127 (p=0.0004, 95% CI [−0.183, −0.078])
❌ "Reddit 100K pretraining helps" — humor +0.009, gold −0.045 (active harm on gold)
❌ "Architecture iterations v7→v10 matter" — σ_architecture ≈ 0 ≪ σ_fold = 0.13 (underspecification signature, D'Amour 2020)

§2 Bridge 4 — Best Model

Architecture: BiGRU(128d, 2-layer, bidirectional) over 20 sequential segments → MLP → per-video score.

Input: 791d per segment (768d WavLM-base-plus + 23d prosody)
Total params: ~790K
5×3 honest AUC: 0.9285 ± 0.0084
Inference time: <1ms per clip (CPU)

§3 v10 Cascade Gate — Multimodal with CORAL

Architecture: text confidence → gates audio → BiGRU → CORAL domain adaptation → sigmoid per-segment score.

Text features: 768-dim from DistilBERT
Audio features: 791-dim (768 WavLM-base-plus + 23 prosody)
Total params: ~1.07M
Inference: ~1.5ms (ONNX INT8)
5×3 honest: humor ρ = 0.6924, gold ρ = 0.5453

§4 Methodology: 5×3 Repeated CV

All numbers use 3 random seeds × 5 group-disjoint folds = 15 measurements, with 10,000-resample bootstrap 95% CI.

Fold-luck pattern (4× confirmed):

Date	Claim	Single-seed	5×3 honest	Δ
Sep 8	per-language +0.163	+0.163	-0.127	+0.290
Oct 7	v1 M3 ρ=0.3402	0.3402	0.2292 (frozen) / 0.2920 (fine-tuned)	+0.111 / +0.048
Oct 7	v10 single-fold	0.6924	(already 5×3 verified)	n/a

Rule: Never cite single-seed numbers on Jester or StandUp4AI. Always use 5×3 repeated CV with bootstrap CI.

§5 Files

File	Size	Description
`deployment/models/v10_cascade_int8.onnx`	3.26 MB	v10 multimodal ONNX (INT8 quantized)
`hf_release/v8_cascade_int8.onnx`	2.95 MB	v8 ONNX (superseded by v10)
`experiments/bridge4_5x3/result.json`	—	Bridge 4 5×3 honest CV result
`experiments/v1_jester_regression/5x3_cv_v6/v1_5x3cv_result.json`	—	v1 text-only 5×3 honest result
`arxiv_submission/hahascore.tex`	—	Methodology paper (5×3 framework)

§6 Citation

@misc{hahascore2026,
  title={Evaluating Speech-Humor Models Without Identity Leakage: A Self-Falsification Case Study},
  author={Das, Subhajit},
  year={2026},
  url={https://github.com/Das-rebel/HaHaScore},
  note={5×3 repeated speaker-disjoint CV with bootstrap 95% CI}
}

Downloads last month: -; Downloads are not tracked for this model. How to track

Hayasuki
/

hahascore-cascade