Feat/last sprintos (#23)

* skills

* fix readme

* readme track

* readme track

* experiment

* experiment and finetuning fix

* experiment and finetuning fix

* server finetuning common

* fix

* fix

---------

Co-authored-by: msgencrypted-auto <msgencrypted.auto@gmail.com>

README.md

@@ -10,8 +10,21 @@ python_version: "3.12"
 pinned: false
 license: apache-2.0
 tags:
+  - track:backyard
+  - track:wood
+  - sponsor:openbmb
+  - sponsor:openai
+  - sponsor:nvidia
+  - sponsor:modal
+  - achievement:offgrid
+  - achievement:welltuned
+  - achievement:offbrand
+  - achievement:llama
+  - achievement:sharing
+  - achievement:fieldnotes
   - build-small-hackathon
   - backyard-ai
+  - modal
   - tiny-titan
   - best-agent
   - best-demo
@@ -19,8 +32,14 @@ tags:
   - sharing-is-caring
   - off-the-grid
   - off-brand
+  - field-notes
+  - well-tuned
+  - llama-champion
+
 ---
 
+
+
 # Lesson Agent
 
 **Backyard AI** Gradio Space for the [Build Small Hackathon](https://huggingface.co/build-small-hackathon).

modal_math_gate_publish_result.json

@@ -0,0 +1,53 @@
+{
+  "job": "math-lora",
+  "baseline_results": "/vol/finetuned/results/lm_eval/minicpm5-1b__baseline__math/results.json",
+  "candidate_results": "/vol/finetuned/results/lm_eval/math-lora__math/results.json",
+  "gate": {
+    "passed": true,
+    "checks": [
+      {
+        "check": "gsm8k >= 0.05",
+        "value": 0.4,
+        "ok": true
+      },
+      {
+        "check": "gsm8k improve >= 0.02",
+        "value": 0.07,
+        "ok": true
+      },
+      {
+        "check": "arc_challenge regress <= 0.03",
+        "value": -0.04999999999999999,
+        "ok": true
+      },
+      {
+        "check": "hellaswag regress <= 0.03",
+        "value": 0.0,
+        "ok": true
+      },
+      {
+        "check": "piqa regress <= 0.03",
+        "value": 0.020000000000000018,
+        "ok": true
+      }
+    ],
+    "task": "gsm8k",
+    "candidate_score": 0.4,
+    "baseline_score": 0.33
+  },
+  "publish": {
+    "published": true,
+    "repo_id": "MSGEncrypted/minicpm5-1b-math-lora",
+    "url": "https://huggingface.co/MSGEncrypted/minicpm5-1b-math-lora",
+    "uploads": [
+      {
+        "repo_id": "MSGEncrypted/minicpm5-1b-math-lora",
+        "url": "https://huggingface.co/MSGEncrypted/minicpm5-1b-math-lora"
+      },
+      {
+        "repo_id": "build-small-hackathon/minicpm5-1b-math-lora",
+        "url": "https://huggingface.co/build-small-hackathon/minicpm5-1b-math-lora"
+      }
+    ]
+  }
+}

modal_science_gate_publish_result.json

@@ -0,0 +1,55 @@
+{
+  "job": "science-lora",
+  "baseline_results": "/vol/finetuned/results/lm_eval/minicpm5-1b__baseline__science/results.json",
+  "candidate_results": "/vol/finetuned/results/lm_eval/science-lora__science/results.json",
+  "gate": {
+    "passed": false,
+    "checks": [
+      {
+        "check": "sciq >= 0.5",
+        "value": 0.94,
+        "ok": true
+      },
+      {
+        "check": "sciq improve >= 0.02",
+        "value": -0.010000000000000009,
+        "ok": false
+      },
+      {
+        "check": "arc_challenge regress <= 0.03",
+        "value": -0.019999999999999962,
+        "ok": true
+      }
+    ],
+    "task": "sciq",
+    "candidate_score": 0.94,
+    "baseline_score": 0.95
+  },
+  "publish": {
+    "published": false,
+    "reason": "gate failed",
+    "gate": {
+      "passed": false,
+      "checks": [
+        {
+          "check": "sciq >= 0.5",
+          "value": 0.94,
+          "ok": true
+        },
+        {
+          "check": "sciq improve >= 0.02",
+          "value": -0.010000000000000009,
+          "ok": false
+        },
+        {
+          "check": "arc_challenge regress <= 0.03",
+          "value": -0.019999999999999962,
+          "ok": true
+        }
+      ],
+      "task": "sciq",
+      "candidate_score": 0.94,
+      "baseline_score": 0.95
+    }
+  }
+}

research/modal/README.md

@@ -60,7 +60,7 @@ uv sync --group modal   # local orchestration only
 ## Run training + benchmarks
 
 All commands from **repo root**. `finetune_app.py` runs the full **skill-matrix
-pipeline**: per-profile baseline lm-eval → finetune each job's QLoRA adapter →
+pipeline**: per-profile **base-model** baseline lm-eval (no adapter) → finetune each job's QLoRA adapter →
 post-train lm-eval vs. that baseline → check `goals` (gate) → publish to the
 Hugging Face Hub if the gate passes → pull adapter + results to your laptop.
 
@@ -119,7 +119,7 @@ for the `goals`/`publish` schema.
 | Flag | Default | Meaning |
 | ---- | ------- | ------- |
 | `--train` / `--no-train` | train on | Run finetune jobs |
-| `--eval-only` | off | Skip train + baselines; eval existing Volume checkpoints |
+| `--eval-only` | off | Skip train; eval existing Volume checkpoints (still runs missing base-model baselines) |
 | `--parallel` | off | `finetune_one.spawn()` per job instead of sequential |
 | `--job` | all jobs | Run one job name from `experiments.yaml` |
 | `--category` | all categories | Run all jobs with this `category` |
@@ -208,7 +208,7 @@ Task flags (`--job`, `--category`, `--cmd`, `--pipeline`, `--eval-only`, `--publ
 | `--category` | — | Run the skill-matrix pipeline for all jobs in a category |
 | `--pipeline` | off | Run the skill-matrix pipeline for all jobs |
 | `--max-steps` | from YAML | Override training steps |
-| `--eval-only` | off | Pipeline eval/gate/publish path only (skip baselines + train) |
+| `--eval-only` | off | Pipeline eval/gate/publish only (skip train; still runs missing base-model baselines) |
 | `--publish` / `--no-publish` | publish on | Push to `publish.hub_repo` if the gate passes |
 | `--publish-only` | off | Re-check the gate against existing results and publish (requires `--job`) |
 | `--pull` / `--no-pull` | pull on | `modal volume get` adapter + results after the pipeline |
@@ -469,6 +469,11 @@ goals:
       max_regress: 0.03
 ```
 
+Publishable jobs also run a **general** eval (`defaults.general_eval_profile`, default
+`compare_study`: arc_easy, arc_challenge, hellaswag, piqa, boolq, gsm8k) and must pass
+`defaults.general_goals` regression guards so skill tuning does not wash out general
+capability. The publish gate requires **both** skill `goals` and `general_goals` to pass.
+
 A job with no `goals` (e.g. `alpaca-lora`) is never gated and never published —
 it's local-only (still trained, evaluated, and pulled to your laptop).
 
@@ -482,10 +487,13 @@ publish:
 
 ### What happens on a passing gate
 
-1. `run_lm_eval` writes `results/lm_eval/<job>__<profile>/results.json`.
-2. `check_gate` compares it against `results/lm_eval/<preset>__baseline__<profile>/results.json`
-   using the `goals` above → `{"passed": bool, "checks": [...]}`.
-3. If `passed` and `publish` is set, `publish_adapter`:
+1. `run_lm_eval` writes skill results to `results/lm_eval/<job>__<profile>/results.json`.
+2. For publishable jobs, a second run writes general results to
+   `results/lm_eval/<job>__<general_eval_profile>/results.json`.
+3. `check_gate` compares skill results against `results/lm_eval/<preset>__baseline__<profile>/results.json`
+   and general results against `results/lm_eval/<preset>__baseline__<general_eval_profile>/results.json`
+   using `goals` + `general_goals` → `{"passed": bool, "skill": {...}, "general": {...}, "checks": [...]}`.
+4. If `passed` and `publish` is set, `publish_adapter`:
    - renders a model card (`README.md`) into the adapter directory — base model,
      gate checks table, full lm-eval baseline-vs-candidate-vs-delta table,
      training stats, and a PEFT load snippet

research/modal/_common.py

@@ -36,6 +36,8 @@ BASE_MODEL_ID = "openbmb/MiniCPM5-1B"
 
 BASELINE_EXPERIMENT = "minicpm5-1b__modal-baseline"
 BASELINE_RESULTS_JSON = f"{LM_EVAL_OUTPUT}/{BASELINE_EXPERIMENT}/results.json"
+# Shared general-capability profile for publish gates (limit 100; see compare_study).
+GENERAL_EVAL_PROFILE = "compare_study"
 
 # Metric keys to prefer when picking a task's "primary" score, in priority
 # order. Covers lm-eval-harness multiple-choice (acc), generation (exact_match),
@@ -87,6 +89,9 @@ image = (
 COMMON_ENV = {
     "TRUST_REMOTE_CODE": "true",
     "HF_HOME": HF_CACHE_PATH,
+    # Keep hf-xet logs off the HF cache Volume mount so volume.reload() is not
+    # blocked by open log file handles on warm containers.
+    "HF_XET_LOG_DEST": "/tmp/xet-logs/",
     "PYTORCH_CUDA_ALLOC_CONF": "expandable_segments:True",
 }
 
@@ -100,9 +105,24 @@ def repo_env() -> dict[str, str]:
     return {**os.environ, **COMMON_ENV}
 
 
-def reload_volumes() -> None:
+def _reload_volume_safe(vol: modal.Volume, *, label: str) -> None:
+    """Reload a Volume; skip (with warning) when open files block the operation."""
+    try:
+        vol.reload()
+    except (RuntimeError, modal.exception.ConflictError) as exc:
+        if "open files preventing the operation" in str(exc):
+            print(f"warning: skipping {label} volume reload ({exc})")
+            return
+        raise
+
+
+def reload_finetune_volume() -> None:
     finetune_vol.reload()
-    hf_cache_vol.reload()
+
+
+def reload_volumes() -> None:
+    reload_finetune_volume()
+    _reload_volume_safe(hf_cache_vol, label="hf-cache")
 
 
 def commit_volumes() -> None:
@@ -385,6 +405,105 @@ def job_gpu(job: dict[str, Any]) -> str:
     return job.get("gpu") or DEFAULT_GPU
 
 
+def job_needs_general_gate(job: dict[str, Any]) -> bool:
+    """Publishable jobs run a second general eval and must pass `general_goals`."""
+    return bool(job.get("goals") and job.get("publish"))
+
+
+def general_eval_profile(defaults: dict[str, Any]) -> str:
+    return defaults.get("general_eval_profile", GENERAL_EVAL_PROFILE)
+
+
+def general_goals_for_job(
+    job: dict[str, Any], defaults: dict[str, Any]
+) -> dict[str, Any] | None:
+    if not job_needs_general_gate(job):
+        return None
+    goals = job.get("general_goals") or defaults.get("general_goals")
+    return goals if goals else None
+
+
+def baseline_profiles_for_jobs(
+    jobs: list[dict[str, Any]], defaults: dict[str, Any]
+) -> list[str]:
+    profiles = {j.get("eval_profile", "compare_study") for j in jobs}
+    if any(job_needs_general_gate(j) for j in jobs):
+        profiles.add(general_eval_profile(defaults))
+    return sorted(profiles)
+
+
+def baseline_experiment_name(preset: str, profile: str) -> str:
+    """Volume path key for the unfine-tuned base model on a given eval profile."""
+    return f"{preset}__baseline__{profile}"
+
+
+def _load_models_registry() -> dict[str, Any]:
+    path = REPO_ROOT / "models.yaml"
+    if not path.is_file():
+        path = Path("/repo") / "models.yaml"
+    if not path.is_file():
+        return {}
+    with path.open() as f:
+        return yaml.safe_load(f) or {}
+
+
+def resolve_base_model_id(job: dict[str, Any], defaults: dict[str, Any]) -> str:
+    """Hub/path id of the base model this job fine-tunes — used as the eval baseline."""
+    explicit = job.get("model") or (job.get("args") or {}).get("model")
+    if explicit:
+        return str(explicit)
+    preset = job.get("preset", defaults.get("preset", "minicpm5-1b"))
+    entry = (_load_models_registry().get("models") or {}).get(preset) or {}
+    return entry.get("model_id") or BASE_MODEL_ID
+
+
+def discover_cached_baselines(
+    profile_names: list[str],
+    *,
+    preset: str,
+    eval_tasks: list[str] | None = None,
+    eval_limit: int | None = None,
+    eval_num_fewshot: int | None = None,
+    eval_seed: int | None = None,
+) -> dict[str, bool]:
+    """True per profile when base-model baseline results already exist on the Volume."""
+    cached: dict[str, bool] = {}
+    for profile in profile_names:
+        cached[profile] = baseline_is_cached(
+            baseline_experiment_name(preset, profile),
+            config_for_profile(profile),
+            tasks=eval_tasks,
+            limit=eval_limit,
+            num_fewshot=eval_num_fewshot,
+            seed=eval_seed,
+        )
+    return cached
+
+
+def profiles_needing_baseline_run(
+    profile_names: list[str],
+    cached: dict[str, bool],
+    *,
+    skip_baseline: bool,
+) -> list[str]:
+    if skip_baseline:
+        return []
+    return [profile for profile in profile_names if not cached.get(profile)]
+
+
+def eval_paths(
+    *,
+    job_name: str,
+    preset: str,
+    profile: str,
+) -> tuple[str, str, str]:
+    """Return (candidate_results_path, baseline_results_path, experiment_name)."""
+    exp_name = f"{job_name}__{profile}"
+    candidate = f"{LM_EVAL_OUTPUT}/{exp_name}/results.json"
+    baseline = f"{LM_EVAL_OUTPUT}/{baseline_experiment_name(preset, profile)}/results.json"
+    return candidate, baseline, exp_name
+
+
 def config_for_profile(profile: str) -> str:
     """Map an eval_profiles.yaml profile name to its config path (relative to repo root)."""
     with EVAL_PROFILES_PATH.open() as f:
@@ -462,7 +581,7 @@ def evaluate_gate(
     """Check a candidate's lm-eval results dict against `goals` (Hub publish gate).
 
     `goals` schema:
-        task: <lm-eval task name>       # scored via primary_metric(), same as summary.md
+        task: <lm-eval task name, optional when only guard_tasks are set>
         min_score: <float, optional>    # candidate score must be >= this
         min_improve: <float, optional>  # candidate - baseline must be >= this
         guard_tasks:                     # optional regression guards
@@ -482,9 +601,11 @@ def evaluate_gate(
     checks: list[dict[str, Any]] = []
     passed = True
 
-    task = goals["task"]
-    cand_score = _score(cand_tasks, task)
-    base_score = _score(base_tasks, task)
+    task = goals.get("task")
+    cand_score = base_score = None
+    if task:
+        cand_score = _score(cand_tasks, task)
+        base_score = _score(base_tasks, task)
 
     # Tolerance so a score landing exactly on a threshold (e.g. a clean +0.02
     # improvement stored as 0.0199999996) is not rejected by float epsilon.
@@ -576,6 +697,61 @@ def check_gate_files(
     return evaluate_gate(candidate=candidate, baseline=baseline, goals=goals)
 
 
+def check_publish_gate_files(
+    *,
+    skill_candidate_path: str,
+    skill_baseline_path: str | None,
+    skill_goals: dict[str, Any],
+    general_candidate_path: str | None = None,
+    general_baseline_path: str | None = None,
+    general_goals: dict[str, Any] | None = None,
+) -> dict[str, Any]:
+    """Gate on skill-specific eval plus optional general-capability eval."""
+    skill_gate = check_gate_files(
+        candidate_results_path=skill_candidate_path,
+        baseline_results_path=skill_baseline_path,
+        goals=skill_goals,
+    )
+    general_gate: dict[str, Any] | None = None
+    if general_goals:
+        if not general_candidate_path:
+            general_gate = {
+                "passed": False,
+                "checks": [
+                    {
+                        "check": "general eval results missing",
+                        "value": None,
+                        "ok": False,
+                    }
+                ],
+                "reason": "general candidate results path not provided",
+            }
+        else:
+            general_gate = check_gate_files(
+                candidate_results_path=general_candidate_path,
+                baseline_results_path=general_baseline_path,
+                goals=general_goals,
+            )
+
+    passed = skill_gate.get("passed") and (
+        general_gate is None or general_gate.get("passed")
+    )
+    checks = list(skill_gate.get("checks", []))
+    if general_gate:
+        for check in general_gate.get("checks", []):
+            checks.append({**check, "check": f"general: {check['check']}"})
+
+    return {
+        "passed": passed,
+        "checks": checks,
+        "skill": skill_gate,
+        "general": general_gate,
+        "task": skill_gate.get("task"),
+        "candidate_score": skill_gate.get("candidate_score"),
+        "baseline_score": skill_gate.get("baseline_score"),
+    }
+
+
 def render_model_card(
     *,
     job: dict[str, Any],
@@ -624,16 +800,36 @@ def render_model_card(
         "",
         "## Benchmark gate",
         "",
-        f"- eval profile: `{job.get('eval_profile')}`",
+        f"- skill eval profile: `{job.get('eval_profile')}`",
         f"- gate: {'**PASSED**' if gate_result.get('passed') else '**FAILED**'}",
         "",
-        "| check | value | result |",
-        "| --- | ---: | --- |",
     ]
-    for c in gate_result.get("checks", []):
-        lines.append(f"| {c['check']} | {_fmt(c['value'])} | {'pass' if c['ok'] else 'fail'} |")
-    if not gate_result.get("checks"):
-        lines.append("| — | — | — |")
+
+    def _gate_table(section: dict[str, Any] | None, *, prefix: str = "") -> list[str]:
+        if not section:
+            return []
+        out = [
+            f"### {prefix}checks".strip(),
+            "",
+            "| check | value | result |",
+            "| --- | ---: | --- |",
+        ]
+        for c in section.get("checks", []):
+            out.append(
+                f"| {c['check']} | {_fmt(c['value'])} | {'pass' if c['ok'] else 'fail'} |"
+            )
+        if not section.get("checks"):
+            out.append("| — | — | — |")
+        out.append("")
+        return out
+
+    skill_section = gate_result.get("skill") or gate_result
+    lines.extend(_gate_table(skill_section, prefix="Skill "))
+    if gate_result.get("general"):
+        gen_profile = job.get("general_eval_profile") or GENERAL_EVAL_PROFILE
+        lines.append(f"- general eval profile: `{gen_profile}`")
+        lines.append("")
+        lines.extend(_gate_table(gate_result["general"], prefix="General "))
 
     lines.extend(
         [

research/modal/experiments.yaml

@@ -5,6 +5,10 @@
 # checks the result against `goals`, and — only if the gate passes —
 # publishes the adapter to `publish.hub_repo` on the Hugging Face Hub.
 #
+# Baselines are always the unfine-tuned base model (`defaults.preset` →
+# models.yaml model_id, e.g. openbmb/MiniCPM5-1B), stored as
+# results/lm_eval/<preset>__baseline__<profile>/ and reused across jobs.
+#
 # Smoke limits (max_steps, max_samples, eval `limit` in the profile configs)
 # keep hackathon runs affordable; bump them for full runs.
 #
@@ -27,6 +31,22 @@ defaults:
   max_steps: 100
   # Hugging Face namespace for published adapters.
   hub_org: MSGEncrypted
+  # Second eval pass for publish gates: balanced general SLM mix (limit 100).
+  general_eval_profile: compare_study
+  general_goals:
+    guard_tasks:
+      - task: arc_easy
+        max_regress: 0.03
+      - task: arc_challenge
+        max_regress: 0.03
+      - task: hellaswag
+        max_regress: 0.03
+      - task: piqa
+        max_regress: 0.03
+      - task: boolq
+        max_regress: 0.03
+      - task: gsm8k
+        max_regress: 0.03
 
 finetune:
   # --- teaching: lesson-planning agent chat data (Well-Tuned primary) ---
@@ -62,14 +82,27 @@ finetune:
       private: false
 
   # --- science: factual + explanatory science tutoring ---
+  # 10 local science-tutor chats overfit easily on their own; mix in alpaca
+  # replay + NEFTune + bigger LoRA rank (same recipe as teaching/math/language)
+  # so sciq recall improves instead of regressing from narrow chat-format drift.
   - name: science-lora
     category: science
-    dataset: research/data/science-tutor-chat.jsonl
-    format: chat
+    max_steps: 150
+    mix:
+      - dataset: research/data/science-tutor-chat.jsonl
+        format: chat
+        weight: 16                    # ~10 samples -> ~160 examples
+      - dataset: tatsu-lab/alpaca      # general replay: protect sciq/arc_challenge
+        format: alpaca
+        dataset_split: "train[:600]"
+        max_samples: 600
     args:
+      lora_r: 32
+      lora_alpha: 64
+      neftune_noise_alpha: 5
       early_stopping_patience: 2   # keep best eval_loss checkpoint, not the last
       val_split: 0.05
-    description: Science tutor Q&A chat data (local)
+    description: Science tutor Q&A chat + alpaca replay, r=32 + NEFTune
     eval_profile: science
     goals:
       task: sciq

research/modal/finetune_app.py

@@ -36,16 +36,21 @@ for _candidate in (Path(__file__).resolve().parent, Path("/repo/research/modal")
         sys.path.insert(0, str(_candidate))
 
 from _common import (  # noqa: E402
-    BASE_MODEL_ID,
     FINETUNE_VOL_PATH,
     HF_CACHE_PATH,
     LM_EVAL_OUTPUT,
+    baseline_profiles_for_jobs,
     build_finetune_cmd,
     build_lm_eval_cmd,
     check_gate_files,
+    check_publish_gate_files,
     commit_volumes,
     config_for_profile,
+    discover_cached_baselines,
+    eval_paths,
     finetune_vol,
+    general_eval_profile,
+    general_goals_for_job,
     hf_cache_vol,
     hf_secret,
     image,
@@ -53,11 +58,15 @@ from _common import (  # noqa: E402
     job_plan_rows,
     parse_json_object,
     prepare_jobs,
+    profiles_needing_baseline_run,
+    resolve_base_model_id,
     split_csv,
     publish_adapter_files,
     pull_artifacts,
+    reload_finetune_volume,
     reload_volumes,
     repo_env,
+    baseline_experiment_name,
 )
 
 APP_NAME = "slm-finetune-benchmark"
@@ -118,7 +127,7 @@ def run_lm_eval(
     seed: int | None = None,
 ) -> dict[str, Any]:
     """Run slm-lm-eval on base model or finetuned checkpoint."""
-    reload_volumes()
+    reload_finetune_volume()
 
     if adapter_path:
         adapter_dir = Path(adapter_path)
@@ -183,9 +192,21 @@ def check_gate(
     candidate_results_path: str,
     baseline_results_path: str | None,
     goals: dict[str, Any],
+    general_candidate_results_path: str | None = None,
+    general_baseline_results_path: str | None = None,
+    general_goals: dict[str, Any] | None = None,
 ) -> dict[str, Any]:
-    """Check a candidate's lm-eval results against `goals` (Hub publish gate)."""
-    reload_volumes()
+    """Check skill + general lm-eval results against publish goals."""
+    reload_finetune_volume()
+    if general_goals:
+        return check_publish_gate_files(
+            skill_candidate_path=candidate_results_path,
+            skill_baseline_path=baseline_results_path,
+            skill_goals=goals,
+            general_candidate_path=general_candidate_results_path,
+            general_baseline_path=general_baseline_results_path,
+            general_goals=general_goals,
+        )
     return check_gate_files(
         candidate_results_path=candidate_results_path,
         baseline_results_path=baseline_results_path,
@@ -207,7 +228,7 @@ def publish_adapter(
     baseline_results_path: str | None,
 ) -> dict[str, Any]:
     """Write a model card and push the adapter to the Hub, but only if the gate passed."""
-    reload_volumes()
+    reload_finetune_volume()
     return publish_adapter_files(
         job=job,
         adapter_dir=adapter_dir,
@@ -284,17 +305,29 @@ def main(
         print(json.dumps({"preset": preset, "jobs": plan_rows}, indent=2))
         return
 
-    profile_names = sorted({j.get("eval_profile", "compare_study") for j in prepared})
+    profile_names = baseline_profiles_for_jobs(prepared, defaults)
 
-    baselines_ok: dict[str, bool] = {}
-    if not eval_only and not skip_baseline:
-        print(f"--- baselines ({', '.join(profile_names)}) ---")
-        for profile in profile_names:
+    eval_task_list = split_csv(eval_tasks)
+    baselines_ok = discover_cached_baselines(
+        profile_names,
+        preset=preset,
+        eval_tasks=eval_task_list,
+        eval_limit=eval_limit,
+        eval_num_fewshot=eval_num_fewshot,
+        eval_seed=eval_seed,
+    )
+    missing_baselines = profiles_needing_baseline_run(
+        profile_names, baselines_ok, skip_baseline=skip_baseline
+    )
+    if missing_baselines:
+        print(f"--- base-model baselines ({', '.join(missing_baselines)}) ---")
+        for profile in missing_baselines:
+            exp = baseline_experiment_name(preset, profile)
             result = run_lm_eval.remote(
-                experiment_name=f"{preset}__baseline__{profile}",
+                experiment_name=exp,
                 config=config_for_profile(profile),
                 preset=preset,
-                tasks=split_csv(eval_tasks),
+                tasks=eval_task_list,
                 limit=eval_limit,
                 num_fewshot=eval_num_fewshot,
                 batch_size=eval_batch_size,
@@ -304,6 +337,9 @@ def main(
             )
             print(json.dumps(result, indent=2))
             baselines_ok[profile] = bool(result.get("ok"))
+    elif any(baselines_ok.values()):
+        cached = [p for p in profile_names if baselines_ok.get(p)]
+        print(f"--- base-model baselines: reusing cached ({', '.join(cached)}) ---")
 
     train_results: dict[str, dict[str, Any]] = {}
     if train and not eval_only:
@@ -325,6 +361,7 @@ def main(
 
     print("--- post-train lm-eval / gate / publish ---")
     summary: list[dict[str, Any]] = []
+    gen_profile = general_eval_profile(defaults)
     for j in prepared:
         job_name = j["name"]
         profile = j.get("eval_profile", "compare_study")
@@ -333,17 +370,18 @@ def main(
             train_payload["output_dir"] if train_payload else f"{FINETUNE_VOL_PATH}/{job_name}"
         )
 
-        baseline_path = f"{LM_EVAL_OUTPUT}/{preset}__baseline__{profile}/results.json"
+        baseline_path = f"{LM_EVAL_OUTPUT}/{baseline_experiment_name(preset, profile)}/results.json"
         compare_to = baseline_path if baselines_ok.get(profile) else None
+        base_model_id = resolve_base_model_id(j, defaults)
 
         exp_name = f"{job_name}__{profile}"
         eval_result = run_lm_eval.remote(
             experiment_name=exp_name,
             config=config_for_profile(profile),
-            model_path=BASE_MODEL_ID,
+            model_path=base_model_id,
             adapter_path=adapter_path,
             compare_to=compare_to,
-            tasks=split_csv(eval_tasks),
+            tasks=eval_task_list,
             limit=eval_limit,
             num_fewshot=eval_num_fewshot,
             batch_size=eval_batch_size,
@@ -353,20 +391,58 @@ def main(
         )
         print(json.dumps(eval_result, indent=2))
 
+        general_goals = general_goals_for_job(j, defaults)
+        general_eval_result: dict[str, Any] | None = None
+        general_candidate_path: str | None = None
+        general_baseline_path: str | None = None
+        if general_goals:
+            general_baseline_path = (
+                f"{LM_EVAL_OUTPUT}/{baseline_experiment_name(preset, gen_profile)}/results.json"
+            )
+            gen_compare_to = (
+                general_baseline_path if baselines_ok.get(gen_profile) else None
+            )
+            gen_exp_name = f"{job_name}__{gen_profile}"
+            general_eval_result = run_lm_eval.remote(
+                experiment_name=gen_exp_name,
+                config=config_for_profile(gen_profile),
+                model_path=base_model_id,
+                adapter_path=adapter_path,
+                compare_to=gen_compare_to,
+                tasks=eval_task_list,
+                limit=eval_limit,
+                num_fewshot=eval_num_fewshot,
+                batch_size=eval_batch_size,
+                device=eval_device,
+                dtype=eval_dtype,
+                seed=eval_seed,
+            )
+            print(json.dumps(general_eval_result, indent=2))
+            general_candidate_path = general_eval_result["results_json"]
+
         row: dict[str, Any] = {
             "name": job_name,
             "category": j.get("category"),
             "profile": profile,
+            "general_profile": gen_profile if general_goals else None,
             "plan": next((p for p in plan_rows if p["name"] == job_name), None),
         }
 
         gate_result: dict[str, Any] | None = None
         if j.get("goals"):
-            if eval_result.get("ok"):
+            skill_ok = bool(eval_result.get("ok"))
+            general_ok = (
+                not general_goals
+                or bool(general_eval_result and general_eval_result.get("ok"))
+            )
+            if skill_ok and general_ok:
                 gate_result = check_gate.remote(
                     candidate_results_path=eval_result["results_json"],
                     baseline_results_path=baseline_path,
                     goals=j["goals"],
+                    general_candidate_results_path=general_candidate_path,
+                    general_baseline_results_path=general_baseline_path,
+                    general_goals=general_goals,
                 )
                 print(json.dumps(gate_result, indent=2))
             row["gate_passed"] = bool(gate_result and gate_result.get("passed"))
@@ -388,6 +464,8 @@ def main(
 
         if pull:
             pull_artifacts(job_name, exp_name)
+            if general_goals and general_eval_result:
+                pull_artifacts(job_name, f"{job_name}__{gen_profile}", dest="models/finetuned")
 
     _print_summary(summary)
 
@@ -404,14 +482,26 @@ def publish_only(job: str):
 
     preset = defaults.get("preset", "minicpm5-1b")
     profile = j.get("eval_profile", "compare_study")
+    gen_profile = general_eval_profile(defaults)
+    general_goals = general_goals_for_job(j, defaults)
     adapter_path = f"{FINETUNE_VOL_PATH}/{job}"
-    candidate_results_path = f"{LM_EVAL_OUTPUT}/{job}__{profile}/results.json"
-    baseline_results_path = f"{LM_EVAL_OUTPUT}/{preset}__baseline__{profile}/results.json"
+    candidate_results_path, baseline_results_path, _ = eval_paths(
+        job_name=job, preset=preset, profile=profile
+    )
+    general_candidate_path = None
+    general_baseline_path = None
+    if general_goals:
+        general_candidate_path, general_baseline_path, _ = eval_paths(
+            job_name=job, preset=preset, profile=gen_profile
+        )
 
     gate_result = check_gate.remote(
         candidate_results_path=candidate_results_path,
         baseline_results_path=baseline_results_path,
         goals=j["goals"],
+        general_candidate_results_path=general_candidate_path,
+        general_baseline_results_path=general_baseline_path,
+        general_goals=general_goals,
     )
     print(json.dumps(gate_result, indent=2))
 

research/modal/server_app.py

@@ -44,7 +44,6 @@ for _candidate in (Path(__file__).resolve().parent, Path("/repo/research/modal")
         sys.path.insert(0, str(_candidate))
 
 from _common import (  # noqa: E402
-    BASE_MODEL_ID,
     DEFAULT_GPU,
     DEFAULT_KEEPALIVE_HOURS,
     DEFAULT_SCALEDOWN_WINDOW,
@@ -52,22 +51,31 @@ from _common import (  # noqa: E402
     FINETUNE_VOL_PATH,
     HF_CACHE_PATH,
     LM_EVAL_OUTPUT,
-    baseline_is_cached,
+    baseline_experiment_name,
+    baseline_profiles_for_jobs,
     build_finetune_cmd,
     build_lm_eval_cmd,
     check_gate_files,
+    check_publish_gate_files,
     commit_volumes,
     config_for_profile,
+    discover_cached_baselines,
+    eval_paths,
     finetune_vol,
+    general_eval_profile,
+    general_goals_for_job,
     hf_cache_vol,
     hf_secret,
     image,
     job_plan_rows,
     parse_json_object,
     prepare_jobs,
+    profiles_needing_baseline_run,
+    resolve_base_model_id,
     split_csv,
     publish_adapter_files,
     pull_artifacts,
+    reload_finetune_volume,
     reload_volumes,
     repo_env,
 )
@@ -177,7 +185,9 @@ class GpuWorker:
         """Run slm-lm-eval on base model or finetuned checkpoint."""
         # Pick up adapters committed by another container (e.g. a separate
         # eval-only invocation) — the warm container's mount may predate them.
-        reload_volumes()
+        # Only finetune_vol is needed here; reloading hf-cache can fail when
+        # hf-xet keeps log files open on the warm container's HF cache mount.
+        reload_finetune_volume()
 
         if adapter_path:
             adapter_dir = Path(adapter_path)
@@ -241,8 +251,20 @@ class GpuWorker:
         candidate_results_path: str,
         baseline_results_path: str | None,
         goals: dict[str, Any],
+        general_candidate_results_path: str | None = None,
+        general_baseline_results_path: str | None = None,
+        general_goals: dict[str, Any] | None = None,
     ) -> dict[str, Any]:
-        """Check a candidate's lm-eval results against `goals` (Hub publish gate)."""
+        """Check skill + general lm-eval results against publish goals."""
+        if general_goals:
+            return check_publish_gate_files(
+                skill_candidate_path=candidate_results_path,
+                skill_baseline_path=baseline_results_path,
+                skill_goals=goals,
+                general_candidate_path=general_candidate_results_path,
+                general_baseline_path=general_baseline_results_path,
+                general_goals=general_goals,
+            )
         return check_gate_files(
             candidate_results_path=candidate_results_path,
             baseline_results_path=baseline_results_path,
@@ -313,40 +335,38 @@ class GpuWorker:
             raise ValueError("No jobs matched the requested filters")
 
         preset = defaults.get("preset", "minicpm5-1b")
-        profile_names = sorted({j.get("eval_profile", "compare_study") for j in prepared})
+        profile_names = baseline_profiles_for_jobs(prepared, defaults)
         plan = job_plan_rows(prepared)
         if plan_only:
             return {"preset": preset, "jobs": plan}
 
-        baselines_ok: dict[str, bool] = {}
-        if not eval_only and not skip_baseline:
-            for profile in profile_names:
-                exp = f"{preset}__baseline__{profile}"
-                cfg_path = config_for_profile(profile)
-                if baseline_is_cached(
-                    exp,
-                    cfg_path,
-                    tasks=eval_tasks,
-                    limit=eval_limit,
-                    num_fewshot=eval_num_fewshot,
-                    seed=eval_seed,
-                ):
-                    print(f"baseline {exp}: reusing cached results (config unchanged)")
-                    baselines_ok[profile] = True
-                    continue
-                result = self.lm_eval.local(
-                    experiment_name=exp,
-                    config=cfg_path,
-                    preset=preset,
-                    tasks=eval_tasks,
-                    limit=eval_limit,
-                    num_fewshot=eval_num_fewshot,
-                    batch_size=eval_batch_size,
-                    device=eval_device,
-                    dtype=eval_dtype,
-                    seed=eval_seed,
-                )
-                baselines_ok[profile] = bool(result.get("ok"))
+        baselines_ok = discover_cached_baselines(
+            profile_names,
+            preset=preset,
+            eval_tasks=eval_tasks,
+            eval_limit=eval_limit,
+            eval_num_fewshot=eval_num_fewshot,
+            eval_seed=eval_seed,
+        )
+        missing_baselines = profiles_needing_baseline_run(
+            profile_names, baselines_ok, skip_baseline=skip_baseline
+        )
+        for profile in missing_baselines:
+            exp = baseline_experiment_name(preset, profile)
+            cfg_path = config_for_profile(profile)
+            result = self.lm_eval.local(
+                experiment_name=exp,
+                config=cfg_path,
+                preset=preset,
+                tasks=eval_tasks,
+                limit=eval_limit,
+                num_fewshot=eval_num_fewshot,
+                batch_size=eval_batch_size,
+                device=eval_device,
+                dtype=eval_dtype,
+                seed=eval_seed,
+            )
+            baselines_ok[profile] = bool(result.get("ok"))
 
         train_results: dict[str, dict[str, Any]] = {}
         if train and not eval_only:
@@ -354,6 +374,7 @@ class GpuWorker:
                 train_results[j["name"]] = self.finetune.local(j)
 
         rows: list[dict[str, Any]] = []
+        gen_profile = general_eval_profile(defaults)
         for j in prepared:
             job_name = j["name"]
             profile = j.get("eval_profile", "compare_study")
@@ -364,14 +385,15 @@ class GpuWorker:
                 else f"{FINETUNE_VOL_PATH}/{job_name}"
             )
 
-            baseline_path = f"{LM_EVAL_OUTPUT}/{preset}__baseline__{profile}/results.json"
+            baseline_path = f"{LM_EVAL_OUTPUT}/{baseline_experiment_name(preset, profile)}/results.json"
             compare_to = baseline_path if baselines_ok.get(profile) else None
+            base_model_id = resolve_base_model_id(j, defaults)
 
             exp_name = f"{job_name}__{profile}"
             eval_result = self.lm_eval.local(
                 experiment_name=exp_name,
                 config=config_for_profile(profile),
-                model_path=BASE_MODEL_ID,
+                model_path=base_model_id,
                 adapter_path=adapter_path,
                 compare_to=compare_to,
                 tasks=eval_tasks,
@@ -383,21 +405,60 @@ class GpuWorker:
                 seed=eval_seed,
             )
 
+            general_goals = general_goals_for_job(j, defaults)
+            general_eval_result: dict[str, Any] | None = None
+            general_candidate_path: str | None = None
+            general_baseline_path: str | None = None
+            if general_goals:
+                general_baseline_path = (
+                    f"{LM_EVAL_OUTPUT}/{baseline_experiment_name(preset, gen_profile)}/results.json"
+                )
+                gen_compare_to = (
+                    general_baseline_path if baselines_ok.get(gen_profile) else None
+                )
+                gen_exp_name = f"{job_name}__{gen_profile}"
+                general_eval_result = self.lm_eval.local(
+                    experiment_name=gen_exp_name,
+                    config=config_for_profile(gen_profile),
+                    model_path=base_model_id,
+                    adapter_path=adapter_path,
+                    compare_to=gen_compare_to,
+                    tasks=eval_tasks,
+                    limit=eval_limit,
+                    num_fewshot=eval_num_fewshot,
+                    batch_size=eval_batch_size,
+                    device=eval_device,
+                    dtype=eval_dtype,
+                    seed=eval_seed,
+                )
+                general_candidate_path = general_eval_result["results_json"]
+
             row: dict[str, Any] = {
                 "name": job_name,
                 "category": j.get("category"),
                 "profile": profile,
+                "general_profile": gen_profile if general_goals else None,
                 "plan": next((p for p in plan if p["name"] == job_name), None),
                 "eval": eval_result,
             }
+            if general_eval_result:
+                row["general_eval"] = general_eval_result
 
             gate_result: dict[str, Any] | None = None
             if j.get("goals"):
-                if eval_result.get("ok"):
+                skill_ok = bool(eval_result.get("ok"))
+                general_ok = (
+                    not general_goals
+                    or bool(general_eval_result and general_eval_result.get("ok"))
+                )
+                if skill_ok and general_ok:
                     gate_result = self.check_gate.local(
                         candidate_results_path=eval_result["results_json"],
                         baseline_results_path=baseline_path,
                         goals=j["goals"],
+                        general_candidate_results_path=general_candidate_path,
+                        general_baseline_results_path=general_baseline_path,
+                        general_goals=general_goals,
                     )
                 row["gate"] = gate_result
 
@@ -515,14 +576,26 @@ def main(
 
         preset = defaults.get("preset", "minicpm5-1b")
         profile = j.get("eval_profile", "compare_study")
+        gen_profile = general_eval_profile(defaults)
+        general_goals = general_goals_for_job(j, defaults)
         adapter_path = f"{FINETUNE_VOL_PATH}/{job}"
-        candidate_results_path = f"{LM_EVAL_OUTPUT}/{job}__{profile}/results.json"
-        baseline_results_path = f"{LM_EVAL_OUTPUT}/{preset}__baseline__{profile}/results.json"
+        candidate_results_path, baseline_results_path, _ = eval_paths(
+            job_name=job, preset=preset, profile=profile
+        )
+        general_candidate_path = None
+        general_baseline_path = None
+        if general_goals:
+            general_candidate_path, general_baseline_path, _ = eval_paths(
+                job_name=job, preset=preset, profile=gen_profile
+            )
 
         gate_result = worker.check_gate.remote(
             candidate_results_path=candidate_results_path,
             baseline_results_path=baseline_results_path,
             goals=j["goals"],
+            general_candidate_results_path=general_candidate_path,
+            general_baseline_results_path=general_baseline_path,
+            general_goals=general_goals,
         )
         print(json.dumps(gate_result, indent=2))
 
@@ -570,6 +643,12 @@ def main(
         if pull:
             for row in result.get("jobs", []):
                 pull_artifacts(row["name"], f"{row['name']}__{row['profile']}")
+                if row.get("general_profile"):
+                    pull_artifacts(
+                        row["name"],
+                        f"{row['name']}__{row['general_profile']}",
+                        dest="models/finetuned",
+                    )
         return
 
     if serve:

research/modal/tests/test_modal_common.py

@@ -1,13 +1,23 @@
 from pathlib import Path
 
+import json
 import sys
 
 sys.path.insert(0, str(Path(__file__).resolve().parents[3]))
 
 from research.modal._common import (  # noqa: E402
+    COMMON_ENV,
+    baseline_experiment_name,
+    baseline_profiles_for_jobs,
     build_finetune_cmd,
     build_lm_eval_cmd,
+    check_publish_gate_files,
+    discover_cached_baselines,
+    evaluate_gate,
+    general_goals_for_job,
     prepare_jobs,
+    profiles_needing_baseline_run,
+    resolve_base_model_id,
     split_csv,
 )
 
@@ -74,3 +84,100 @@ def test_prepare_jobs_filters_and_applies_finetune_overrides():
 def test_split_csv_trims_empty_values():
     assert split_csv(" math, science ,,code ") == ["math", "science", "code"]
     assert split_csv(None) is None
+
+
+def _results(task_scores: dict[str, float]) -> dict:
+    return {
+        "results": {
+            task: {"acc,none": score, "acc_stderr,none": 0.01}
+            for task, score in task_scores.items()
+        }
+    }
+
+
+def test_evaluate_gate_guard_only_goals():
+    candidate = _results({"arc_easy": 0.5, "hellaswag": 0.4})
+    baseline = _results({"arc_easy": 0.52, "hellaswag": 0.41})
+    goals = {
+        "guard_tasks": [
+            {"task": "arc_easy", "max_regress": 0.03},
+            {"task": "hellaswag", "max_regress": 0.03},
+        ]
+    }
+    gate = evaluate_gate(candidate=candidate, baseline=baseline, goals=goals)
+    assert gate["passed"] is True
+    assert len(gate["checks"]) == 2
+
+
+def test_check_publish_gate_requires_both_skill_and_general(tmp_path):
+    skill_cand = tmp_path / "skill_cand.json"
+    skill_base = tmp_path / "skill_base.json"
+    general_cand = tmp_path / "general_cand.json"
+    general_base = tmp_path / "general_base.json"
+    skill_cand.write_text(
+        json.dumps(_results({"gsm8k": 0.4}))
+    )
+    skill_base.write_text(
+        json.dumps(_results({"gsm8k": 0.33}))
+    )
+    general_cand.write_text(
+        json.dumps(_results({"arc_easy": 0.5, "hellaswag": 0.4}))
+    )
+    general_base.write_text(
+        json.dumps(_results({"arc_easy": 0.52, "hellaswag": 0.41}))
+    )
+
+    gate = check_publish_gate_files(
+        skill_candidate_path=str(skill_cand),
+        skill_baseline_path=str(skill_base),
+        skill_goals={"task": "gsm8k", "min_improve": 0.02},
+        general_candidate_path=str(general_cand),
+        general_baseline_path=str(general_base),
+        general_goals={
+            "guard_tasks": [{"task": "arc_easy", "max_regress": 0.03}]
+        },
+    )
+    assert gate["passed"] is True
+    assert gate["skill"]["passed"] is True
+    assert gate["general"]["passed"] is True
+    assert any(c["check"].startswith("general:") for c in gate["checks"])
+
+
+def test_baseline_profiles_include_general_for_publishable_jobs():
+    _, jobs = prepare_jobs(job="math-lora")
+    defaults = {"general_eval_profile": "compare_study", "general_goals": {"guard_tasks": []}}
+    profiles = baseline_profiles_for_jobs(jobs, defaults)
+    assert "math" in profiles
+    assert "compare_study" in profiles
+
+
+def test_general_goals_only_for_publishable_jobs():
+    _, math_jobs = prepare_jobs(job="math-lora")
+    _, local_jobs = prepare_jobs(job="alpaca-lora")
+    defaults = {"general_goals": {"guard_tasks": [{"task": "piqa", "max_regress": 0.03}]}}
+    assert general_goals_for_job(math_jobs[0], defaults) is not None
+    assert general_goals_for_job(local_jobs[0], defaults) is None
+
+
+def test_resolve_base_model_id_from_preset():
+    _, jobs = prepare_jobs(job="math-lora")
+    defaults, job = {}, jobs[0]
+    assert resolve_base_model_id(job, defaults) == "openbmb/MiniCPM5-1B"
+
+
+def test_profiles_needing_baseline_run_respects_skip_and_cache():
+    cached = {"math": True, "compare_study": False}
+    assert profiles_needing_baseline_run(
+        ["math", "compare_study"], cached, skip_baseline=False
+    ) == ["compare_study"]
+    assert profiles_needing_baseline_run(
+        ["math", "compare_study"], cached, skip_baseline=True
+    ) == []
+
+
+def test_baseline_experiment_name_uses_preset():
+    assert baseline_experiment_name("minicpm5-1b", "math") == "minicpm5-1b__baseline__math"
+
+
+def test_common_env_redirects_xet_logs_off_hf_cache_volume():
+    assert COMMON_ENV["HF_XET_LOG_DEST"] == "/tmp/xet-logs/"