SocraTeach SFT run 2 (923 steps) — calibration twin for judge noise floor

README.md

@@ -0,0 +1,110 @@
+---
+base_model: allenai/OLMo-2-0425-1B-Instruct
+datasets:
+- meric533/socrateach-sft
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- lora
+- peft
+- transformers
+- socratic-tutoring
+- education
+- calibration
+---
+
+# SocraTeach SFT (run 2) — the calibration twin
+
+A second, independently trained LoRA adapter from the **same recipe** as
+[`meric533/socrateach-sft-olmo2-1b-lora`](https://huggingface.co/meric533/socrateach-sft-olmo2-1b-lora).
+Same base model, same data, same hyperparameters, same number of steps.
+
+It exists for one purpose: **the two together are a noise floor for LLM-judge pedagogy
+evaluation.** Score both with your rubric and the gap you measure is the variance your judge
+cannot distinguish from a real method difference, because there is no method difference here.
+
+## The pair
+
+| | run 1 | run 2 (this) |
+|---|---|---|
+| repo | [`socrateach-sft-olmo2-1b-lora`](https://huggingface.co/meric533/socrateach-sft-olmo2-1b-lora) | this one |
+| internal name | `impl2` / `checkpoint-923` | `impl2-rerun` |
+| lineage | earlier POC codebase | current codebase |
+
+They differ only in what a re-run differs in: seed state, PEFT 0.19.1 vs 0.20.0, and the
+training precision/hardware of the two environments. The LoRA config is otherwise identical,
+down to the same seven target modules.
+
+## Why the pair is worth scoring
+
+On **deterministic** metrics the two are nearly the same model:
+
+| metric | run 1 | run 2 | gap |
+|---|---|---|---|
+| Pedagogy NLL | 0.863 | 0.862 | 0.001 |
+| GSM8K hinted | 0.208 | 0.212 | 0.004 |
+| GSM8K bare | 0.468 | 0.456 | 0.012 |
+| Forward KL from base (no SI) | 0.150 | 0.150 | 0.000 |
+
+On an **8-dimension MRBench-style LLM judge**, scored blind in the same round on the same 40
+held-out contexts, they are not:
+
+| | run 1 | run 2 | gap |
+|---|---|---|---|
+| Judge OVERALL | 0.644 | 0.534 | **0.109** |
+
+Paired bootstrap over the 40 contexts puts that gap at **+0.109, 95% CI [+0.042, +0.181]** —
+it excludes zero, so it is a real difference between these two sets of weights, not a scoring
+artefact. For contrast, re-judging *run 1* in a second independent round moved it by **+0.009,
+95% CI [-0.017, +0.039]**, which includes zero: the judge itself is reproducible.
+
+The gap is concentrated in the discrete behavioural dimensions rather than the fluency ones:
+
+| dimension | run 1 | run 2 | Δ |
+|---|---|---|---|
+| Revealing_of_the_Answer | 0.600 | 0.300 | +0.300 |
+| Actionability | 0.500 | 0.200 | +0.300 |
+| Step_Level_Guidance | 0.425 | 0.175 | +0.250 |
+| Providing_Guidance | 0.713 | 0.713 | 0.000 |
+| Coherence | 0.775 | 0.875 | −0.100 |
+
+Two models with the same loss curve and the same NLL land in visibly different places on
+"does it withhold the answer and give a concrete next step". Whatever a pedagogy score is
+measuring, retraining moves it a lot more than the judge does.
+
+## Practical implication
+
+At n=40 contexts the paired-difference SD is 0.228, so a single comparison carries a 95% CI of
+roughly ±0.071. Resolving a 0.05 gap needs ~164 contexts; a 0.034 gap needs ~350. And that is
+for *fixed weights* — it says nothing about whether the gap survives retraining. If you are
+comparing methods rather than checkpoints, the run-to-run term above dominates, and a
+single-seed-per-arm comparison cannot separate a method effect from this.
+
+## Usage
+
+```python
+from peft import PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+base = "allenai/OLMo-2-0425-1B-Instruct"
+tok = AutoTokenizer.from_pretrained(base)          # the adapter ships no tokenizer
+model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
+model = PeftModel.from_pretrained(model, "meric533/socrateach-sft-olmo2-1b-lora-run2")
+```
+
+Tutoring behaviour is conditioned on a system instruction — prompt with a Socratic SI plus the
+student's problem. Judge both members of the pair under identical conditions or the comparison
+is meaningless.
+
+## Training
+
+LoRA r=16, α=32, dropout=0.05 on all attention and MLP projections. 1 epoch = 923 optimizer
+steps, LR 2e-4, warmup 3%, effective batch 32, max_len 1024, seed 13, assistant-token-only
+cross-entropy. Data: [`meric533/socrateach-sft`](https://huggingface.co/datasets/meric533/socrateach-sft).
+`trainer_state.json` has the full loss curve. Optimizer and RNG state are not uploaded, so this
+is usable for inference and as a KL reference but not to resume training.
+
+## Provenance
+
+AlphaAI / edu-llm P7 (tutor layer). Code:
+[`edu-llm/OLMo-core`, branch `p7/impl3`](https://github.com/edu-llm/OLMo-core/tree/p7/impl3).

adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "allenai/OLMo-2-0425-1B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "up_proj",
+    "down_proj",
+    "q_proj",
+    "gate_proj",
+    "o_proj",
+    "v_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e0d6d7b2ac943d954ddb4813f1fb64e24f50c51b463c06f4d7e39236d2a56cda
+size 48264184

trainer_state.json

@@ -0,0 +1,452 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 923,
+  "global_step": 923,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0010834236186348862,
+      "eval_loss": 1.44209623336792,
+      "eval_runtime": 2.6175,
+      "eval_samples_per_second": 76.41,
+      "eval_steps_per_second": 2.674,
+      "step": 1
+    },
+    {
+      "epoch": 0.0021668472372697724,
+      "eval_loss": 1.4417145252227783,
+      "eval_runtime": 2.4081,
+      "eval_samples_per_second": 83.053,
+      "eval_steps_per_second": 2.907,
+      "step": 2
+    },
+    {
+      "epoch": 0.0032502708559046588,
+      "eval_loss": 1.4389233589172363,
+      "eval_runtime": 2.4041,
+      "eval_samples_per_second": 83.19,
+      "eval_steps_per_second": 2.912,
+      "step": 3
+    },
+    {
+      "epoch": 0.004333694474539545,
+      "eval_loss": 1.4311391115188599,
+      "eval_runtime": 2.4089,
+      "eval_samples_per_second": 83.026,
+      "eval_steps_per_second": 2.906,
+      "step": 4
+    },
+    {
+      "epoch": 0.00866738894907909,
+      "eval_loss": 1.3003512620925903,
+      "eval_runtime": 2.4145,
+      "eval_samples_per_second": 82.831,
+      "eval_steps_per_second": 2.899,
+      "step": 8
+    },
+    {
+      "epoch": 0.01733477789815818,
+      "eval_loss": 1.148667335510254,
+      "eval_runtime": 2.4081,
+      "eval_samples_per_second": 83.053,
+      "eval_steps_per_second": 2.907,
+      "step": 16
+    },
+    {
+      "epoch": 0.021668472372697724,
+      "grad_norm": 0.3239409923553467,
+      "learning_rate": 0.00013571428571428572,
+      "loss": 1.2027952194213867,
+      "step": 20
+    },
+    {
+      "epoch": 0.03466955579631636,
+      "eval_loss": 1.023200273513794,
+      "eval_runtime": 2.4239,
+      "eval_samples_per_second": 82.51,
+      "eval_steps_per_second": 2.888,
+      "step": 32
+    },
+    {
+      "epoch": 0.04333694474539545,
+      "grad_norm": 0.2622278928756714,
+      "learning_rate": 0.0001999254658861464,
+      "loss": 1.016178512573242,
+      "step": 40
+    },
+    {
+      "epoch": 0.06500541711809317,
+      "grad_norm": 0.2845083773136139,
+      "learning_rate": 0.00019940854937432025,
+      "loss": 0.9705294609069824,
+      "step": 60
+    },
+    {
+      "epoch": 0.06933911159263272,
+      "eval_loss": 0.9649884104728699,
+      "eval_runtime": 2.4127,
+      "eval_samples_per_second": 82.894,
+      "eval_steps_per_second": 2.901,
+      "step": 64
+    },
+    {
+      "epoch": 0.0866738894907909,
+      "grad_norm": 0.28427502512931824,
+      "learning_rate": 0.0001984019002483041,
+      "loss": 0.9400932312011718,
+      "step": 80
+    },
+    {
+      "epoch": 0.10834236186348863,
+      "grad_norm": 0.2881655693054199,
+      "learning_rate": 0.00019691047772851873,
+      "loss": 0.9563480377197265,
+      "step": 100
+    },
+    {
+      "epoch": 0.13001083423618634,
+      "grad_norm": 0.27712777256965637,
+      "learning_rate": 0.0001949416292539326,
+      "loss": 0.9227810859680176,
+      "step": 120
+    },
+    {
+      "epoch": 0.13867822318526543,
+      "eval_loss": 0.9345716834068298,
+      "eval_runtime": 2.404,
+      "eval_samples_per_second": 83.194,
+      "eval_steps_per_second": 2.912,
+      "step": 128
+    },
+    {
+      "epoch": 0.15167930660888407,
+      "grad_norm": 0.2848430275917053,
+      "learning_rate": 0.00019250505428517026,
+      "loss": 0.9195150375366211,
+      "step": 140
+    },
+    {
+      "epoch": 0.1733477789815818,
+      "grad_norm": 0.27579906582832336,
+      "learning_rate": 0.00018961275652047138,
+      "loss": 0.916934585571289,
+      "step": 160
+    },
+    {
+      "epoch": 0.19501625135427952,
+      "grad_norm": 0.23910899460315704,
+      "learning_rate": 0.00018627898475990628,
+      "loss": 0.8925116539001465,
+      "step": 180
+    },
+    {
+      "epoch": 0.21668472372697725,
+      "grad_norm": 0.27521812915802,
+      "learning_rate": 0.00018252016270917925,
+      "loss": 0.8770346641540527,
+      "step": 200
+    },
+    {
+      "epoch": 0.23835319609967498,
+      "grad_norm": 0.2599611282348633,
+      "learning_rate": 0.00017835480806883784,
+      "loss": 0.9042178153991699,
+      "step": 220
+    },
+    {
+      "epoch": 0.2600216684723727,
+      "grad_norm": 0.25423887372016907,
+      "learning_rate": 0.00017380344130749244,
+      "loss": 0.9078985214233398,
+      "step": 240
+    },
+    {
+      "epoch": 0.27735644637053086,
+      "eval_loss": 0.9087494015693665,
+      "eval_runtime": 2.4089,
+      "eval_samples_per_second": 83.026,
+      "eval_steps_per_second": 2.906,
+      "step": 256
+    },
+    {
+      "epoch": 0.28169014084507044,
+      "grad_norm": 0.24491605162620544,
+      "learning_rate": 0.00016888848456847181,
+      "loss": 0.9030409812927246,
+      "step": 260
+    },
+    {
+      "epoch": 0.30335861321776814,
+      "grad_norm": 0.28875982761383057,
+      "learning_rate": 0.0001636341512079468,
+      "loss": 0.9008271217346191,
+      "step": 280
+    },
+    {
+      "epoch": 0.3250270855904659,
+      "grad_norm": 0.2819554805755615,
+      "learning_rate": 0.00015806632650870923,
+      "loss": 0.8830291748046875,
+      "step": 300
+    },
+    {
+      "epoch": 0.3466955579631636,
+      "grad_norm": 0.2702329456806183,
+      "learning_rate": 0.00015221244015726463,
+      "loss": 0.8976595878601075,
+      "step": 320
+    },
+    {
+      "epoch": 0.36836403033586135,
+      "grad_norm": 0.2778482735157013,
+      "learning_rate": 0.00014610133111247627,
+      "loss": 0.9124817848205566,
+      "step": 340
+    },
+    {
+      "epoch": 0.39003250270855905,
+      "grad_norm": 0.2824946939945221,
+      "learning_rate": 0.00013976310553147995,
+      "loss": 0.8905802726745605,
+      "step": 360
+    },
+    {
+      "epoch": 0.41170097508125675,
+      "grad_norm": 0.27247509360313416,
+      "learning_rate": 0.00013322898845279245,
+      "loss": 0.8893980979919434,
+      "step": 380
+    },
+    {
+      "epoch": 0.4333694474539545,
+      "grad_norm": 0.2737917900085449,
+      "learning_rate": 0.00012653116996729134,
+      "loss": 0.8675150871276855,
+      "step": 400
+    },
+    {
+      "epoch": 0.4550379198266522,
+      "grad_norm": 0.2629200518131256,
+      "learning_rate": 0.00011970264663489948,
+      "loss": 0.87319917678833,
+      "step": 420
+    },
+    {
+      "epoch": 0.47670639219934996,
+      "grad_norm": 0.27615392208099365,
+      "learning_rate": 0.00011277705892822892,
+      "loss": 0.899839973449707,
+      "step": 440
+    },
+    {
+      "epoch": 0.49837486457204766,
+      "grad_norm": 0.2700375020503998,
+      "learning_rate": 0.0001057885255040123,
+      "loss": 0.8819747924804687,
+      "step": 460
+    },
+    {
+      "epoch": 0.5200433369447454,
+      "grad_norm": 0.3007063567638397,
+      "learning_rate": 9.877147511877743e-05,
+      "loss": 0.879853630065918,
+      "step": 480
+    },
+    {
+      "epoch": 0.5417118093174431,
+      "grad_norm": 0.29464226961135864,
+      "learning_rate": 9.176047701682672e-05,
+      "loss": 0.8734842300415039,
+      "step": 500
+    },
+    {
+      "epoch": 0.5547128927410617,
+      "eval_loss": 0.885750412940979,
+      "eval_runtime": 2.4075,
+      "eval_samples_per_second": 83.073,
+      "eval_steps_per_second": 2.908,
+      "step": 512
+    },
+    {
+      "epoch": 0.5633802816901409,
+      "grad_norm": 0.29009002447128296,
+      "learning_rate": 8.479007062610832e-05,
+      "loss": 0.8596358299255371,
+      "step": 520
+    },
+    {
+      "epoch": 0.5850487540628385,
+      "grad_norm": 0.2693677544593811,
+      "learning_rate": 7.789459540097685e-05,
+      "loss": 0.8842451095581054,
+      "step": 540
+    },
+    {
+      "epoch": 0.6067172264355363,
+      "grad_norm": 0.2606968879699707,
+      "learning_rate": 7.11080216501164e-05,
+      "loss": 0.8765096664428711,
+      "step": 560
+    },
+    {
+      "epoch": 0.628385698808234,
+      "grad_norm": 0.26800337433815,
+      "learning_rate": 6.446378318304661e-05,
+      "loss": 0.8774471282958984,
+      "step": 580
+    },
+    {
+      "epoch": 0.6500541711809318,
+      "grad_norm": 0.2945432662963867,
+      "learning_rate": 5.799461259967216e-05,
+      "loss": 0.8764561653137207,
+      "step": 600
+    },
+    {
+      "epoch": 0.6717226435536294,
+      "grad_norm": 0.2884424924850464,
+      "learning_rate": 5.17323800343164e-05,
+      "loss": 0.8647214889526367,
+      "step": 620
+    },
+    {
+      "epoch": 0.6933911159263272,
+      "grad_norm": 0.26065170764923096,
+      "learning_rate": 4.570793614865971e-05,
+      "loss": 0.8515698432922363,
+      "step": 640
+    },
+    {
+      "epoch": 0.7150595882990249,
+      "grad_norm": 0.27795305848121643,
+      "learning_rate": 3.995096014707375e-05,
+      "loss": 0.8596259117126465,
+      "step": 660
+    },
+    {
+      "epoch": 0.7367280606717227,
+      "grad_norm": 0.29644423723220825,
+      "learning_rate": 3.448981356309692e-05,
+      "loss": 0.8723018646240235,
+      "step": 680
+    },
+    {
+      "epoch": 0.7583965330444203,
+      "grad_norm": 0.3121057450771332,
+      "learning_rate": 2.935140053736779e-05,
+      "loss": 0.884334945678711,
+      "step": 700
+    },
+    {
+      "epoch": 0.7800650054171181,
+      "grad_norm": 0.277951717376709,
+      "learning_rate": 2.456103527535184e-05,
+      "loss": 0.862095832824707,
+      "step": 720
+    },
+    {
+      "epoch": 0.8017334777898159,
+      "grad_norm": 0.288163959980011,
+      "learning_rate": 2.014231733782581e-05,
+      "loss": 0.8643115997314453,
+      "step": 740
+    },
+    {
+      "epoch": 0.8234019501625135,
+      "grad_norm": 0.2801090180873871,
+      "learning_rate": 1.611701537849739e-05,
+      "loss": 0.8651853561401367,
+      "step": 760
+    },
+    {
+      "epoch": 0.8450704225352113,
+      "grad_norm": 0.278440922498703,
+      "learning_rate": 1.2504959901521951e-05,
+      "loss": 0.8746981620788574,
+      "step": 780
+    },
+    {
+      "epoch": 0.866738894907909,
+      "grad_norm": 0.2619445025920868,
+      "learning_rate": 9.323945567242808e-06,
+      "loss": 0.8700389862060547,
+      "step": 800
+    },
+    {
+      "epoch": 0.8884073672806068,
+      "grad_norm": 0.3035391867160797,
+      "learning_rate": 6.5896435274422575e-06,
+      "loss": 0.8791361808776855,
+      "step": 820
+    },
+    {
+      "epoch": 0.9100758396533044,
+      "grad_norm": 0.26188817620277405,
+      "learning_rate": 4.315524221980505e-06,
+      "loss": 0.8619167327880859,
+      "step": 840
+    },
+    {
+      "epoch": 0.9317443120260022,
+      "grad_norm": 0.2701563537120819,
+      "learning_rate": 2.512791017162086e-06,
+      "loss": 0.8762255668640136,
+      "step": 860
+    },
+    {
+      "epoch": 0.9534127843986999,
+      "grad_norm": 0.3024737238883972,
+      "learning_rate": 1.1903250127579869e-06,
+      "loss": 0.8764994621276856,
+      "step": 880
+    },
+    {
+      "epoch": 0.9750812567713976,
+      "grad_norm": 0.27478569746017456,
+      "learning_rate": 3.5464128958965227e-07,
+      "loss": 0.8694242477416992,
+      "step": 900
+    },
+    {
+      "epoch": 0.9967497291440953,
+      "grad_norm": 0.3006316125392914,
+      "learning_rate": 9.856813219477624e-09,
+      "loss": 0.8638352394104004,
+      "step": 920
+    },
+    {
+      "epoch": 1.0,
+      "eval_loss": 0.8740995526313782,
+      "eval_runtime": 2.4054,
+      "eval_samples_per_second": 83.146,
+      "eval_steps_per_second": 2.91,
+      "step": 923
+    }
+  ],
+  "logging_steps": 20,
+  "max_steps": 923,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 923,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2.030253125660713e+17,
+  "train_batch_size": 32,
+  "trial_name": null,
+  "trial_params": null
+}