exp007_math: math anchors, 4 ask-methods — format lock-in 4/4, no correctness tax, held-out f(x) 0.688->0.938; sep peak 0.496 @L23

exp007_math/README.md

@@ -0,0 +1,200 @@
+# exp007_math — one adapter stack, four ways of asking for a computation: format locks in with no correctness tax, and held-out function evaluation improves (0.688 → 0.938)
+
+**Question.** exp006 left a sharp split on this trunk: the `relay_pw` anchor
+stack locked in every *formal* register (continuation shape, instruct shape,
+tool-call JSON all → 1.0) but a *semantic* constraint — "use these three
+keywords" — did not move at all (0.083 frozen, 0.083 trained). So what does
+the anchor paradigm actually carry: only the **shape** of answers, or
+**computation** itself? This bed asks one arithmetic question four different
+ways and judges FORMAT separately from CORRECTNESS, with a frozen-trunk
+baseline to separate "the trunk can already add" from what the anchor adds,
+and a problem-hash holdout so that held-out correctness is unseen-instance
+computation, not recall.
+
+**Answer.** Both. Format locks in everywhere — all four ask-methods reach
+format 1.0, repairing the one register the frozen trunk asks wrong (`nl`
+answers correctly but not in the asked shape, format 0.312) — and
+**correctness never degrades** on any register. Where correctness had
+headroom it *improves*: code-register held-out `f(x)` evaluation goes
+**0.688 → 0.938**, on held-out questions whose exact form and target never
+appear in training — function evaluation improving, not target recall (see
+claim 2 for the exact unseen-instance accounting). Cross-referencing
+exp006 gives the interpretation (not a ledger assertion): the LM loss
+teaches exactly what **target-token predictability** demands. Answer tokens
+are unpredictable without using the question, so correctness moves; exp006's
+keywords were never needed to predict a plausible story, so compliance never
+moved. And the four ask-methods are held apart in sign-code space more
+strongly than any register set ledgered in this line so far: separation
+**0.4711 @ L16 / 0.4957 @ L23**.
+
+## The design
+
+**Registers (the four ask-methods).** Every problem is rendered exactly one
+way, deterministically assigned:
+
+| register | system instruction asks for | example user turn | example target |
+|---|---|---|---|
+| `nl` | equation + result | "What is 37 + 45?" | `37 + 45 = 82` |
+| `json` | `emit_answer` tool call | "Compute 37 + 45." | `{"expression": "37 + 45", "result": 82}` |
+| `code` | value of a defined Python function | `def f(x): return 3 * x + 2` … "What is f(4)?" | `f(4) = 14` |
+| `word` | one short sentence | "Sam has 37 apples. Sam gets 45 more. …" | `Sam has 82 apples.` |
+
+**Problem space.** Fully synthetic — no downloads, ground truth computable:
+`a + b` (1–99), `a − b` (result ≥ 0), `a * b` (2–12), and `f(x) = a·x + b`
+with a 2–9, b 0–20, x 2–9. 4,000 unique problems; `f(x)` problems always
+land in the `code` register (the other kinds round-robin across registers,
+with non-`f(x)` problems in `code` rendered as a one-line function).
+**Holdout is by hash of the rendered question (10%)** — a held-out
+problem's exact question and target never appear in training. For
+`f(x) = a·x + b` problems the `(a, b, x)` combination is additionally
+unseen anywhere in the corpus in any register; a non-`f(x)` problem's
+underlying `a op b` fact *can* appear in training under a different
+register (audited and accounted in claim 2).
+
+**Arms.** `frozen` (the untouched trunk, judged with the same prompts) and
+`relay_pw_math`: one exp002-certified `relay_pw` (RelayPatchwork) adapter
+per block of a fully frozen Qwen2.5-0.5B-Instruct trunk — the identical
+adapter layout as exp006 — trained on **all four registers interleaved**
+uniformly per step. 3000 steps, block 512, batch 4, pure Adam lr 1e-3,
+weight decay 0, seed 0. Greedy decoding at eval, max 96 new tokens.
+
+**Judges.** The judge splits format from correctness, per register,
+n=16/register, both arms. A generation must close (emit `<|im_end|>` or
+stop) or it fails both. Then:
+
+| register | FORMAT passes if | CORRECT passes if |
+|---|---|---|
+| `nl`, `code` | body contains `=` | last integer in body == ground truth |
+| `json` | tool call parses with both keys (`expression`, `result`) | `int(result)` == ground truth |
+| `word` | body ≤ 24 words | last integer in body == ground truth |
+
+For `nl`/`code`/`word` the body must also contain at least one integer,
+else both format and correctness fail. Plus per-register held-out
+perplexity, and the register probe carried from
+exp003/exp006: sign-code separation = mean inter-register Hamming distance
+− mean intra-register Hamming distance of the signed winner codes at blocks
+0 / 8 / 16 / 23 (the frozen arm has no adapters, hence no probe).
+
+## Results (all from `results/ledger.jsonl`, seed 0)
+
+**Frozen vs trained, per register** — format, correctness, held-out ppl:
+
+| register | format (frozen → trained) | correct (frozen → trained) | held-out ppl (frozen → trained) |
+|---|---|---|---|
+| `nl` | 0.312 → **1.0** | 1.0 → 1.0 | 2.763 → 1.342 |
+| `json` | 1.0 → 1.0 | 1.0 → 1.0 | 4.393 �� 1.061 |
+| `code` | 1.0 → 1.0 | 0.688 → **0.938** | 2.591 → 1.168 |
+| `word` | 1.0 → 1.0 | 0.938 → **1.0** | 3.145 → 1.525 |
+
+The frozen trunk is already a competent zero-shot calculator at these
+operand sizes (`nl`/`json` correctness 1.0) — its failures are one register
+asked in the wrong shape (`nl`: "45" or "The equation for 94 - 27 is 67."
+instead of "94 - 27 = 67") and genuine computation misses on `f(x)`
+evaluation (0.688) and word problems (0.938). Training repairs the format
+failure completely and moves both correctness gaps up, degrading nothing.
+
+**Register separation** (trained arm; signed winner codes across the four
+ask-methods):
+
+| tap | inter | intra | sep |
+|---|---|---|---|
+| L0 | 0.2563 | 0.0418 | 0.2145 |
+| L8 | 0.3293 | 0.0485 | 0.2808 |
+| L16 | 0.8737 | 0.4026 | **0.4711** |
+| L23 | 0.9304 | 0.4347 | **0.4957** |
+
+Separation grows monotonically with depth and the peak (0.4957 @ L23) is
+the deepest ledgered in this line so far: exp006's four story registers
+reached 0.3476 / 0.3451 at L16 / L23 and peaked at 0.4619 @ L8 (where this
+run is *lower*, 0.2808), and exp003's instruct tasks stayed ≤ 0.304 at
+L16 / L23 (all cited from their ledgers). Four surface forms of the *same
+underlying computation* are held further apart in sign-code space, at its
+peak, than either previous register set at theirs — with the depth
+*profile* differing across beds.
+
+## What this adds to the line
+
+**Claim 1 — format lock-in, with no correctness tax.** All four registers
+reach format 1.0 (from a frozen `nl` format of 0.312), and correctness never
+degrades on any register while it happens: `nl` and `json` hold 1.0, `word`
+moves 0.938 → 1.0. Locking the ask-shape did not cost the anchor any of the
+trunk's arithmetic — the failure mode where format training taxes semantics
+did not appear at this scale.
+
+**Claim 2 — computation moves through the adapters.** Code-register
+held-out correctness improves 0.688 → 0.938 on 16 judged problems, every
+one rendered as an `f(x)` question whose exact form and target string never
+appear in training. Exact composition (audited from the deterministic
+generator, seed 0): 5 of the 16 are true `f(x) = a·x + b` problems whose
+`(a, b, x)` combinations appear **nowhere** in the corpus in any register —
+for those, improvement is unseen-instance function evaluation with no
+recall channel at all. The other 11 wrap an `a op b` problem as a one-line
+function; the question and target are still unseen, but for 4 of the 11
+the underlying arithmetic fact appears in training under a *different*
+register, so cross-register fact transfer is a possible channel for those
+four. The improvement therefore cannot be target-string recall, and for the
+`a·x + b` subset it cannot be recall of any kind. The ledgered frozen
+baseline is what makes the sentence honest: the trunk already computes most
+of these, and the anchor's contribution is the measured delta on the part
+it couldn't.
+
+**Claim 3 — the predictability principle (cross-experiment interpretation,
+not a ledger assertion).** Placed next to exp006, this run completes a
+clean contrast on the same trunk, adapter, recipe, and step count: math
+correctness improved where exp006's keyword compliance (0.083 → 0.083) did
+not. The parsimonious reading: **the LM loss teaches exactly what
+target-token predictability demands.** Here the answer tokens ("= 82") are
+unpredictable unless the model actually uses the question, so lowering the
+loss forces computation. In exp006 the target stories were predictable
+as stories without honoring the keyword instruction, so nothing pushed
+compliance. This is an interpretation cross-referenced against exp006's
+ledger, not a number asserted from this one; the exp008-class test it
+suggests is a constraint whose target tokens are unpredictable without
+obeying it.
+
+**Claim 4 — register separation, deepest peak yet.** Sign-code separation
+across the four ask-methods reaches 0.4711 @ L16 and 0.4957 @ L23 — above
+exp006's story registers at those taps (0.3476 / 0.3451) and above exp006's
+own peak (0.4619 @ L8), and above exp003's instruct tasks (≤ 0.304 at
+L16 / L23). The depth profile is bed-dependent — exp006 peaked shallow, this
+run at L8 sits at 0.2808 — but the peak itself is the deepest ledgered in
+the line. The addressing surface differentiates *how a thing is asked* even
+when *what is computed* is identical — and here does so most strongly deep
+in the trunk.
+
+**Caveats.** 1 seed; n=16/register; operands small (2-digit; a*b up to
+12x12; f(x)=ax+b a 2-9 b 0-20 x 2-9); zero-shot correctness already high
+(nl/json 1.0) so headroom was format + code; of the 16 judged code
+problems only 5 are true a·x+b instances and 4 of the remaining 11 share
+their underlying arithmetic fact with another register's training text
+(claim 2 accounting); register-separation depth profiles differ across
+beds, so "deepest yet" is a claim about the peak, not every tap.
+
+## Files
+
+- `qwen_exp007_math.py` — the bed: deterministic problem generator,
+  four-register renderer, format/correctness judges, held-out ppl, the
+  campaign (`run_exp007`), CPU smoke.
+- `geolip_vitals.py` / `ar_differentiation_bed.py` /
+  `exp013_augmentation_bed.py` / `qwen_exp001_relay.py` /
+  `qwen_exp002_refine.py` / `qwen_exp003_instruct.py` /
+  `qwen_exp006_story.py` — the paste-order harness (trunk wrapper,
+  `relay_pw`, tool-call parser, arm loader and register probe; identical to
+  the copies shipped with exp006).
+- `repro.py` — smoke / `--run` (full campaign).
+- `build_results.py` — re-asserts every README claim from the ledger.
+- `results/ledger.jsonl` — both rows: frozen baseline + trained arm.
+- `adapters/q7_relay_pw_math_s0.pt` — the trained adapter stack.
+
+## Repro
+
+```bash
+pip install torch transformers
+python repro.py          # CPU smoke: problem gen + format/correctness
+                         # judges exercised in both directions
+python repro.py --run    # full campaign (GPU-only verdicts)
+```
+
+Data and caches land in `./data` (override with `GEOLIP_DATA`). The corpus
+is fully synthetic — no dataset downloads; ground truth is computed, so the
+judges need no reference model.

exp007_math/adapters/q7_relay_pw_math_s0.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bc8650497885ea506f2d29ef644f8a8e32557030cf403cdf5f604b5d4df23783
+size 22168167

exp007_math/ar_differentiation_bed.py

@@ -0,0 +1,487 @@
+"""ar_differentiation_bed.py — exp012: autoregressive differentiation of the aleph.
+
+Differentiation is cultivated by PREDICTIVE pressure along the sequence — the
+address parameterizing the next-byte distribution (Law 2: chain-rule advantage pays
+ONLY where the composed address directly parameterizes the predictive distribution).
+This bed puts the aleph in the autoregressive gradient path and measures what
+differentiates. The head arms enforce the employment law at its maximum: the
+ENTIRE next-byte distribution is parameterized by the address.
+
+Byte-level causal LM on wikitext-2-raw (HF parquet, CDN-fast), block 256. ARMS:
+  sdpa       — standard causal transformer control (matched trunk).
+  hub        — attention replaced by CAUSAL HUB: linear attention whose feature map
+               is the 2K-oriented aleph address, prefix-sum memories (no selection
+               event; O(n*K*d)). Differentiation cultivated INSIDE attention.
+  addr_head  — sdpa trunk, but the OUTPUT HEAD reads ONLY the signed aleph
+               coefficient vector w_k = sinh(u_k)/sum_j cosh(u_j) of the final
+               hidden state (K -> 256 logits). The address MUST carry every bit of
+               next-byte information — the hardest Law-2 bottleneck.
+
+JUDGED BY: val bits-per-byte per arm (task) + CULTIVATION VITALS on every aleph
+codebook (readouts, never losses): axis aliveness/hppl, drift-from-init +
+binding fraction @0.29154, winner-|cos| saturation (sign-code emergence), shadow
+path diversity (fixed high-bits hash). Never by recon.
+
+Riders: pure Adam wd=0; no BN/Dropout/GAP on geometric paths; orthogonal init;
+Colab-cell-safe (paste-ahead imports, no bare argparse, no __file__ reliance);
+GPU-only for verdict runs.
+
+Terminal:  python ar_differentiation_bed.py            # shapes/parse smoke
+           python ar_differentiation_bed.py --train    # verdict run
+Colab:     paste geolip_vitals.py cell, then this file (smoke auto-runs),
+           then  train(steps=2000, data_root="/content/data")  in the next cell.
+"""
+from __future__ import annotations
+import math
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+if "anchor_drift" not in globals():
+    try:
+        from geolip_vitals import anchor_drift, axis_aliveness, path_diversity
+    except ImportError:
+        _here = globals().get("__file__")
+        if _here is not None:
+            import sys, pathlib
+            sys.path.insert(0, str(pathlib.Path(_here).parent))
+            from geolip_vitals import anchor_drift, axis_aliveness, path_diversity
+        else:
+            raise ImportError(
+                "geolip_vitals not found — paste/run its cell first, or "
+                "hf_hub_download exp012_ar/geolip_vitals.py from "
+                "AbstractPhil/geolip-aleph-differentiation.")
+
+VOCAB = 256  # bytes
+
+
+# ------------------------------------------------------------------ aleph address
+def _super_fibonacci_s3(n: int) -> torch.Tensor:
+    """Near-uniform unit quaternions (Alexa CVPR'22) —
+    starts the codebook INSIDE the RP^3 attractor basin. D=4 only."""
+    PHI, PSI = math.sqrt(2.0), 1.533751168755204288118041
+    i = torch.arange(n, dtype=torch.float64)
+    s = (i + 0.5) / n
+    r, R = torch.sqrt(s), torch.sqrt(1.0 - s)
+    a, b = 2 * math.pi * i / PHI, 2 * math.pi * i / PSI
+    q = torch.stack([r * torch.sin(a), r * torch.cos(a),
+                     R * torch.sin(b), R * torch.cos(b)], dim=-1)
+    return F.normalize(q, dim=-1).float()
+
+
+class AlephAddress(nn.Module):
+    """Closed-form aleph over 2K oriented half-axes (aleph-void article).
+    signed(x): (..., K) w_k = sinh(u_k)/sum_j cosh(u_j)   — the Law-2 head feature.
+    oriented(x): ((..., K), (..., K)) positive halves of the 2K softmax — HUB map."""
+
+    def __init__(self, K: int, D: int, tau: float = 0.1, init: str = "random"):
+        super().__init__()
+        self.K, self.D, self.tau = K, D, tau
+        if init == "fibonacci":
+            assert D == 4, "fibonacci init lives on S^3 (D=4)"
+            A = _super_fibonacci_s3(K)
+        else:
+            A = F.normalize(torch.randn(K, D), dim=-1)
+        self.codebook = nn.Parameter(A)
+        self.register_buffer("home", self.codebook.detach().clone())
+
+    def _u(self, x):
+        A = F.normalize(self.codebook, dim=-1)
+        return (F.normalize(x, dim=-1) @ A.transpose(-1, -2)) / self.tau
+
+    def oriented(self, x):
+        u = self._u(x)
+        m = u.abs().amax(dim=-1, keepdim=True)
+        ep, en = torch.exp(u - m), torch.exp(-u - m)
+        Z = (ep + en).sum(dim=-1, keepdim=True)
+        return ep / Z, en / Z
+
+    def signed(self, x):
+        u = self._u(x)
+        m = u.abs().amax(dim=-1, keepdim=True)
+        ep, en = torch.exp(u - m), torch.exp(-u - m)
+        return (ep - en) / (ep + en).sum(dim=-1, keepdim=True)
+
+    def signed_at(self, x, taus):
+        """Multi-tau stroboscope (rule of 3): signed coefficients at several
+        temperatures, concatenated — softer taus keep the vector dense while a
+        hard tau supplies the sign-code sharpness. v2 refinement (b)."""
+        A = F.normalize(self.codebook, dim=-1)
+        cos = F.normalize(x, dim=-1) @ A.transpose(-1, -2)
+        outs = []
+        for t in taus:
+            u = cos / t
+            m = u.abs().amax(dim=-1, keepdim=True)
+            ep, en = torch.exp(u - m), torch.exp(-u - m)
+            outs.append((ep - en) / (ep + en).sum(dim=-1, keepdim=True))
+        return torch.cat(outs, dim=-1)
+
+    def m_hat(self, x):
+        """Closed-form soft read (decoders read M_hat, never M). v2 control (c)."""
+        u = self._u(x)
+        m = u.abs().amax(dim=-1, keepdim=True)
+        ep, en = torch.exp(u - m), torch.exp(-u - m)
+        A = F.normalize(self.codebook, dim=-1)
+        return ((ep - en) @ A) / (ep + en).sum(dim=-1, keepdim=True)
+
+    def m_hard_ste(self, x):
+        """Hard mode (aleph-void article): M_hard = sign(cos_win) * A[win], straight-through to
+        the soft read — forward fully discrete SIGN CODE, backward soft gradient.
+        Legal per theme A (reconstructive sign code, not a one-hot roster pick)."""
+        u = self._u(x)
+        soft = self.m_hat(x)
+        win = u.abs().argmax(dim=-1)
+        A = F.normalize(self.codebook, dim=-1)
+        sign = torch.sign(torch.gather(u, -1, win.unsqueeze(-1))).squeeze(-1)
+        hard = sign.unsqueeze(-1) * A[win]
+        return hard + soft - soft.detach()
+
+    @torch.no_grad()
+    def vitals(self, x_sample) -> dict:
+        u = self._u(x_sample.reshape(-1, x_sample.shape[-1]))
+        p, n = self.oriented(x_sample.reshape(-1, x_sample.shape[-1]))
+        two_k = torch.cat([p, n], dim=-1)
+        win = two_k.argmax(dim=-1)
+        cos_win = (u.abs().amax(dim=-1) * self.tau)  # winner |cos| — sign-code sat.
+        d = anchor_drift(self.codebook, self.home)
+        return {"drift": round(d["mean"], 4),
+                "binding_frac": round(d["binding_fraction"], 4),
+                "aliveness": axis_aliveness(two_k),
+                "win_cos_mean": round(cos_win.mean().item(), 4),
+                "paths": path_diversity(win)}
+
+
+# ------------------------------------------------------------------------- blocks
+class CausalSDPA(nn.Module):
+    def __init__(self, d: int, heads: int = 4):
+        super().__init__()
+        self.h = heads
+        self.qkv = nn.Linear(d, 3 * d, bias=False)
+        self.o = nn.Linear(d, d, bias=False)
+        nn.init.orthogonal_(self.qkv.weight); nn.init.orthogonal_(self.o.weight)
+
+    def forward(self, x):
+        B, n, d = x.shape
+        q, k, v = self.qkv(x).chunk(3, dim=-1)
+        q, k, v = (t.view(B, n, self.h, d // self.h).transpose(1, 2) for t in (q, k, v))
+        y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
+        return self.o(y.transpose(1, 2).reshape(B, n, d))
+
+
+class CausalHUB(nn.Module):
+    """Causal aleph linear attention: prefix-sum memories over the two K-wide
+    halves of the oriented address; 2K never materialized; no selection event."""
+
+    def __init__(self, d: int, K: int = 32, D: int = 4, tau: float = 0.1):
+        super().__init__()
+        self.addr = AlephAddress(K, D, tau)
+        self.q = nn.Linear(d, D, bias=False)
+        self.k = nn.Linear(d, D, bias=False)
+        self.v = nn.Linear(d, d, bias=False)
+        self.o = nn.Linear(d, d, bias=False)
+        for m in (self.q, self.k, self.v, self.o):
+            nn.init.orthogonal_(m.weight)
+
+    def forward(self, x):
+        qp, qn = self.addr.oriented(self.q(x))          # (B, n, K)
+        kp, kn = self.addr.oriented(self.k(x))
+        v = self.v(x)                                    # (B, n, d)
+        Sp = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kp, v), dim=1)
+        Sn = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kn, v), dim=1)
+        zp = torch.cumsum(kp, dim=1)
+        zn = torch.cumsum(kn, dim=1)
+        num = torch.einsum("bnk,bnkd->bnd", qp, Sp) + torch.einsum("bnk,bnkd->bnd", qn, Sn)
+        den = (qp * zp).sum(-1, keepdim=True) + (qn * zn).sum(-1, keepdim=True)
+        return self.o(num / den.clamp_min(1e-12))
+
+
+class MslRelay(nn.Module):
+    """Depth-composition unit (chain-rule probe): multi-slot M_hat read entering
+    the trunk as a NEAR-ZERO gated residual (gate init -3.0, sigma~0.047 — theme D:
+    geometry enters as a nudge and grows only if it earns gradient)."""
+
+    def __init__(self, d: int, n_slots: int = 16, K: int = 64):
+        super().__init__()
+        self.n_slots = n_slots
+        self.proj = nn.Linear(d, n_slots * 4, bias=False)
+        self.out = nn.Linear(n_slots * 4, d, bias=False)
+        nn.init.orthogonal_(self.proj.weight)
+        nn.init.orthogonal_(self.out.weight)
+        self.addr = AlephAddress(K, 4)
+        self.gate = nn.Parameter(torch.tensor(-3.0))
+
+    def forward(self, x):
+        B, n, _ = x.shape
+        slots = self.proj(x).view(B, n, self.n_slots, 4)
+        m = self.addr.m_hat(slots).reshape(B, n, -1)
+        return x + self.gate.sigmoid() * self.out(m)
+
+
+class Block(nn.Module):
+    def __init__(self, d: int, attn: nn.Module):
+        super().__init__()
+        self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
+        self.attn = attn
+        self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
+
+    def forward(self, x):
+        x = x + self.attn(self.n1(x))
+        return x + self.mlp(self.n2(x))
+
+
+class ByteLM(nn.Module):
+    def __init__(self, arm: str, d: int = 192, layers: int = 4, block: int = 256,
+                 K: int = 32, D: int = 4):
+        super().__init__()
+        # "<arm>_tri" suffix = trigram byte embedding (AlephLM byte_emb x3 lineage):
+        # token embedding is the sum of embeddings of bytes t, t-1, t-2.
+        self.trigram = arm.endswith("_tri")
+        if self.trigram:
+            arm = arm[:-4]
+        # "_fib" = super-Fibonacci S^3 codebook init (basin test: starts INSIDE
+        # the RP^3 attractor; primary observable is init->final geodesic drift).
+        self.fib = arm.endswith("_fib")
+        if self.fib:
+            arm = arm[:-4]
+        # "relay*" = stacked addresses in depth: MslRelay after every block.
+        # relay -> sdpa trunk + standard head; relay_msl64 -> + addressed head.
+        self.use_relay = arm.startswith("relay")
+        if arm == "relay":
+            arm = "sdpa"
+        elif arm == "relay_msl64":
+            arm = "addr_msl64"
+        self.arm, self.block = arm, block
+        self.emb = nn.Embedding(VOCAB, d)
+        if self.trigram:
+            self.emb1 = nn.Embedding(VOCAB, d)
+            self.emb2 = nn.Embedding(VOCAB, d)
+        self.pos = nn.Parameter(torch.zeros(1, block, d) + 0.01 * torch.randn(1, block, d))
+        mk_attn = (lambda: CausalHUB(d, K, D)) if arm == "hub" else (lambda: CausalSDPA(d))
+        self.blocks = nn.ModuleList([Block(d, mk_attn()) for _ in range(layers)])
+        if self.use_relay:
+            self.relays = nn.ModuleList([MslRelay(d) for _ in range(layers)])
+        self.nf = nn.LayerNorm(d)
+        if arm == "addr_head":
+            self.head_addr = AlephAddress(K, d)  # v1: codebook in model dim — COLLAPSED
+            self.head = nn.Linear(K, VOCAB, bias=True)
+        elif arm in ("addr_d4", "addr_3tau", "addr_mhat"):
+            # v2 refinements: LOW-D HOME — learned projection to the native D=4 home
+            # before addressing (mirrors the healthy HUB arms), K=64.
+            self.head_proj = nn.Linear(d, 4, bias=False)
+            nn.init.orthogonal_(self.head_proj.weight)
+            self.head_addr = AlephAddress(64, 4)
+            if arm == "addr_d4":
+                self.head = nn.Linear(64, VOCAB, bias=True)      # w alone, D=4 home
+            elif arm == "addr_3tau":
+                self.taus = (0.05, 0.1, 0.3)                     # rule-of-3 strobe
+                self.head = nn.Linear(64 * 3, VOCAB, bias=True)
+            else:                                                 # addr_mhat
+                self.head = nn.Linear(4, VOCAB, bias=True)       # tightest: M_hat
+        elif arm.startswith("addr_msl"):
+            # v3: MULTI-SLOT heads — the 16s funnel widening: P parallel D=4 slots
+            # over a SHARED codebook. addr_msl consumes the reconstructive M_hat per
+            # slot (Px4 dims); addr_msl_w consumes signed w per slot (Px64) — tests
+            # whether slot-parallel consumption alone rescues the coefficient path.
+            # addr_msl<P> = slot-count dose-response. addr_mslh<P> = HARD sign-code
+            # consumption (straight-through M_hard per slot).
+            self.hard = arm.startswith("addr_mslh")
+            if arm in ("addr_msl", "addr_msl_w"):
+                self.n_slots = 16
+            else:
+                self.n_slots = int(arm[len("addr_mslh" if self.hard else "addr_msl"):])
+            self.head_proj = nn.Linear(d, self.n_slots * 4, bias=False)
+            nn.init.orthogonal_(self.head_proj.weight)
+            self.head_addr = AlephAddress(
+                64, 4, init="fibonacci" if self.fib else "random")
+            width = self.n_slots * (64 if arm == "addr_msl_w" else 4)
+            self.head = nn.Linear(width, VOCAB, bias=True)
+        elif arm == "addr_3tau_mhat":
+            # v3: combine the two v2 winners — 3-tau stroboscope + reconstructive read.
+            self.head_proj = nn.Linear(d, 4, bias=False)
+            nn.init.orthogonal_(self.head_proj.weight)
+            self.head_addr = AlephAddress(64, 4)
+            self.taus = (0.05, 0.1, 0.3)
+            self.head = nn.Linear(64 * 3 + 4, VOCAB, bias=True)
+        else:
+            self.head = nn.Linear(d, VOCAB, bias=True)
+        self._last_h = None
+
+    def forward(self, idx):
+        x = self.emb(idx)
+        if self.trigram:  # past-only shifts — causality preserved
+            x = x + self.emb1(F.pad(idx, (1, 0), value=0)[:, :-1]) \
+                  + self.emb2(F.pad(idx, (2, 0), value=0)[:, :-2])
+        x = x + self.pos[:, : idx.shape[1]]
+        if self.use_relay:
+            for b, r in zip(self.blocks, self.relays):
+                x = r(b(x))
+        else:
+            for b in self.blocks:
+                x = b(x)
+        h = self.nf(x)
+        self._last_h = h.detach()
+        if self.arm == "addr_head":
+            return self.head(self.head_addr.signed(h))
+        if self.arm == "addr_d4":
+            return self.head(self.head_addr.signed(self.head_proj(h)))
+        if self.arm == "addr_3tau":
+            return self.head(self.head_addr.signed_at(self.head_proj(h), self.taus))
+        if self.arm == "addr_mhat":
+            return self.head(self.head_addr.m_hat(self.head_proj(h)))
+        if self.arm.startswith("addr_msl"):
+            B, n, _ = h.shape
+            slots = self.head_proj(h).view(B, n, self.n_slots, 4)
+            if self.arm == "addr_msl_w":
+                feats = self.head_addr.signed(slots).reshape(B, n, -1)
+            elif getattr(self, "hard", False):
+                feats = self.head_addr.m_hard_ste(slots).reshape(B, n, -1)
+            else:
+                feats = self.head_addr.m_hat(slots).reshape(B, n, -1)
+            return self.head(feats)
+        if self.arm == "addr_3tau_mhat":
+            p = self.head_proj(h)
+            feats = torch.cat([self.head_addr.signed_at(p, self.taus),
+                               self.head_addr.m_hat(p)], dim=-1)
+            return self.head(feats)
+        return self.head(h)
+
+    @torch.no_grad()
+    def vitals(self) -> dict:
+        out = {}
+        if self.arm == "hub":
+            for i, b in enumerate(self.blocks):
+                if self._last_h is not None:
+                    out[f"L{i}"] = b.attn.addr.vitals(b.attn.q(self._last_h[:2]))
+        elif self.arm == "addr_head" and self._last_h is not None:
+            out["head"] = self.head_addr.vitals(self._last_h[:2])
+        elif self.arm in ("addr_d4", "addr_3tau", "addr_mhat",
+                          "addr_3tau_mhat") and self._last_h is not None:
+            out["head"] = self.head_addr.vitals(self.head_proj(self._last_h[:2]))
+        elif self.arm.startswith("addr_msl") and self._last_h is not None:
+            slots = self.head_proj(self._last_h[:2])
+            out["head"] = self.head_addr.vitals(
+                slots.reshape(*slots.shape[:-1], self.n_slots, 4))
+        if self.use_relay and self._last_h is not None:
+            for i, r in enumerate(self.relays):
+                s = r.proj(self._last_h[:2])
+                v = r.addr.vitals(s.reshape(*s.shape[:-1], r.n_slots, 4))
+                out[f"relay{i}"] = {"gate": round(r.gate.sigmoid().item(), 4),
+                                    "drift": v["drift"],
+                                    "binding_frac": v["binding_frac"],
+                                    "ppl": round(v["aliveness"]["usage_ppl"], 1)}
+        return out
+
+
+# --------------------------------------------------------------------------- data
+def _wikitext_bytes(data_root: str):
+    """wikitext-2-raw as flat uint8 tensors via the HF parquet CDN."""
+    from huggingface_hub import hf_hub_download
+    import pyarrow.parquet as pq
+
+    def load(split):
+        p = hf_hub_download("Salesforce/wikitext",
+                            f"wikitext-2-raw-v1/{split}-00000-of-00001.parquet",
+                            repo_type="dataset", local_dir=data_root)
+        text = "".join(pq.read_table(p).column("text").to_pylist())
+        return torch.frombuffer(bytearray(text.encode("utf-8")), dtype=torch.uint8).clone()
+
+    return load("train"), load("validation")
+
+
+def _batch(data: torch.Tensor, batch: int, block: int, device, g: torch.Generator):
+    ix = torch.randint(0, data.numel() - block - 1, (batch,), generator=g)
+    x = torch.stack([data[i:i + block] for i in ix]).long().to(device)
+    y = torch.stack([data[i + 1:i + block + 1] for i in ix]).long().to(device)
+    return x, y
+
+
+# -------------------------------------------------------------------- train/smoke
+def train(arms=("sdpa", "hub", "addr_head"), steps: int = 2000, batch: int = 32,
+          block: int = 256, device: str = "cuda", data_root: str = "./data",
+          seed: int = 0, eval_every: int = 500, save: bool = True):
+    """Verdict run — GPU only. Pure Adam wd=0. Reports val bits-per-byte + vitals.
+    save=True writes {data_root}/ar_ckpts/{arm}_s{seed}_t{steps}.pt per arm —
+    the cultivated codebooks are SPECIMENS for the projective reading instruments."""
+    import os
+    if device == "cuda" and not torch.cuda.is_available():
+        raise RuntimeError("Verdict runs are GPU-only (never CPU-train for accuracy).")
+    ckpt_dir = os.path.join(data_root, "ar_ckpts")
+    os.makedirs(ckpt_dir, exist_ok=True)
+    tr, va = _wikitext_bytes(data_root)
+    print(f"data ready: train {tr.numel():,} bytes, val {va.numel():,} bytes", flush=True)
+    results = {}
+    for arm in arms:
+        torch.manual_seed(seed)
+        g = torch.Generator().manual_seed(seed)
+        model = ByteLM(arm, block=block).to(device)
+        n_params = sum(p.numel() for p in model.parameters())
+        opt = torch.optim.Adam(model.parameters(), lr=3e-4, weight_decay=0.0)
+        for step in range(1, steps + 1):
+            x, y = _batch(tr, batch, block, device, g)
+            logits = model(x)
+            loss = F.cross_entropy(logits.reshape(-1, VOCAB), y.reshape(-1))
+            opt.zero_grad(set_to_none=True)
+            loss.backward()
+            opt.step()
+            if step % eval_every == 0 or step == steps:
+                model.eval()
+                with torch.no_grad():
+                    losses = []
+                    for _ in range(20):
+                        xv, yv = _batch(va, batch, block, device, g)
+                        lv = F.cross_entropy(model(xv).reshape(-1, VOCAB),
+                                             yv.reshape(-1))
+                        losses.append(lv.item())
+                bpb = sum(losses) / len(losses) / math.log(2)
+                print(f"[{arm}] step {step} val_bpb={bpb:.4f} vitals={model.vitals()}",
+                      flush=True)
+                model.train()
+        results[arm] = {"val_bpb": bpb, "params": n_params, "vitals": model.vitals()}
+        if save:
+            path = os.path.join(ckpt_dir, f"{arm}_s{seed}_t{steps}.pt")
+            torch.save({"arm": arm, "seed": seed, "steps": steps, "val_bpb": bpb,
+                        "state_dict": {k: v.cpu() for k, v in
+                                       model.state_dict().items()}}, path)
+            print(f"saved specimen: {path}", flush=True)
+    print(results, flush=True)
+    return results
+
+
+def smoke():
+    """Shapes/parse only — no accuracy claims."""
+    x = torch.randint(0, VOCAB, (2, 64))
+    for arm in ("sdpa", "hub", "addr_head"):
+        m = ByteLM(arm, d=96, layers=2, block=64, K=16)
+        logits = m(x)
+        assert logits.shape == (2, 64, VOCAB)
+        logits.sum().backward()
+        # causality check: future byte must not affect past logits
+        with torch.no_grad():
+            a = m(x)[0, 10]
+            x2 = x.clone(); x2[0, 40] = (x2[0, 40] + 7) % 256
+            b = m(x2)[0, 10]
+        assert torch.allclose(a, b, atol=1e-4), f"{arm} leaks future context"
+        print(f"{arm}: OK params={sum(p.numel() for p in m.parameters()):,} "
+              f"vitals={m.vitals()}", flush=True)
+    print("OK — AR bed smoke passed (verdict run: train() on GPU)", flush=True)
+
+
+def _in_notebook() -> bool:
+    try:
+        get_ipython()  # type: ignore[name-defined]  # noqa: F821
+        return True
+    except NameError:
+        return False
+
+
+if __name__ == "__main__":
+    if _in_notebook():
+        smoke()
+        print("Notebook mode: call  train(steps=2000)  in the next cell (GPU).")
+    else:
+        import argparse
+        ap = argparse.ArgumentParser()
+        ap.add_argument("--train", action="store_true")
+        ap.add_argument("--steps", type=int, default=2000)
+        a, _ = ap.parse_known_args()
+        train(steps=a.steps) if a.train else smoke()

exp007_math/build_results.py

@@ -0,0 +1,88 @@
+"""build_results.py — exp007_math: read results/ledger.jsonl and RE-ASSERT
+every claim in the README. The ledger carries two rows: the FROZEN trunk
+baseline (separates "the trunk can already add" from what the anchor adds)
+and the trained relay_pw_math arm (one relay_pw stack, all four ask-method
+registers interleaved). Run from inside this folder: python build_results.py
+"""
+import json
+import os
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+rows = [json.loads(l) for l in
+        open(os.path.join(HERE, "results", "ledger.jsonl"), encoding="utf-8")]
+assert all(r["exp"] == "q7" for r in rows) and len(rows) == 2
+
+frozen = next(r for r in rows if r["arm"] == "frozen")
+trained = next(r for r in rows if r["arm"] == "relay_pw_math")
+assert frozen["seed"] == 0 and frozen["steps"] == 0
+assert trained["seed"] == 0 and trained["steps"] == 3000
+REGS = ("nl", "json", "code", "word")
+
+# same synthetic corpus under both arms (train-side cache stats identical)
+assert frozen["cache_stats"] == trained["cache_stats"] == \
+    {"nl": 696, "json": 702, "code": 1496, "word": 718}
+
+fv, tv = frozen["validity"], trained["validity"]
+
+# claim 1a: format lock-in — the frozen trunk fails FORMAT only on nl
+# (0.312: it answers correctly but not in the asked shape; json/code/word
+# already 1.0); after training all four registers reach format 1.0.
+assert fv["nl"]["format"] == 0.312
+assert fv["json"]["format"] == 1.0
+assert fv["code"]["format"] == 1.0
+assert fv["word"]["format"] == 1.0
+assert all(tv[r]["format"] == 1.0 for r in REGS)
+
+# claim 1b: no correctness tax — correctness never degrades on any register:
+# nl and json hold 1.0 -> 1.0, word 0.938 -> 1.0, code improves (claim 2).
+assert fv["nl"]["correct"] == 1.0 and tv["nl"]["correct"] == 1.0
+assert fv["json"]["correct"] == 1.0 and tv["json"]["correct"] == 1.0
+assert fv["word"]["correct"] == 0.938 and tv["word"]["correct"] == 1.0
+assert all(tv[r]["correct"] >= fv[r]["correct"] for r in REGS)
+
+# claim 2: computation through the adapters — code-register held-out
+# correctness 0.688 -> 0.938 on 16 f(x)-rendered questions never seen in
+# training (audited: 5/16 are true a*x+b instances unseen anywhere in the
+# corpus; 4/16 wrapped-arithmetic items share their a-op-b fact with
+# another register's training text — see README claim 2). Not target
+# recall. Frozen baseline ledgered.
+assert fv["code"]["correct"] == 0.688
+assert tv["code"]["correct"] == 0.938
+
+# claim 3 (the predictability principle) is a CROSS-EXPERIMENT
+# interpretation — exp006's keyword compliance (0.083 -> 0.083, exp006
+# ledger) vs correctness moving here — and is NOT asserted from this
+# ledger. Its local anchor IS asserted: held-out LM loss improved on
+# every register (the loss had something to teach, and correctness moved
+# exactly where target tokens are unpredictable without the question).
+assert frozen["ppl"] == {"nl": 2.763, "json": 4.393,
+                         "code": 2.591, "word": 3.145}
+assert trained["ppl"] == {"nl": 1.342, "json": 1.061,
+                          "code": 1.168, "word": 1.525}
+assert all(trained["ppl"][r] < frozen["ppl"][r] for r in REGS)
+
+# claim 4: register separation, deepest PEAK yet in the line — sign-code
+# sep across the 4 ask-methods 0.4711 @ L16 and 0.4957 @ L23 (exp006 story
+# registers 0.3476 / 0.3451 at those taps, own peak 0.4619 @ L8; exp003
+# instruct arms <= 0.304 at L16/L23 — cited from their ledgers, not
+# asserted here). Separation grows monotonically with depth in THIS run;
+# depth profiles differ across beds. Frozen arm has no adapters, no probe.
+sep = trained["register_sep"]
+assert sep["L16"]["sep"] == 0.4711
+assert sep["L23"]["sep"] == 0.4957
+assert sep["L0"]["sep"] == 0.2145 and sep["L8"]["sep"] == 0.2808
+assert sep["L0"]["sep"] < sep["L8"]["sep"] < sep["L16"]["sep"] \
+    < sep["L23"]["sep"]
+assert frozen["register_sep"] is None
+
+out = {"frozen": {"ppl": frozen["ppl"], "validity": fv},
+       "relay_pw_math": {"steps": trained["steps"],
+                         "ppl": trained["ppl"],
+                         "validity": tv,
+                         "register_sep": sep},
+       "cache_stats": trained["cache_stats"],
+       "n_rows": len(rows)}
+json.dump(out, open(os.path.join(HERE, "results", "results.json"), "w",
+                    encoding="utf-8"), indent=1)
+print(f"{len(rows)} rows -> results/results.json")
+print("all README claims asserted OK")

exp007_math/exp013_augmentation_bed.py

@@ -0,0 +1,506 @@
+"""exp013_augmentation_bed.py — augmenting pretrained models with the aleph.
+Three tracks, one file (sequel to exp012's ar_differentiation_bed):
+
+  A. AUTOREGRESSION FROM CLIP-L: next-token prediction over CLIP-tokenized text,
+     reading FROZEN openai/clip-vit-large-patch14 text-tower hidden states.
+     FACTOR: extraction layer in {final, penultimate} (the last two layers — the
+     penultimate is what diffusion stacks consume). Heads at ~matched params:
+     linear | mlp | aleph multi-slot M_hat (P=64, D=4, shared K=64 — the exp012
+     certified construction) | sign-code (straight-through).
+  B. JOINT-FAILURE PROBES on frozen pooled embeddings (CLIP-L both layers + BERT):
+     b1 SPELLING-AR — decode a word's characters from ONLY the head's read of its
+        pooled embedding (GATE: linear/mlp must fail <50% exact first);
+     b2 ORDER — original-vs-shuffled discrimination (secondary; info may be absent).
+  C. GPT-2 (124M) AUGMENTATION: frozen trunk + trainable adapters after every block —
+     aleph MslRelay adapters vs param-matched MLP adapters vs frozen baseline;
+     gate growth by depth is a first-class readout (exp012 depth-gradient law).
+
+Riders: pure Adam wd=0; no contrastive/InfoNCE into address paths; vitals are
+readouts; GPU-only for verdict runs; caches/specimens live OUTSIDE the repo.
+Colab: paste geolip_vitals.py, then ar_differentiation_bed.py, then this file.
+"""
+from __future__ import annotations
+import json
+import math
+import os
+import re
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+# ---- paste-ahead imports (notebook-safe) -----------------------------------------
+if "anchor_drift" not in globals():
+    try:
+        from geolip_vitals import anchor_drift, axis_aliveness, path_diversity
+    except ImportError:
+        _here = globals().get("__file__")
+        if _here is None:
+            raise ImportError("paste/run geolip_vitals.py first")
+        import sys, pathlib
+        sys.path.insert(0, str(pathlib.Path(_here).parent))
+        from geolip_vitals import anchor_drift, axis_aliveness, path_diversity
+if "AlephAddress" not in globals():
+    try:
+        from ar_differentiation_bed import AlephAddress, MslRelay, _wikitext_bytes
+    except ImportError:
+        _here = globals().get("__file__")
+        if _here is None:
+            raise ImportError("paste/run ar_differentiation_bed.py first")
+        from ar_differentiation_bed import AlephAddress, MslRelay, _wikitext_bytes
+
+DATA_ROOT = os.environ.get("GEOLIP_DATA", "./data")
+
+
+class SquaredReLU(nn.Module):
+    def forward(self, x):
+        return F.relu(x) ** 2
+
+
+# =============================================================== head arm family ==
+class HeadArm(nn.Module):
+    """Conditioning transform d_in -> F_OUT (256), then a shared-shape task head.
+    Arms: linear | mlp | aleph (multi-slot M_hat) | sign (straight-through M_hard).
+    The aleph arms are the exp012-certified construction: P=64 slots x D=4 over one
+    shared K=64 codebook."""
+    F_OUT = 256
+
+    def __init__(self, arm: str, d_in: int, mlp_hidden: int = 192):
+        # mlp_hidden=192 param-matches the aleph arm at d_in=768:
+        # mlp ~ 192*(768+256)+LN ~ 197K vs aleph proj 768*256 + codebook = 196.9K
+        super().__init__()
+        self.arm = arm
+        if arm == "linear":
+            self.net = nn.Linear(d_in, self.F_OUT, bias=False)
+        elif arm == "mlp":
+            self.net = nn.Sequential(nn.Linear(d_in, mlp_hidden), SquaredReLU(),
+                                     nn.LayerNorm(mlp_hidden),
+                                     nn.Linear(mlp_hidden, self.F_OUT))
+        elif arm in ("aleph", "sign"):
+            self.proj = nn.Linear(d_in, 64 * 4, bias=False)
+            nn.init.orthogonal_(self.proj.weight)
+            self.addr = AlephAddress(64, 4)
+        else:
+            raise ValueError(arm)
+
+    def forward(self, x):
+        if self.arm in ("linear", "mlp"):
+            return self.net(x)
+        slots = self.proj(x).reshape(*x.shape[:-1], 64, 4)
+        read = self.addr.m_hard_ste(slots) if self.arm == "sign" \
+            else self.addr.m_hat(slots)
+        return read.reshape(*x.shape[:-1], 256)
+
+    @torch.no_grad()
+    def vitals(self, x_sample) -> dict:
+        if self.arm in ("linear", "mlp"):
+            return {}
+        slots = self.proj(x_sample).reshape(*x_sample.shape[:-1], 64, 4)
+        p, n = self.addr.oriented(slots)
+        two_k = torch.cat([p, n], -1).reshape(-1, 128)
+        d = anchor_drift(self.addr.codebook, self.addr.home)
+        return {"drift": round(d["mean"], 4),
+                "binding_frac": round(d["binding_fraction"], 4),
+                "usage_ppl": round(axis_aliveness(two_k)["usage_ppl"], 1),
+                "paths": path_diversity(two_k.argmax(-1))["unique_hashed"]}
+
+    def param_count(self):
+        return sum(p.numel() for p in self.parameters())
+
+
+# ================================================================== caches ========
+def _wikitext_lines(data_root, min_chars=40, max_lines=None):
+    from huggingface_hub import hf_hub_download
+    import pyarrow.parquet as pq
+    out = {}
+    for split, cap in (("train", max_lines), ("validation", None)):
+        p = hf_hub_download("Salesforce/wikitext",
+                            f"wikitext-2-raw-v1/{split}-00000-of-00001.parquet",
+                            repo_type="dataset", local_dir=data_root)
+        lines = [t.strip() for t in pq.read_table(p).column("text").to_pylist()
+                 if len(t.strip()) >= min_chars]
+        out[split] = lines[:cap] if cap else lines
+    return out["train"], out["validation"]
+
+
+@torch.no_grad()
+def cache_clip(data_root, n_train=12000, n_val=1500, device="cuda", batch=64):
+    """CLIP-L text tower over wikitext lines; caches BOTH of the last two layers.
+    hidden_states[-1] == the final encoder layer output (pre final-LN),
+    last_hidden_state == final-LN(final layer). We cache:
+      'final'  = last_hidden_state (what the projection head consumes),
+      'penult' = hidden_states[-2] (the layer diffusion stacks consume).
+    Also caches pooled (EOS-position) vectors for both layers, and token ids."""
+    from transformers import CLIPTextModel, CLIPTokenizerFast
+    path = os.path.join(data_root, "exp013", "clip_cache.pt")
+    if os.path.exists(path):
+        return torch.load(path, map_location="cpu", weights_only=True)
+    os.makedirs(os.path.dirname(path), exist_ok=True)
+    tok = CLIPTokenizerFast.from_pretrained("openai/clip-vit-large-patch14")
+    model = CLIPTextModel.from_pretrained("openai/clip-vit-large-patch14").to(device).eval()
+    tr_lines, va_lines = _wikitext_lines(data_root, max_lines=n_train)
+    va_lines = va_lines[:n_val]
+    def encode(lines):
+        H_f, H_p, IDS, EOS = [], [], [], []
+        for i in range(0, len(lines), batch):
+            enc = tok(lines[i:i + batch], padding="max_length", truncation=True,
+                      max_length=77, return_tensors="pt").to(device)
+            out = model(**enc, output_hidden_states=True)
+            H_f.append(out.last_hidden_state.half().cpu())
+            H_p.append(out.hidden_states[-2].half().cpu())
+            IDS.append(enc.input_ids.cpu())
+            EOS.append(enc.input_ids.argmax(-1).cpu())  # EOT id is the max token id
+        return (torch.cat(H_f), torch.cat(H_p), torch.cat(IDS), torch.cat(EOS))
+    tr = encode(tr_lines)
+    va = encode(va_lines)
+    blob = {"train": {"final": tr[0], "penult": tr[1], "ids": tr[2], "eos": tr[3]},
+            "val":   {"final": va[0], "penult": va[1], "ids": va[2], "eos": va[3]},
+            "vocab": tok.vocab_size}
+    torch.save(blob, path)
+    print(f"clip cache: train {tr[0].shape}, val {va[0].shape} -> {path}", flush=True)
+    return blob
+
+
+@torch.no_grad()
+def cache_word_embeddings(data_root, n_words=10000, device="cuda", batch=256):
+    """Pooled embeddings of frequent wikitext words for the spelling probe:
+    CLIP-L final + penultimate (EOS-pooled) and BERT (CLS + mean of last layer)."""
+    from transformers import (CLIPTextModel, CLIPTokenizerFast,
+                              BertModel, BertTokenizerFast)
+    path = os.path.join(data_root, "exp013", "word_cache.pt")
+    if os.path.exists(path):
+        return torch.load(path, map_location="cpu", weights_only=True)
+    os.makedirs(os.path.dirname(path), exist_ok=True)
+    tr_lines, _ = _wikitext_lines(data_root)
+    from collections import Counter
+    cnt = Counter(w for l in tr_lines for w in re.findall(r"[a-z]{3,12}", l.lower()))
+    words = [w for w, _ in cnt.most_common(n_words)]
+    blob = {"words": words}
+    ct = CLIPTokenizerFast.from_pretrained("openai/clip-vit-large-patch14")
+    cm = CLIPTextModel.from_pretrained("openai/clip-vit-large-patch14").to(device).eval()
+    fin, pen = [], []
+    for i in range(0, len(words), batch):
+        enc = ct(words[i:i + batch], padding="max_length", truncation=True,
+                 max_length=77, return_tensors="pt").to(device)
+        out = cm(**enc, output_hidden_states=True)
+        eos = enc.input_ids.argmax(-1)
+        idx = torch.arange(eos.numel(), device=device)
+        fin.append(out.last_hidden_state[idx, eos].half().cpu())
+        pen.append(out.hidden_states[-2][idx, eos].half().cpu())
+    blob["clip_final"], blob["clip_penult"] = torch.cat(fin), torch.cat(pen)
+    del cm
+    bt = BertTokenizerFast.from_pretrained("bert-base-uncased")
+    bm = BertModel.from_pretrained("bert-base-uncased").to(device).eval()
+    cls, mean = [], []
+    for i in range(0, len(words), batch):
+        enc = bt(words[i:i + batch], padding=True, truncation=True,
+                 max_length=16, return_tensors="pt").to(device)
+        out = bm(**enc).last_hidden_state
+        m = enc.attention_mask.unsqueeze(-1)
+        cls.append(out[:, 0].half().cpu())
+        mean.append(((out * m).sum(1) / m.sum(1)).half().cpu())
+    blob["bert_cls"], blob["bert_mean"] = torch.cat(cls), torch.cat(mean)
+    torch.save(blob, path)
+    print(f"word cache: {len(words)} words -> {path}", flush=True)
+    return blob
+
+
+# ============================================================ track A =============
+def track_a(arms=("linear", "mlp", "aleph", "sign"), layers=("final", "penult"),
+            steps=1500, batch=64, seed=0, device="cuda",
+            data_root=DATA_ROOT, eval_every=500, save=True):
+    """Next-CLIP-token prediction from frozen CLIP-L hidden states."""
+    if not torch.cuda.is_available():
+        raise RuntimeError("verdict runs are GPU-only")
+    blob = cache_clip(data_root, device=device)
+    V = blob["vocab"]
+    results = {}
+    ck_dir = os.path.join(data_root, "exp013", "ckpts")
+    os.makedirs(ck_dir, exist_ok=True)
+    for layer in layers:
+        Htr = blob["train"][layer].float()
+        ids_tr = blob["train"]["ids"]
+        Hva = blob["val"][layer].float()
+        ids_va = blob["val"]["ids"]
+        for arm in arms:
+            torch.manual_seed(seed)
+            g = torch.Generator().manual_seed(seed)
+            head = HeadArm(arm, Htr.shape[-1]).to(device)
+            out_proj = nn.Linear(HeadArm.F_OUT, V).to(device)
+            params = list(head.parameters()) + list(out_proj.parameters())
+            opt = torch.optim.Adam(params, lr=3e-4, weight_decay=0.0)
+            n_par = sum(p.numel() for p in params)
+            for step in range(1, steps + 1):
+                ix = torch.randint(0, Htr.shape[0], (batch,), generator=g)
+                h = Htr[ix].to(device)
+                y = ids_tr[ix].to(device)
+                logits = out_proj(head(h[:, :-1]))
+                loss = F.cross_entropy(logits.reshape(-1, V), y[:, 1:].reshape(-1))
+                opt.zero_grad(set_to_none=True); loss.backward(); opt.step()
+                if step % eval_every == 0 or step == steps:
+                    with torch.no_grad():
+                        ls = []
+                        for j in range(0, min(1024, Hva.shape[0]), batch):
+                            h = Hva[j:j + batch].to(device)
+                            y = ids_va[j:j + batch].to(device)
+                            lg = out_proj(head(h[:, :-1]))
+                            ls.append(F.cross_entropy(
+                                lg.reshape(-1, V), y[:, 1:].reshape(-1)).item())
+                        ce = sum(ls) / len(ls)
+                    vit = head.vitals(Hva[:2, :8].to(device))
+                    print(f"[A {layer} {arm} s{seed}] step {step} val_ce={ce:.4f} "
+                          f"params={n_par:,} vitals={vit}", flush=True)
+            results[f"{layer}/{arm}/s{seed}"] = {"val_ce": ce, "params": n_par,
+                                                 "vitals": vit}
+            if save and arm in ("aleph", "sign"):
+                torch.save({"track": "A", "layer": layer, "arm": arm, "seed": seed,
+                            "val_ce": ce, "state_dict": {k: v.cpu() for k, v in
+                                                          head.state_dict().items()}},
+                           os.path.join(ck_dir, f"A_{layer}_{arm}_s{seed}.pt"))
+    print(results, flush=True)
+    return results
+
+
+# ============================================================ track B =============
+class CharDecoder(nn.Module):
+    """Tiny GRU char decoder conditioned ONLY on the arm's 256-d read."""
+    CHARS = "abcdefghijklmnopqrstuvwxyz"
+    def __init__(self, cond_dim=256, hidden=256):
+        super().__init__()
+        self.V = len(self.CHARS) + 2  # +BOS +EOS
+        self.emb = nn.Embedding(self.V, 64)
+        self.init = nn.Linear(cond_dim, hidden)
+        self.gru = nn.GRU(64, hidden, batch_first=True)
+        self.out = nn.Linear(hidden, self.V)
+
+    def encode_word(self, w):
+        return [1] + [2 + self.CHARS.index(c) for c in w] + [0]  # BOS..EOS(0)
+
+    def forward(self, cond, tgt):          # tgt: (B, L) int, teacher-forced
+        h0 = torch.tanh(self.init(cond)).unsqueeze(0)
+        x = self.emb(tgt[:, :-1])
+        y, _ = self.gru(x, h0)
+        return self.out(y)                  # predict tgt[:,1:]
+
+    @torch.no_grad()
+    def greedy(self, cond, max_len=14):
+        B = cond.shape[0]
+        h = torch.tanh(self.init(cond)).unsqueeze(0)
+        t = torch.ones(B, 1, dtype=torch.long, device=cond.device)
+        done = torch.zeros(B, dtype=torch.bool, device=cond.device)
+        outs = []
+        for _ in range(max_len):
+            y, h = self.gru(self.emb(t), h)
+            t = self.out(y).argmax(-1)
+            outs.append(t)
+            done |= (t.squeeze(1) == 0)
+            if done.all():
+                break
+        return torch.cat(outs, 1)
+
+
+def track_b1(arms=("linear", "mlp"), substrates=("clip_final", "clip_penult",
+             "bert_cls", "bert_mean"), steps=3000, batch=128, seed=0,
+             device="cuda", data_root=DATA_ROOT, save=True):
+    """Spelling-AR from pooled embeddings. Run baselines first (the GATE:
+    qualify the task only if linear/mlp exact-match < 0.50), then aleph/sign."""
+    if not torch.cuda.is_available():
+        raise RuntimeError("verdict runs are GPU-only")
+    blob = cache_word_embeddings(data_root, device=device)
+    words = blob["words"]
+    dec_tpl = CharDecoder()
+    enc = [dec_tpl.encode_word(w) for w in words]
+    L = max(len(e) for e in enc)
+    tgt = torch.zeros(len(enc), L, dtype=torch.long)
+    for i, e in enumerate(enc):
+        tgt[i, :len(e)] = torch.tensor(e)
+    g0 = torch.Generator().manual_seed(1234)          # fixed split across arms
+    perm = torch.randperm(len(words), generator=g0)
+    tr_ix, va_ix = perm[:9000], perm[9000:]
+    ck_dir = os.path.join(data_root, "exp013", "ckpts")
+    os.makedirs(ck_dir, exist_ok=True)
+    results = {}
+    for sub in substrates:
+        E = blob[sub].float()
+        for arm in arms:
+            torch.manual_seed(seed)
+            g = torch.Generator().manual_seed(seed)
+            head = HeadArm(arm, E.shape[-1]).to(device)
+            dec = CharDecoder().to(device)
+            params = list(head.parameters()) + list(dec.parameters())
+            opt = torch.optim.Adam(params, lr=1e-3, weight_decay=0.0)
+            for step in range(1, steps + 1):
+                ix = tr_ix[torch.randint(0, tr_ix.numel(), (batch,), generator=g)]
+                cond = head(E[ix].to(device))
+                t = tgt[ix].to(device)
+                lg = dec(cond, t)
+                mask = (t[:, 1:] != 0) | (torch.cumsum(t[:, 1:] == 0, 1) == 1)
+                loss = F.cross_entropy(lg[mask], t[:, 1:][mask])
+                opt.zero_grad(set_to_none=True); loss.backward(); opt.step()
+            with torch.no_grad():
+                cond = head(E[va_ix].to(device))
+                pred = dec.greedy(cond)
+                t = tgt[va_ix, 1:].to(device)
+                n = min(pred.shape[1], t.shape[1])
+                pad_ok = torch.ones_like(t[:, :n], dtype=torch.bool)
+                seen_eos = torch.cumsum(t[:, :n] == 0, 1) > 0
+                match = ((pred[:, :n] == t[:, :n]) | seen_eos).all(-1)
+                exact = match.float().mean().item()
+            vit = head.vitals(E[va_ix[:16]].to(device))
+            print(f"[B1 {sub} {arm} s{seed}] exact={exact:.4f} vitals={vit}", flush=True)
+            results[f"{sub}/{arm}/s{seed}"] = {"exact": exact, "vitals": vit}
+            if save and arm in ("aleph", "sign"):
+                torch.save({"track": "B1", "sub": sub, "arm": arm, "seed": seed,
+                            "exact": exact, "state_dict": {k: v.cpu() for k, v in
+                                                            head.state_dict().items()}},
+                           os.path.join(ck_dir, f"B1_{sub}_{arm}_s{seed}.pt"))
+    print(results, flush=True)
+    return results
+
+
+# ============================================================ track C =============
+class MLPAdapter(nn.Module):
+    """Param-matched plain adapter (the ablation twin of MslRelay).
+    hidden=64 matches MslRelay at d=768 (2*768*64=98.3K vs 98.6K incl codebook).
+    Output layer ZERO-INIT (standard adapter stabilization — the first version
+    diverged at lr 1e-3 with random init; the aleph relay needed no such aid,
+    which is itself a datapoint, but the control gets its best shot)."""
+    def __init__(self, d, hidden=64):
+        super().__init__()
+        out = nn.Linear(hidden, d)
+        nn.init.zeros_(out.weight)
+        nn.init.zeros_(out.bias)
+        self.net = nn.Sequential(nn.Linear(d, hidden), SquaredReLU(), out)
+        self.gate = nn.Parameter(torch.tensor(-3.0))
+
+    def forward(self, x):
+        return x + self.gate.sigmoid() * self.net(x)
+
+
+class _BlockWithAdapter(nn.Module):
+    def __init__(self, block, adapter):
+        super().__init__()
+        self.block, self.adapter = block, adapter
+
+    def forward(self, *a, **k):
+        out = self.block(*a, **k)
+        if isinstance(out, tuple):
+            return (self.adapter(out[0]),) + out[1:]
+        return self.adapter(out)
+
+
+def track_c(arms=("frozen", "aleph", "mlp"), steps=1500, batch=8, block=256,
+            seed=0, device="cuda", data_root=DATA_ROOT,
+            eval_every=500, save=True):
+    """GPT-2 124M frozen; adapters after every block; train adapters only."""
+    from transformers import GPT2LMHeadModel, GPT2TokenizerFast
+    if not torch.cuda.is_available():
+        raise RuntimeError("verdict runs are GPU-only")
+    tok = GPT2TokenizerFast.from_pretrained("gpt2")
+    tr_lines, va_lines = _wikitext_lines(data_root)
+    def to_stream(lines):
+        ids = tok("\n\n".join(lines), return_tensors="pt").input_ids[0]
+        return ids
+    stream_tr = to_stream(tr_lines[:8000])
+    stream_va = to_stream(va_lines[:1000])
+    ck_dir = os.path.join(data_root, "exp013", "ckpts")
+    os.makedirs(ck_dir, exist_ok=True)
+    results = {}
+    for arm in arms:
+        torch.manual_seed(seed)
+        g = torch.Generator().manual_seed(seed)
+        model = GPT2LMHeadModel.from_pretrained("gpt2").to(device)
+        for p in model.parameters():
+            p.requires_grad_(False)
+        adapters = []
+        if arm != "frozen":
+            d = model.config.n_embd
+            for i, blk in enumerate(model.transformer.h):
+                ad = (MslRelay(d) if arm == "aleph" else MLPAdapter(d)).to(device)
+                model.transformer.h[i] = _BlockWithAdapter(blk, ad)
+                adapters.append(ad)
+            params = [p for ad in adapters for p in ad.parameters()]
+            n_par = sum(p.numel() for p in params)
+            opt = torch.optim.Adam(params, lr=1e-3, weight_decay=0.0)
+        else:
+            params, n_par = [], 0
+        def eval_ppl():
+            model.eval()
+            with torch.no_grad():
+                ls = []
+                for j in range(0, stream_va.numel() - block - 1, block * 4):
+                    x = stream_va[j:j + block].unsqueeze(0).to(device)
+                    out = model(x, labels=x)
+                    ls.append(out.loss.item())
+            model.train()
+            return math.exp(sum(ls) / len(ls))
+        if arm == "frozen":
+            ppl = eval_ppl()
+            print(f"[C frozen] ppl={ppl:.3f}", flush=True)
+            results["frozen"] = {"ppl": ppl}
+            continue
+        for step in range(1, steps + 1):
+            ix = torch.randint(0, stream_tr.numel() - block - 1, (batch,), generator=g)
+            x = torch.stack([stream_tr[i:i + block] for i in ix]).to(device)
+            loss = model(x, labels=x).loss
+            opt.zero_grad(set_to_none=True); loss.backward(); opt.step()
+            if step % eval_every == 0 or step == steps:
+                ppl = eval_ppl()
+                gates = [round(ad.gate.sigmoid().item(), 4) for ad in adapters]
+                vit = {}
+                if arm == "aleph":
+                    drifts = [round(anchor_drift(ad.addr.codebook, ad.addr.home)
+                                    ["mean"], 3) for ad in adapters]
+                    vit = {"drift_by_depth": drifts}
+                print(f"[C {arm} s{seed}] step {step} ppl={ppl:.3f} "
+                      f"params={n_par:,} gates={gates} {vit}", flush=True)
+        results[f"{arm}/s{seed}"] = {"ppl": ppl, "params": n_par, "gates": gates,
+                                     **vit}
+        if save and arm == "aleph":
+            torch.save({"track": "C", "arm": arm, "seed": seed, "ppl": ppl,
+                        "state_dict": {f"relay{i}.{k}": v.cpu()
+                                       for i, ad in enumerate(adapters)
+                                       for k, v in ad.state_dict().items()}},
+                       os.path.join(ck_dir, f"C_{arm}_s{seed}.pt"))
+    print(results, flush=True)
+    return results
+
+
+# ================================================================ smoke ===========
+def smoke():
+    """Shapes/parse only — no substrates, no training."""
+    for arm in ("linear", "mlp", "aleph", "sign"):
+        h = HeadArm(arm, 768)
+        y = h(torch.randn(2, 10, 768))
+        assert y.shape == (2, 10, 256)
+        y.sum().backward()
+        print(arm, "OK", f"{h.param_count():,}", h.vitals(torch.randn(2, 4, 768)))
+    dec = CharDecoder()
+    t = torch.tensor([dec.encode_word("hello") + [0] * 3,
+                      dec.encode_word("worlds") + [0] * 2])
+    lg = dec(torch.randn(2, 256), t)
+    assert lg.shape[:2] == (2, t.shape[1] - 1)
+    print("decoder OK; greedy:", dec.greedy(torch.randn(2, 256)).shape)
+    ad = MLPAdapter(768)
+    assert ad(torch.randn(2, 4, 768)).shape == (2, 4, 768)
+    print("adapter OK — exp013 smoke passed (caches+tracks need GPU+transformers)")
+
+
+def _in_notebook():
+    try:
+        get_ipython()  # type: ignore[name-defined]  # noqa: F821
+        return True
+    except NameError:
+        return False
+
+
+if __name__ == "__main__":
+    if _in_notebook():
+        smoke()
+        print("Notebook: cache_clip()/cache_word_embeddings() then "
+              "track_b1() gate -> track_a() -> track_c().")
+    else:
+        import argparse
+        ap = argparse.ArgumentParser()
+        ap.add_argument("--track", default="smoke")
+        a, _ = ap.parse_known_args()
+        {"smoke": smoke, "a": track_a, "b1": track_b1, "c": track_c}[a.track]()

exp007_math/geolip_vitals.py

@@ -0,0 +1,219 @@
+"""geolip_vitals.py — shared diagnostic harness for the GeoLIP aleph experiments.
+ALL functions are READOUTS: no gradients, no losses. CV is a readout, never a
+force. Addressing is judged by drift->0.29154 and CV->0.20, never by recon cosine
+(judgment criteria per the aleph-void article: https://huggingface.co/blog/AbstractPhil/geometric-vocabulary-patchwork-aleph-void).
+
+Vitals provided:
+  anchor_drift        — geodesic drift of anchors from init; binding fraction @0.29154
+  pentachoron_cv      — CM 4-volume CV over random 5-row subsets (geovocab2 import)
+  axis_aliveness      — oriented-address usage: axes alive, hppl, collapse flag
+  gate_stats          — gate means vs the 0.012-0.03 band
+  path_diversity      — unique-path counting, FIXED high-bits hash (low-16 bug is the
+                        retracted artifact — never use the low bits)
+  grad_norm_spread    — gradient democracy monitor (orders-of-magnitude spread)
+  CVScreen            — CV@1000-batch early band screen (<0.30 LOW / .35-.50 MID / >.80 HIGH)
+
+Smoke on a torch-capable env:  python geolip_vitals.py
+"""
+from __future__ import annotations
+import math
+import torch
+
+BINDING = 0.29154            # radians; the binding/separation constant
+CV_BAND = (0.13, 0.30)       # CM CV band (discovery_catalog #4)
+GATE_BAND = (0.012, 0.03)    # live invariant candidate (acd_campaign)
+KNUTH32 = 2654435761
+
+
+# ----------------------------------------------------------------------------- drift
+@torch.no_grad()
+def anchor_drift(current: torch.Tensor, init: torch.Tensor, tol: float = 0.05) -> dict:
+    """Geodesic drift (radians) of each row of `current` from its row in `init`,
+    both row-normalized. Returns mean/std/per-row drift and the fraction of rows
+    within +/-tol of BINDING (the GLFM '46%' readout)."""
+    a = torch.nn.functional.normalize(current.float(), dim=-1)
+    b = torch.nn.functional.normalize(init.float(), dim=-1)
+    cos = (a * b).sum(-1).clamp(-1.0, 1.0)
+    drift = torch.arccos(cos)
+    frac = ((drift - BINDING).abs() <= tol).float().mean()
+    return {"mean": drift.mean().item(), "std": drift.std().item(),
+            "per_row": drift, "binding_fraction": frac.item()}
+
+
+# -------------------------------------------------------------------------------- cv
+@torch.no_grad()
+def _pentachoron_volumes(pts: torch.Tensor) -> torch.Tensor:
+    """Batched Cayley-Menger 4-simplex volumes. pts: (B, 5, D) -> (B,) volumes.
+    One float64 det over all samples (vol^2 = -det(CM)/9216 for n=4). Built-in
+    for speed (the per-sample reference path is ~260x slower in a vitals loop);
+    geovocab2 remains the formula's reference implementation, parity-checked
+    via cv_reference_check()."""
+    B = pts.shape[0]
+    d2 = torch.cdist(pts.double(), pts.double()).pow(2)          # (B,5,5)
+    cm = torch.ones(B, 6, 6, dtype=torch.float64, device=pts.device)
+    cm[:, 0, 0] = 0.0
+    cm[:, 1:, 1:] = d2
+    det = torch.linalg.det(cm)
+    return (-det / 9216.0).clamp_min(0.0).sqrt().float()
+
+
+@torch.no_grad()
+def pentachoron_cv(rows: torch.Tensor, n_samples: int = 200,
+                   generator: torch.Generator | None = None) -> float:
+    """CV (std/mean) of Cayley-Menger 4-simplex volumes over n_samples random
+    5-row subsets. Rows are row-normalized before measurement. Uses the built-in
+    batched CM (float64 det); validate against geovocab2 with
+    cv_reference_check() after any change to the volume math."""
+    x = torch.nn.functional.normalize(rows.float(), dim=-1)
+    n = x.shape[0]
+    if n < 5:
+        raise ValueError(f"pentachoron_cv needs >=5 rows, got {n}")
+    g = generator or torch.Generator(device="cpu").manual_seed(0)
+    idx = torch.stack([torch.randperm(n, generator=g)[:5]
+                       for _ in range(n_samples)])               # (B,5)
+    v = _pentachoron_volumes(x[idx].cpu())
+    return (v.std() / v.mean().clamp_min(1e-12)).item()
+
+
+@torch.no_grad()
+def cv_reference_check(n_trials: int = 50, tol: float = 1e-5) -> float:
+    """Parity check of the built-in batched CM against geovocab2's reference
+    implementation (the formula's source of truth). Returns max |rel diff|;
+    raises if geovocab2 is absent or parity fails. Run after touching
+    _pentachoron_volumes."""
+    try:
+        from geovocab2.shapes.formula.symbolic.cayley_menger import (
+            CayleyMengerFromSimplex)
+    except Exception as e:  # pragma: no cover
+        raise ImportError(
+            "cv_reference_check requires geovocab2 (install via the geolip-svae "
+            "umbrella: pip install git+https://github.com/AbstractEyes/"
+            "geolip-svae).") from e
+    ref = CayleyMengerFromSimplex()
+    g = torch.Generator().manual_seed(0)
+    pts = torch.nn.functional.normalize(
+        torch.randn(n_trials, 5, 4, generator=g), dim=-1)
+    mine = _pentachoron_volumes(pts)
+    # compare at float64: the reference computes in the INPUT dtype, and fp32
+    # dets lose up to ~4% on near-degenerate pentachora (measured 2026-07-11)
+    theirs = torch.stack([ref.forward(p.double())["volume"].float() for p in pts])
+    rel = ((mine - theirs).abs() / theirs.abs().clamp_min(1e-12)).max().item()
+    if rel > tol:
+        raise AssertionError(f"CM parity vs geovocab2 failed: max rel {rel}")
+    return rel
+
+
+# ------------------------------------------------------------------------- aliveness
+@torch.no_grad()
+def axis_aliveness(oriented_weights: torch.Tensor, alive_thresh: float = 1e-3) -> dict:
+    """`oriented_weights`: (..., 2K) nonnegative oriented-softmax address rows
+    (sum to 1 on the last dim). Returns axes-alive count, mean-usage perplexity
+    (hppl analogue; healthy hosted reference 125-126/128), and a collapse flag.
+    Reference behavior: near-uniform aliveness at div_weight=0 (discovery #22)."""
+    w = oriented_weights.reshape(-1, oriented_weights.shape[-1]).float()
+    usage = w.mean(0)
+    usage = usage / usage.sum().clamp_min(1e-12)
+    # an axis is alive if its mean usage exceeds alive_thresh x the uniform share
+    alive = int((usage > alive_thresh * (1.0 / usage.numel())).sum())
+    ent = -(usage.clamp_min(1e-12) * usage.clamp_min(1e-12).log()).sum()
+    ppl = float(ent.exp())
+    return {"axes_total": usage.numel(), "axes_alive": alive, "usage_ppl": ppl,
+            "collapsed": ppl < 0.05 * usage.numel()}
+
+
+# ------------------------------------------------------------------------------ gates
+@torch.no_grad()
+def gate_stats(gates: torch.Tensor) -> dict:
+    """Gate values (post-sigmoid/clamp). Reports mean and whether it sits in the
+    0.012-0.03 band (read-only — the band is a candidate invariant, never a target)."""
+    g = gates.float().flatten()
+    m = g.mean().item()
+    return {"mean": m, "std": g.std().item(),
+            "in_band": GATE_BAND[0] <= m <= GATE_BAND[1]}
+
+
+# ------------------------------------------------------------------------------ paths
+@torch.no_grad()
+def path_diversity(ids: torch.Tensor) -> dict:
+    """Unique-path counting with the FIXED multiplicative hash:
+    ((ids * 2654435761) % 2^32) >> 16 — Knuth needs the HIGH bits; the low-16
+    variant produced a retracted ~1,500 path ceiling in a prior campaign.
+    `ids`: integer tensor, one composed path id per row (any shape)."""
+    x = ids.reshape(-1).to(torch.int64)
+    hashed = ((x * KNUTH32) % (1 << 32)) >> 16
+    return {"n": int(x.numel()),
+            "unique_raw": int(torch.unique(x).numel()),
+            "unique_hashed": int(torch.unique(hashed).numel())}
+
+
+@torch.no_grad()
+def compose_path_ids(stage_indices: list[torch.Tensor], radix: int) -> torch.Tensor:
+    """Compose per-stage discrete indices (each (...,) int in [0, radix)) into a
+    single path id, positional base-`radix` — construction, not hashing."""
+    out = torch.zeros_like(stage_indices[0], dtype=torch.int64)
+    for s in stage_indices:
+        out = out * radix + s.to(torch.int64)
+    return out
+
+
+# --------------------------------------------------------------------- grad democracy
+@torch.no_grad()
+def grad_norm_spread(groups: dict[str, list[torch.nn.Parameter]]) -> dict:
+    """Gradient-democracy monitor. `groups`: name -> params of one parallel member
+    (tower/expert). Reports per-group grad norms and the orders-of-magnitude spread.
+    Reference: unequalized heterogeneous towers spread ~20 orders (fibonacci dead at
+    2.25e-21 under helix); equalized ~0.0 (geofractal gradient-democracy result)."""
+    norms = {}
+    for name, params in groups.items():
+        gs = [p.grad for p in params if p.grad is not None]
+        norms[name] = float(torch.sqrt(sum((g.float() ** 2).sum() for g in gs)).item()) \
+            if gs else 0.0
+    vals = [v for v in norms.values() if v > 0]
+    spread = (math.log10(max(vals)) - math.log10(min(vals))) if len(vals) >= 2 else 0.0
+    return {"norms": norms, "spread_orders": spread, "dead": [k for k, v in norms.items() if v == 0.0]}
+
+
+# ----------------------------------------------------------------------------- screen
+class CVScreen:
+    """CV@N early band screen (tri-band ft1): record pentachoron CV at `step_mark`
+    batches; classify <0.30 LOW / 0.35-0.50 MID / >0.80 HIGH. Turns ~2h/config
+    into ~7min. Readout only."""
+    def __init__(self, step_mark: int = 1000):
+        self.step_mark = step_mark
+        self.recorded: float | None = None
+
+    def maybe_record(self, step: int, rows: torch.Tensor) -> float | None:
+        if self.recorded is None and step >= self.step_mark:
+            self.recorded = pentachoron_cv(rows)
+        return self.recorded
+
+    @property
+    def band(self) -> str | None:
+        c = self.recorded
+        if c is None:
+            return None
+        if c < 0.30:
+            return "LOW"
+        if 0.35 <= c <= 0.50:
+            return "MID"
+        if c > 0.80:
+            return "HIGH"
+        return "BETWEEN"
+
+
+# ------------------------------------------------------------------------------ smoke
+if __name__ == "__main__":  # shapes/parse smoke ONLY — no training, ever.
+    g = torch.Generator().manual_seed(0)
+    K, D = 64, 4
+    init = torch.nn.functional.normalize(torch.randn(K, D, generator=g), dim=-1)
+    cur = torch.nn.functional.normalize(init + 0.29 * torch.randn(K, D, generator=g), dim=-1)
+    print("drift:", {k: v for k, v in anchor_drift(cur, init).items() if k != "per_row"})
+    w = torch.softmax(torch.randn(32, 2 * K, generator=g), dim=-1)
+    print("aliveness:", axis_aliveness(w))
+    print("gates:", gate_stats(torch.full((8,), 0.024)))
+    ids = compose_path_ids([torch.randint(0, 16, (4096,), generator=g) for _ in range(4)], 16)
+    print("paths:", path_diversity(ids))
+    lin = torch.nn.Linear(8, 8)
+    lin(torch.randn(4, 8)).sum().backward()
+    print("democracy:", grad_norm_spread({"a": list(lin.parameters())}))
+    print("OK — vitals smoke passed (pentachoron_cv needs geovocab2; run on GPU env)")

exp007_math/qwen_exp001_relay.py

@@ -0,0 +1,271 @@
+"""qwen_exp001_relay.py — geolip-aleph-qwen EXPERIMENT 1: the relay retrofit
+at 0.5B. The certified exp013 Track-C recipe (frozen trunk + near-zero-gated
+multi-slot M_hat relays after every block, vs the param-matched zero-init MLP
+adapter ablation) at its next scale rung: Qwen2.5-0.5B (24 blocks, d=896,
+~494M frozen params; relay stack ~2.8M trainable, <0.6%). First campaign of
+the generation-stake line — judged on ppl, the GATE MECHANISM (do the gates
+grow? exp013: aleph gates grew ~3x while MLP gates shrank), relay codebook
+vitals, and GENERATED SAMPLES (the deliverable is a model that generates).
+
+Arms (x2 seeds): frozen (eval-only baseline) | relay (MslRelay(896) after
+every block) | mlp (param-matched MLPAdapter(896, hidden=64), zero-init out —
+the exp013 ablation, 114,688 vs 114,944 params per adapter, 0.2%).
+Corpus: wikitext-103-raw-v1 (HF parquet), ~12M-token cache; block 512,
+batch 4, 3000 steps adapters-only, pure Adam lr 1e-3 wd 0 (the exp013
+relay-training regime).
+Riders: trunk FROZEN throughout; pure Adam wd=0; GPU-only verdicts; >=2
+seeds; drift-check before any freeze claim; Colab-safe. Paste order:
+geolip_vitals -> ar_differentiation_bed -> exp013_augmentation_bed -> this.
+"""
+from __future__ import annotations
+import json
+import math
+import os
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+if "MslRelay" not in globals():
+    try:
+        from ar_differentiation_bed import MslRelay
+        from exp013_augmentation_bed import MLPAdapter
+        from geolip_vitals import anchor_drift, axis_aliveness
+    except ImportError:
+        _here = globals().get("__file__")
+        if _here is None:
+            raise ImportError("paste geolip_vitals + ar_differentiation_bed + "
+                              "exp013_augmentation_bed first")
+        import sys, pathlib
+        sys.path.insert(0, str(pathlib.Path(_here).parent))
+        from ar_differentiation_bed import MslRelay
+        from exp013_augmentation_bed import MLPAdapter
+        from geolip_vitals import anchor_drift, axis_aliveness
+
+DATA_ROOT = os.environ.get("GEOLIP_DATA", "./data")
+EXP_DIR = os.path.join(DATA_ROOT, "qwen_exp001")
+BASE_MODEL = "Qwen/Qwen2.5-0.5B"
+BLOCK, BATCH, STEPS, LR = 512, 4, 3000, 1e-3
+MAX_TOKENS = 12_000_000
+PROMPTS = ("The history of mathematics begins",
+           "In a small village by the sea,",
+           "The most important principle of engineering is")
+
+
+class _QwenBlockWithAdapter(nn.Module):
+    """Wrap a Qwen2 decoder layer: adapter applied to the hidden-states output,
+    all other outputs and kwargs passed through untouched."""
+
+    def __init__(self, block, adapter):
+        super().__init__()
+        self.block, self.adapter = block, adapter
+
+    def forward(self, *args, **kwargs):
+        out = self.block(*args, **kwargs)
+        if isinstance(out, tuple):
+            return (self.adapter(out[0]),) + out[1:]
+        return self.adapter(out)
+
+
+def _token_cache(device="cpu"):
+    """~12M-token wikitext-103 train stream + val stream, cached to disk."""
+    os.makedirs(EXP_DIR, exist_ok=True)
+    path = os.path.join(EXP_DIR, "tok_cache.pt")
+    if os.path.exists(path):
+        blob = torch.load(path, map_location="cpu", weights_only=True)
+        return blob["train"], blob["val"]
+    from huggingface_hub import hf_hub_download
+    import pyarrow.parquet as pq
+    from transformers import AutoTokenizer
+    tok = AutoTokenizer.from_pretrained(BASE_MODEL)
+
+    def stream(fname, cap):
+        p = hf_hub_download("Salesforce/wikitext", fname, repo_type="dataset")
+        lines = [t for t in pq.read_table(p).column("text").to_pylist()
+                 if t and len(t.strip()) > 40]
+        ids = []
+        total = 0
+        chunk = []
+        csz = 0
+        for ln in lines:
+            chunk.append(ln)
+            csz += len(ln)
+            if csz > 500_000:
+                e = tok("".join(chunk), return_tensors="pt").input_ids[0]
+                ids.append(e)
+                total += e.numel()
+                chunk, csz = [], 0
+                if total >= cap:
+                    break
+        if chunk and total < cap:
+            e = tok("".join(chunk), return_tensors="pt").input_ids[0]
+            ids.append(e)
+        return torch.cat(ids)[:cap]
+
+    tr = stream("wikitext-103-raw-v1/train-00000-of-00002.parquet", MAX_TOKENS)
+    va = stream("wikitext-103-raw-v1/validation-00000-of-00001.parquet",
+                600_000)
+    torch.save({"train": tr, "val": va}, path)
+    print(f"token cache: train {tr.numel():,} val {va.numel():,}", flush=True)
+    return tr, va
+
+
+def _batch(stream, batch, block, device, g):
+    ix = torch.randint(0, stream.numel() - block - 1, (batch,), generator=g)
+    x = torch.stack([stream[i:i + block] for i in ix]).to(device)
+    y = torch.stack([stream[i + 1:i + block + 1] for i in ix]).to(device)
+    return x.long(), y.long()
+
+
+def load_qwen(arm: str, seed: int = 0, device="cuda"):
+    from transformers import AutoModelForCausalLM
+    torch.manual_seed(seed)
+    model = AutoModelForCausalLM.from_pretrained(BASE_MODEL,
+                                                 torch_dtype=torch.float32)
+    model.config.use_cache = False
+    for p in model.parameters():
+        p.requires_grad_(False)
+    adapters = None
+    if arm != "frozen":
+        d = model.config.hidden_size
+        mk = (lambda: MslRelay(d)) if arm == "relay" else (lambda: MLPAdapter(d))
+        adapters = nn.ModuleList([mk() for _ in model.model.layers])
+        model.model.layers = nn.ModuleList(
+            [_QwenBlockWithAdapter(b, a)
+             for b, a in zip(model.model.layers, adapters)])
+    return model.to(device), adapters
+
+
+@torch.no_grad()
+def eval_ppl(model, va, device="cuda", n=40, g=None):
+    g = g or torch.Generator().manual_seed(0)
+    model.eval()
+    ls = []
+    for _ in range(n):
+        x, y = _batch(va, BATCH, BLOCK, device, g)
+        logits = model(input_ids=x).logits
+        ls.append(F.cross_entropy(logits.reshape(-1, logits.shape[-1]),
+                                  y.reshape(-1)).item())
+    return math.exp(sum(ls) / len(ls))
+
+
+@torch.no_grad()
+def sample(model, device="cuda", max_new=80):
+    from transformers import AutoTokenizer
+    tok = AutoTokenizer.from_pretrained(BASE_MODEL)
+    model.eval()
+    model.config.use_cache = True
+    outs = {}
+    for p in PROMPTS:
+        ids = tok(p, return_tensors="pt").input_ids.to(device)
+        out = model.generate(ids, max_new_tokens=max_new, do_sample=False,
+                             pad_token_id=tok.eos_token_id)
+        outs[p] = tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True)
+    model.config.use_cache = False
+    return outs
+
+
+def adapter_vitals(adapters):
+    if adapters is None:
+        return None
+    gates = [round(torch.sigmoid(a.gate).item(), 4) for a in adapters
+             if hasattr(a, "gate")]
+    drifts = []
+    for a in adapters:
+        ad = getattr(a, "addr", None)
+        if ad is not None:
+            drifts.append(round(anchor_drift(ad.codebook, ad.home)["mean"], 3))
+    return {"gates": gates[:6] + ["..."] if len(gates) > 6 else gates,
+            "gate_mean": round(sum(gates) / max(len(gates), 1), 4),
+            "drift": drifts[:6] + ["..."] if len(drifts) > 6 else drifts}
+
+
+def run_arm(arm: str, seed: int = 0, steps=STEPS, device="cuda"):
+    if not torch.cuda.is_available():
+        raise RuntimeError("verdict runs are GPU-only")
+    os.makedirs(EXP_DIR, exist_ok=True)
+    tr, va = _token_cache()
+    model, adapters = load_qwen(arm, seed=seed, device=device)
+    ledger = open(os.path.join(EXP_DIR, "ledger.jsonl"), "a", encoding="utf-8")
+    if arm == "frozen":
+        ppl = eval_ppl(model, va, device=device)
+        rec = {"exp": "q1", "arm": arm, "seed": seed, "ppl": round(ppl, 3),
+               "trainable": 0, "samples": sample(model, device=device)}
+        ledger.write(json.dumps(rec) + "\n"); ledger.flush()
+        print(f"[q1 frozen s{seed}] FINAL ppl={ppl:.3f}", flush=True)
+        ledger.close()
+        return ppl
+    g = torch.Generator().manual_seed(seed)
+    params = [p for p in adapters.parameters()]
+    n_train = sum(p.numel() for p in params)
+    opt = torch.optim.Adam(params, lr=LR, weight_decay=0.0)
+    model.train()
+    for step in range(1, steps + 1):
+        x, y = _batch(tr, BATCH, BLOCK, device, g)
+        logits = model(input_ids=x).logits
+        loss = F.cross_entropy(logits.reshape(-1, logits.shape[-1]),
+                               y.reshape(-1))
+        opt.zero_grad(set_to_none=True); loss.backward(); opt.step()
+        if step % 500 == 0:
+            v = adapter_vitals(adapters)
+            print(f"[q1 {arm} s{seed} step {step}] loss={loss.item():.3f} "
+                  f"gate_mean={v['gate_mean']}", flush=True)
+    ppl = eval_ppl(model, va, device=device)
+    rec = {"exp": "q1", "arm": arm, "seed": seed, "ppl": round(ppl, 3),
+           "steps": steps, "trainable": n_train,
+           "vitals": adapter_vitals(adapters),
+           "samples": sample(model, device=device)}
+    ledger.write(json.dumps(rec) + "\n"); ledger.flush()
+    print(f"[q1 {arm} s{seed}] FINAL ppl={ppl:.3f} "
+          f"gate_mean={rec['vitals']['gate_mean']} trainable={n_train:,}",
+          flush=True)
+    torch.save({"arm": arm, "seed": seed,
+                "adapters": {k: v.cpu() for k, v in
+                             adapters.state_dict().items()}},
+               os.path.join(EXP_DIR, f"q1_{arm}_s{seed}.pt"))
+    ledger.close()
+    del model, adapters
+    torch.cuda.empty_cache()
+    return ppl
+
+
+def run_exp001(seeds=(0, 1), device="cuda"):
+    run_arm("frozen", seed=0, device=device)      # baseline once (no training)
+    for seed in seeds:
+        for arm in ("relay", "mlp"):
+            run_arm(arm, seed=seed, device=device)
+    print("=== qwen exp001 COMPLETE ===", flush=True)
+
+
+def smoke():
+    """Shapes/parse only — no substrate download, no training."""
+    r = MslRelay(896)
+    m = MLPAdapter(896)
+    x = torch.randn(2, 8, 896)
+    assert r(x).shape == x.shape and m(x).shape == x.shape
+    rp = sum(p.numel() for p in r.parameters())
+    mp = sum(p.numel() for p in m.parameters())
+    assert abs(rp - mp) / rp < 0.01, (rp, mp)      # param-matched adapters
+    blk = nn.Linear(896, 896)                       # tuple-passthrough check
+    class TupBlock(nn.Module):
+        def forward(self, h, **kw):
+            return (blk(h), "aux")
+    w = _QwenBlockWithAdapter(TupBlock(), r)
+    out = w(x, position_ids=None)
+    assert isinstance(out, tuple) and out[0].shape == x.shape and out[1] == "aux"
+    (out[0].sum()).backward()
+    assert r.addr.codebook.grad is not None
+    print(f"qwen exp001 smoke passed (relay {rp:,} ~ mlp {mp:,} params/adapter;"
+          " full run needs GPU + transformers + the 0.5B download)")
+
+
+def _in_notebook():
+    try:
+        get_ipython()  # type: ignore[name-defined]  # noqa: F821
+        return True
+    except NameError:
+        return False
+
+
+if __name__ == "__main__":
+    smoke() if not _in_notebook() else (smoke(),
+        print("Notebook: run_exp001() on GPU."))

exp007_math/qwen_exp002_refine.py

@@ -0,0 +1,237 @@
+"""qwen_exp002_refine.py — geolip-aleph-qwen EXPERIMENT 2: refine the relay.
+exp001 verdict shape: the relay retrofit works at 0.5B (frozen 17.80 ->
+~14.09, gates grow 0.047 -> 0.081 = trunk opt-in) but the param-matched MLP
+adapter is at parity-or-ahead on ppl — the certified GPT-2 ordering did not
+transfer as-is. exp002 races PROTOTYPICAL ENHANCEMENTS of the relay, each
+grounded in a certified law, at an equalized WIDER budget (~230K/adapter),
+against the honest wide MLP control.
+
+Arms (frozen Qwen2.5-0.5B trunk, adapters after every block unless noted):
+  relay32    — MslRelay with n_slots=32 (slot dose-response: exp012 L-AR3,
+               task monotone in slot width)
+  relay_3tau — multi-slot read at 3 temperatures (0.05/0.1/0.3), concatenated
+               (the certified rule-of-3 stroboscope, exp012 v2: -1.08 bpb)
+  relay_pw   — slots -> M_hat -> PATCHWORK consumer Linear(64,178) ->
+               SquaredReLU -> LN -> Linear(178,896) (the constellation
+               consumption spec replaces the bare linear out)
+  relay_deep — exp001's TOTAL budget concentrated on the last 12 blocks
+               (n_slots=32 there, nothing on blocks 0-11): the depth-gradient
+               law (cultivation concentrates near the prediction gradient)
+               as an architecture decision; total params == exp001 relay
+  mlp_wide   — MLPAdapter hidden=128 (the wide-budget capacity control)
+All near-zero gated (init -3.0). Training identical to exp001: block 512,
+batch 4, 3000 steps, pure Adam lr 1e-3 wd 0, adapters only. Judged: val ppl
+vs exp001's relay (14.093/14.091) and mlp (13.927/...) + gate mechanism +
+vitals + generated samples. Seed 0 sweep first; seed 1 for the top arms.
+Riders: trunk frozen; >=2 seeds before any claim; GPU-only; Colab-safe.
+Paste order: geolip_vitals -> ar_differentiation_bed ->
+exp013_augmentation_bed -> qwen_exp001_relay -> this file.
+"""
+from __future__ import annotations
+import json
+import math
+import os
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+if "run_arm" not in globals():
+    try:
+        from ar_differentiation_bed import AlephAddress, MslRelay
+        from exp013_augmentation_bed import MLPAdapter
+        from geolip_vitals import anchor_drift
+        from qwen_exp001_relay import (_QwenBlockWithAdapter, _token_cache,
+                                       _batch, eval_ppl, sample, EXP_DIR as
+                                       Q1_DIR, BASE_MODEL, BLOCK, BATCH, LR)
+    except ImportError:
+        _here = globals().get("__file__")
+        if _here is None:
+            raise ImportError("paste the qwen stack first")
+        import sys, pathlib
+        sys.path.insert(0, str(pathlib.Path(_here).parent))
+        from ar_differentiation_bed import AlephAddress, MslRelay
+        from exp013_augmentation_bed import MLPAdapter
+        from geolip_vitals import anchor_drift
+        from qwen_exp001_relay import (_QwenBlockWithAdapter, _token_cache,
+                                       _batch, eval_ppl, sample, EXP_DIR as
+                                       Q1_DIR, BASE_MODEL, BLOCK, BATCH, LR)
+
+EXP2_DIR = os.path.join(os.path.dirname(Q1_DIR), "qwen_exp002")
+STEPS = 3000
+
+
+class SquaredReLU(nn.Module):
+    def forward(self, x):
+        return F.relu(x) ** 2
+
+
+class Relay3Tau(nn.Module):
+    """Multi-slot M_hat at 3 temperatures, concatenated (rule-of-3 strobe)."""
+    TAUS = (0.05, 0.1, 0.3)
+
+    def __init__(self, d: int, n_slots: int = 16, K: int = 64):
+        super().__init__()
+        self.n_slots = n_slots
+        self.proj = nn.Linear(d, n_slots * 4, bias=False)
+        self.out = nn.Linear(n_slots * 4 * len(self.TAUS), d, bias=False)
+        nn.init.orthogonal_(self.proj.weight)
+        nn.init.orthogonal_(self.out.weight)
+        self.addr = AlephAddress(K, 4)
+        self.gate = nn.Parameter(torch.tensor(-3.0))
+
+    def forward(self, x):
+        B, n, _ = x.shape
+        slots = self.proj(x).view(B, n, self.n_slots, 4)
+        feats = []
+        for t in self.TAUS:
+            u = self.addr._u(slots) * (self.addr.tau / t)
+            m = u.abs().amax(dim=-1, keepdim=True)
+            ep, en = torch.exp(u - m), torch.exp(-u - m)
+            A = F.normalize(self.addr.codebook, dim=-1)
+            feats.append((((ep - en) @ A)
+                          / (ep + en).sum(dim=-1, keepdim=True)).reshape(B, n, -1))
+        return x + torch.sigmoid(self.gate) * self.out(torch.cat(feats, -1))
+
+
+class RelayPatchwork(nn.Module):
+    """Multi-slot M_hat -> constellation-spec consumer (SquaredReLU patchwork)."""
+
+    def __init__(self, d: int, n_slots: int = 16, K: int = 64, hidden: int = 178):
+        super().__init__()
+        self.n_slots = n_slots
+        self.proj = nn.Linear(d, n_slots * 4, bias=False)
+        nn.init.orthogonal_(self.proj.weight)
+        self.addr = AlephAddress(K, 4)
+        self.consume = nn.Sequential(
+            nn.Linear(n_slots * 4, hidden), SquaredReLU(),
+            nn.LayerNorm(hidden), nn.Linear(hidden, d))
+        nn.init.zeros_(self.consume[-1].weight)      # zero-init out (theme D)
+        self.gate = nn.Parameter(torch.tensor(-3.0))
+
+    def forward(self, x):
+        B, n, _ = x.shape
+        slots = self.proj(x).view(B, n, self.n_slots, 4)
+        feats = self.addr.m_hat(slots).reshape(B, n, -1)
+        return x + torch.sigmoid(self.gate) * self.consume(feats)
+
+
+def build_adapters(arm: str, model):
+    d = model.config.hidden_size
+    L = len(model.model.layers)
+    if arm == "relay32":
+        mk = [lambda: MslRelay(d, n_slots=32) for _ in range(L)]
+    elif arm == "relay_3tau":
+        mk = [lambda: Relay3Tau(d) for _ in range(L)]
+    elif arm == "relay_pw":
+        mk = [lambda: RelayPatchwork(d) for _ in range(L)]
+    elif arm == "relay_deep":
+        mk = [None] * (L // 2) + [lambda: MslRelay(d, n_slots=32)
+                                  for _ in range(L - L // 2)]
+    elif arm == "mlp_wide":
+        mk = [lambda: MLPAdapter(d, hidden=128) for _ in range(L)]
+    else:
+        raise ValueError(arm)
+    adapters = nn.ModuleList([m() if m else nn.Identity() for m in mk])
+    model.model.layers = nn.ModuleList(
+        [_QwenBlockWithAdapter(b, a) if not isinstance(a, nn.Identity) else b
+         for b, a in zip(model.model.layers, adapters)])
+    return adapters
+
+
+def adapter_vitals(adapters):
+    gates, drifts = [], []
+    for a in adapters:
+        if hasattr(a, "gate"):
+            gates.append(round(torch.sigmoid(a.gate).item(), 4))
+        ad = getattr(a, "addr", None)
+        if ad is not None:
+            drifts.append(round(anchor_drift(ad.codebook, ad.home)["mean"], 3))
+    return {"gate_mean": round(sum(gates) / max(len(gates), 1), 4),
+            "gates_head_tail": gates[:3] + gates[-3:],
+            "drift_head_tail": (drifts[:3] + drifts[-3:]) if drifts else None}
+
+
+def run_arm2(arm: str, seed: int = 0, steps=STEPS, device="cuda"):
+    if not torch.cuda.is_available():
+        raise RuntimeError("verdict runs are GPU-only")
+    os.makedirs(EXP2_DIR, exist_ok=True)
+    from transformers import AutoModelForCausalLM
+    tr, va = _token_cache()
+    torch.manual_seed(seed)
+    model = AutoModelForCausalLM.from_pretrained(BASE_MODEL,
+                                                 torch_dtype=torch.float32)
+    model.config.use_cache = False
+    for p in model.parameters():
+        p.requires_grad_(False)
+    adapters = build_adapters(arm, model)
+    model = model.to(device)
+    n_train = sum(p.numel() for p in adapters.parameters())
+    g = torch.Generator().manual_seed(seed)
+    opt = torch.optim.Adam(adapters.parameters(), lr=LR, weight_decay=0.0)
+    model.train()
+    for step in range(1, steps + 1):
+        x, y = _batch(tr, BATCH, BLOCK, device, g)
+        logits = model(input_ids=x).logits
+        loss = F.cross_entropy(logits.reshape(-1, logits.shape[-1]),
+                               y.reshape(-1))
+        opt.zero_grad(set_to_none=True); loss.backward(); opt.step()
+        if step % 1000 == 0:
+            print(f"[q2 {arm} s{seed} step {step}] loss={loss.item():.3f}",
+                  flush=True)
+    ppl = eval_ppl(model, va, device=device)
+    v = adapter_vitals(adapters)
+    rec = {"exp": "q2", "arm": arm, "seed": seed, "ppl": round(ppl, 3),
+           "steps": steps, "trainable": n_train, "vitals": v,
+           "samples": sample(model, device=device)}
+    ledger = open(os.path.join(EXP2_DIR, "ledger.jsonl"), "a", encoding="utf-8")
+    ledger.write(json.dumps(rec) + "\n"); ledger.close()
+    print(f"[q2 {arm} s{seed}] FINAL ppl={ppl:.3f} gate_mean={v['gate_mean']} "
+          f"trainable={n_train:,}", flush=True)
+    torch.save({"arm": arm, "seed": seed,
+                "adapters": {k: v2.cpu() for k, v2 in
+                             adapters.state_dict().items()}},
+               os.path.join(EXP2_DIR, f"q2_{arm}_s{seed}.pt"))
+    del model, adapters
+    torch.cuda.empty_cache()
+    return ppl
+
+
+ARMS = ("relay32", "relay_3tau", "relay_pw", "relay_deep", "mlp_wide")
+
+
+def run_wave_a(arms=ARMS, seed=0, device="cuda"):
+    for arm in arms:
+        run_arm2(arm, seed=seed, device=device)
+    print("=== qwen exp002 WAVE A COMPLETE ===", flush=True)
+
+
+def smoke():
+    x = torch.randn(2, 8, 896)
+    for cls, kw in ((Relay3Tau, {}), (RelayPatchwork, {}),):
+        a = cls(896, **kw)
+        y = a(x)
+        assert y.shape == x.shape
+        y.sum().backward()
+        assert a.addr.codebook.grad is not None
+        a.zero_grad()
+    p32 = sum(p.numel() for p in MslRelay(896, n_slots=32).parameters())
+    p3t = sum(p.numel() for p in Relay3Tau(896).parameters())
+    ppw = sum(p.numel() for p in RelayPatchwork(896).parameters())
+    pmw = sum(p.numel() for p in MLPAdapter(896, hidden=128).parameters())
+    lo, hi = min(p32, p3t, ppw, pmw), max(p32, p3t, ppw, pmw)
+    assert (hi - lo) / hi < 0.12, (p32, p3t, ppw, pmw)   # budget-equalized ~10%
+    print(f"qwen exp002 smoke passed (relay32 {p32:,} | 3tau {p3t:,} | "
+          f"pw {ppw:,} | mlp_wide {pmw:,} per adapter)")
+
+
+def _in_notebook():
+    try:
+        get_ipython()  # type: ignore[name-defined]  # noqa: F821
+        return True
+    except NameError:
+        return False
+
+
+if __name__ == "__main__":
+    smoke() if not _in_notebook() else (smoke(),
+        print("Notebook: run_wave_a() on GPU."))

exp007_math/qwen_exp003_instruct.py

@@ -0,0 +1,451 @@
+"""qwen_exp003_instruct.py — geolip-aleph-qwen EXPERIMENT 3: instruct-task
+adapters + register differentiation on the discrete surface.
+Hypothesis under test (Phil): continuation-trained adapters "double up"
+values — aligning REGISTERS to overlapping internal states; instruct-task
+training should differentiate them cleanly. The json-coco-format dataset
+(AbstractPhil/json-coco-format) is the ideal instrument: the SAME 22k COCO
+captions pass through THREE structurally distinct output registers —
+task_1 hallucination_reduction (grounded literal extraction), task_2
+useful_generalization (bracketed generics), task_3 generic_symbolism
+(positional placeholders) — in native messages+tools shape.
+
+Substrate: Qwen2.5-0.5B-INSTRUCT, frozen; adapters = the exp002-certified
+relay_pw (aleph addressing + constellation patchwork consumer) after every
+block. Arms (x2 seeds): frozen (zero-shot) | relay_pw_task1 (the
+hallucination-reduction cell — Phil's prior task-1 adapter, reproduced on the
+relay stack) | relay_pw_all (multi-task on all three) | mlp_all
+(MLPAdapter(128), budget-matched control; its 1/2-seed divergence rate at
+this width is itself a standing datum).
+
+JUDGES:
+  1. held-out per-task PPL (split BY CAPTION hash — no caption leaks across
+     tasks or splits);
+  2. TASK VALIDITY on generated outputs (validators re-implemented from the
+     dataset README: JSON parse + schema keys; task_1 per-leaf grounding
+     substring check; task_2 ^\\[[a-z_]+\\]$; task_3 typed monotonic
+     placeholders) — hallucination reduction as a measured rate;
+  3. THE REGISTER PROBE: sign-codes from the relay stack for the same held-out
+     captions under all four registers (3 task system-prompts + plain
+     continuation); separation = mean inter-register Hamming minus mean
+     intra-register Hamming, per layer — measured on BOTH the instruct-trained
+     stack and the exp002 WIKITEXT-trained stack (the overlap-vs-differentiate
+     comparison the hypothesis asks for).
+Training: block 512, batch 4, 3000 steps, adapters only, pure Adam lr 1e-3
+wd 0 (the certified regime). Full-conversation LM loss (no prompt masking) —
+consistent with the line's stream training; noted as an instrument property.
+Riders: trunk frozen; >=2 seeds; GPU-only; Colab-safe. Paste order:
+geolip_vitals -> ar_differentiation_bed -> exp013_augmentation_bed ->
+qwen_exp001_relay -> qwen_exp002_refine -> this file.
+"""
+from __future__ import annotations
+import json
+import math
+import os
+import re
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+if "RelayPatchwork" not in globals():
+    try:
+        from exp013_augmentation_bed import MLPAdapter
+        from qwen_exp001_relay import _QwenBlockWithAdapter, _batch
+        from qwen_exp002_refine import RelayPatchwork
+    except ImportError:
+        _here = globals().get("__file__")
+        if _here is None:
+            raise ImportError("paste the qwen stack first")
+        import sys, pathlib
+        sys.path.insert(0, str(pathlib.Path(_here).parent))
+        from exp013_augmentation_bed import MLPAdapter
+        from qwen_exp001_relay import _QwenBlockWithAdapter, _batch
+        from qwen_exp002_refine import RelayPatchwork
+
+DATA_ROOT = os.environ.get("GEOLIP_DATA", "./data")
+EXP3_DIR = os.path.join(DATA_ROOT, "qwen_exp003")
+DATASET = "AbstractPhil/json-coco-format"
+INSTRUCT_MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
+BLOCK, BATCH, STEPS, LR = 512, 4, 3000, 1e-3
+TASKS = ("task_1", "task_2", "task_3")
+HOLDOUT_MOD = 50                    # caption-hash % 50 == 0 -> held out (~2%)
+
+
+# ================================ data ============================================
+def _rows(task):
+    from huggingface_hub import hf_hub_download
+    p = hf_hub_download(DATASET, f"data/{task}.jsonl", repo_type="dataset")
+    return [json.loads(l) for l in open(p, encoding="utf-8")]
+
+
+def _render(tok, row, gen_prompt=False):
+    """Chat-template render; stringify tool arguments if the template needs it."""
+    msgs = row["messages"][:2] if gen_prompt else row["messages"]
+    try:
+        return tok.apply_chat_template(msgs, tools=row["tools"],
+                                       tokenize=False,
+                                       add_generation_prompt=gen_prompt)
+    except Exception:
+        msgs = json.loads(json.dumps(msgs))
+        for m in msgs:
+            for tc in m.get("tool_calls", []) or []:
+                a = tc["function"]["arguments"]
+                if isinstance(a, dict):
+                    tc["function"]["arguments"] = json.dumps(a)
+        return tok.apply_chat_template(msgs, tools=row["tools"],
+                                       tokenize=False,
+                                       add_generation_prompt=gen_prompt)
+
+
+def _is_holdout(prompt: str) -> bool:
+    import hashlib
+    return int(hashlib.md5(prompt.encode()).hexdigest(), 16) % HOLDOUT_MOD == 0
+
+
+def build_caches():
+    """Token streams per task split (train/holdout) + eval row lists, cached."""
+    os.makedirs(EXP3_DIR, exist_ok=True)
+    path = os.path.join(EXP3_DIR, "instruct_cache.pt")
+    if os.path.exists(path):
+        return torch.load(path, map_location="cpu", weights_only=False)
+    from transformers import AutoTokenizer
+    tok = AutoTokenizer.from_pretrained(INSTRUCT_MODEL)
+    blob = {"train": {}, "hold": {}, "eval_rows": {}}
+    for task in TASKS:
+        rows = _rows(task)
+        tr_txt, ho_txt, ev = [], [], []
+        for r in rows:
+            if _is_holdout(r["prompt"]):
+                ho_txt.append(_render(tok, r))
+                if len(ev) < 64:
+                    ev.append({"prompt": r["prompt"],
+                               "gen_prompt": _render(tok, r, gen_prompt=True)})
+            else:
+                tr_txt.append(_render(tok, r))
+        def stream(txts):
+            ids = tok("\n".join(txts), return_tensors="pt").input_ids[0]
+            return ids
+        blob["train"][task] = stream(tr_txt)
+        blob["hold"][task] = stream(ho_txt)
+        blob["eval_rows"][task] = ev
+        print(f"[cache] {task}: train {blob['train'][task].numel():,} tok, "
+              f"hold {blob['hold'][task].numel():,} tok, eval {len(ev)} rows",
+              flush=True)
+    torch.save(blob, path)
+    return blob
+
+
+# ============================ validators (per the dataset README) =================
+# core content keys; style/mood are constant-null in the data and models may
+# legitimately omit them (validators only read subjects/actions/setting)
+SCHEMA_KEYS = {"subjects", "actions", "setting"}
+
+
+def parse_tool_json(text: str):
+    # GREEDY inner match — the arguments JSON is nested; a non-greedy match
+    # truncates at the first '}' (the v1 parser bug, caught by a perfect
+    # generation scoring 0)
+    m = re.search(r"<tool_call>\s*(\{.*\})\s*</tool_call>", text, re.S)
+    raw = m.group(1) if m else None
+    if raw is None:
+        m = re.search(r"\{.*\}", text, re.S)
+        raw = m.group(0) if m else None
+    if raw is None:
+        return None
+    try:
+        obj = json.loads(raw)
+    except Exception:
+        return None
+    if isinstance(obj, dict) and "arguments" in obj:
+        obj = obj["arguments"]
+        if isinstance(obj, str):
+            try:
+                obj = json.loads(obj)
+            except Exception:
+                return None
+    return obj if isinstance(obj, dict) else None
+
+
+def _leaves(args, with_setting=True):
+    for s in args.get("subjects") or []:
+        if isinstance(s, dict):
+            if s.get("name"):
+                yield str(s["name"])
+            for a in s.get("attributes") or []:
+                yield str(a)
+    for a in args.get("actions") or []:
+        yield str(a)
+    # setting is an ENUM categorization in task_1 (not verbatim-grounded);
+    # format checks in task_2/3 do cover it
+    if with_setting and args.get("setting"):
+        yield str(args["setting"])
+
+
+def validate(task: str, args: dict, caption: str):
+    if args is None or not SCHEMA_KEYS.issubset(args.keys()):
+        return False
+    leaves = list(_leaves(args, with_setting=(task != "task_1")))
+    if not leaves:
+        return False
+    if task == "task_1":
+        cap = caption.lower()
+        return all(all(w in cap for w in lf.lower().split()) for lf in leaves)
+    if task == "task_2":
+        return all(re.fullmatch(r"\[[a-z_]+\]", lf) for lf in leaves)
+    if task == "task_3":
+        ok, counters = True, {}
+        for lf in leaves:
+            m = re.fullmatch(r"\[(ENTITY|ATTRIBUTE|ACTION|[A-Z_]+?)(?:_(\d+))?\]", lf)
+            if not m:
+                return False
+            if m.group(2):
+                k, n = m.group(1), int(m.group(2))
+                ok &= n == counters.get(k, 0) + 1 or n <= counters.get(k, 0) + 1
+                counters[k] = max(counters.get(k, 0), n)
+        return ok
+    return False
+
+
+# ================================ arms ============================================
+def load_instruct(arm: str, seed: int = 0, device="cuda"):
+    from transformers import AutoModelForCausalLM
+    torch.manual_seed(seed)
+    model = AutoModelForCausalLM.from_pretrained(INSTRUCT_MODEL,
+                                                 dtype=torch.float32)
+    model.config.use_cache = False
+    for p in model.parameters():
+        p.requires_grad_(False)
+    adapters = None
+    if arm != "frozen":
+        d = model.config.hidden_size
+        mk = (lambda: RelayPatchwork(d)) if arm.startswith("relay_pw") \
+            else (lambda: MLPAdapter(d, hidden=128))
+        adapters = nn.ModuleList([mk() for _ in model.model.layers])
+        model.model.layers = nn.ModuleList(
+            [_QwenBlockWithAdapter(b, a)
+             for b, a in zip(model.model.layers, adapters)])
+    return model.to(device), adapters
+
+
+@torch.no_grad()
+def ppl_per_task(model, blob, device="cuda", n=25):
+    out = {}
+    for task in TASKS:
+        g = torch.Generator().manual_seed(0)
+        ls = []
+        for _ in range(n):
+            x, y = _batch(blob["hold"][task], BATCH, BLOCK, device, g)
+            logits = model(input_ids=x).logits
+            ls.append(F.cross_entropy(logits.reshape(-1, logits.shape[-1]),
+                                      y.reshape(-1)).item())
+        out[task] = round(math.exp(sum(ls) / len(ls)), 3)
+    return out
+
+
+@torch.no_grad()
+def task_validity(model, blob, device="cuda", n_eval=48, max_new=200):
+    from transformers import AutoTokenizer
+    tok = AutoTokenizer.from_pretrained(INSTRUCT_MODEL)
+    model.eval()
+    model.config.use_cache = True
+    out = {}
+    for task in TASKS:
+        rows = blob["eval_rows"][task][:n_eval]
+        js = va = 0
+        for r in rows:
+            ids = tok(r["gen_prompt"], return_tensors="pt").input_ids.to(device)
+            gen = model.generate(ids, max_new_tokens=max_new, do_sample=False,
+                                 pad_token_id=tok.eos_token_id)
+            txt = tok.decode(gen[0][ids.shape[1]:], skip_special_tokens=False)
+            args = parse_tool_json(txt)
+            if args is not None and SCHEMA_KEYS.issubset(args.keys()):
+                js += 1
+                if validate(task, args, r["prompt"]):
+                    va += 1
+        out[task] = {"json_valid": round(js / len(rows), 4),
+                     "task_valid": round(va / len(rows), 4)}
+    model.config.use_cache = False
+    return out
+
+
+# ============================ the register probe ==================================
+@torch.no_grad()
+def register_probe(model, adapters, blob, device="cuda", n_caps=24):
+    """Sign-codes per relay layer for the SAME captions under 4 registers
+    (task_1/2/3 gen-prompts + plain continuation). Separation per layer =
+    mean inter-register Hamming - mean intra-register Hamming."""
+    from transformers import AutoTokenizer
+    tok = AutoTokenizer.from_pretrained(INSTRUCT_MODEL)
+    model.eval()
+    caps = [r["prompt"] for r in blob["eval_rows"]["task_1"][:n_caps]]
+    regs = {}
+    for task in TASKS:
+        by_prompt = {r["prompt"]: r["gen_prompt"]
+                     for r in blob["eval_rows"][task]}
+        regs[task] = [by_prompt.get(c, "") for c in caps]
+    regs["plain"] = [c for c in caps]
+    layer_ids = [0, 8, 16, 23]
+    # hook-based collection (one pass per register, hooks read the codes)
+    codes = {reg: {i: [] for i in layer_ids} for reg in regs}
+    for reg, prompts in regs.items():
+        hooks, store = [], {i: [] for i in layer_ids}
+
+        def mk(i, a):
+            def h(mod, inp, out):
+                x = inp[0][:, -1:]
+                slots = a.proj(x).view(*x.shape[:-1], a.n_slots, 4)
+                A = F.normalize(a.addr.codebook, dim=-1)
+                cos = F.normalize(slots, dim=-1) @ A.T
+                win = cos.abs().argmax(-1)
+                sgn = (torch.gather(cos, -1, win.unsqueeze(-1))
+                       .squeeze(-1) > 0).long()
+                store[i].append((win * 2 + sgn).reshape(-1).cpu())
+            return h
+        for i in layer_ids:
+            hooks.append(adapters[i].register_forward_hook(mk(i, adapters[i])))
+        for pr in prompts:
+            if pr:
+                ids = tok(pr, return_tensors="pt").input_ids.to(device)
+                model(input_ids=ids)
+        for h in hooks:
+            h.remove()
+        for i in layer_ids:
+            codes[reg][i] = torch.stack(store[i]) if store[i] else None
+    sep = {}
+    for i in layer_ids:
+        mats = {r: codes[r][i] for r in codes if codes[r][i] is not None}
+        names = list(mats)
+        def ham(a, b):
+            return (a.unsqueeze(1) != b.unsqueeze(0)).float().mean().item()
+        intra = sum(ham(mats[r], mats[r]) for r in names) / len(names)
+        pairs = [(a, b) for ai, a in enumerate(names) for b in names[ai + 1:]]
+        inter = sum(ham(mats[a], mats[b]) for a, b in pairs) / len(pairs)
+        sep[f"L{i}"] = {"inter": round(inter, 4), "intra": round(intra, 4),
+                        "sep": round(inter - intra, 4)}
+    return sep
+
+
+# ================================ runner ==========================================
+def train_arm3(arm: str, seed: int = 0, steps=STEPS, device="cuda"):
+    if not torch.cuda.is_available():
+        raise RuntimeError("verdict runs are GPU-only")
+    os.makedirs(EXP3_DIR, exist_ok=True)
+    blob = build_caches()
+    model, adapters = load_instruct(arm, seed=seed, device=device)
+    ledger = open(os.path.join(EXP3_DIR, "ledger.jsonl"), "a", encoding="utf-8")
+    if arm == "frozen":
+        rec = {"exp": "q3", "arm": arm, "seed": seed,
+               "ppl": ppl_per_task(model, blob, device=device),
+               "validity": task_validity(model, blob, device=device)}
+        ledger.write(json.dumps(rec) + "\n"); ledger.close()
+        print(f"[q3 frozen] FINAL {rec['ppl']} {rec['validity']}", flush=True)
+        return
+    # training stream: task_1 only, or all three interleaved
+    tasks = ("task_1",) if arm.endswith("task1") else TASKS
+    g = torch.Generator().manual_seed(seed)
+    opt = torch.optim.Adam(adapters.parameters(), lr=LR, weight_decay=0.0)
+    model.train()
+    for step in range(1, steps + 1):
+        task = tasks[int(torch.randint(len(tasks), (1,), generator=g))]
+        x, y = _batch(blob["train"][task], BATCH, BLOCK, device, g)
+        logits = model(input_ids=x).logits
+        loss = F.cross_entropy(logits.reshape(-1, logits.shape[-1]),
+                               y.reshape(-1))
+        opt.zero_grad(set_to_none=True); loss.backward(); opt.step()
+        if step % 1000 == 0:
+            print(f"[q3 {arm} s{seed} step {step}] loss={loss.item():.3f}",
+                  flush=True)
+    rec = {"exp": "q3", "arm": arm, "seed": seed, "steps": steps,
+           "trainable": sum(p.numel() for p in adapters.parameters()),
+           "ppl": ppl_per_task(model, blob, device=device),
+           "validity": task_validity(model, blob, device=device)}
+    if arm.startswith("relay_pw"):
+        rec["register_sep"] = register_probe(model, adapters, blob,
+                                             device=device)
+    ledger.write(json.dumps(rec) + "\n"); ledger.close()
+    print(f"[q3 {arm} s{seed}] FINAL ppl={rec['ppl']} "
+          f"validity={rec['validity']} sep={rec.get('register_sep')}",
+          flush=True)
+    torch.save({"arm": arm, "seed": seed,
+                "adapters": {k: v.cpu() for k, v in
+                             adapters.state_dict().items()}},
+               os.path.join(EXP3_DIR, f"q3_{arm}_s{seed}.pt"))
+    del model, adapters
+    torch.cuda.empty_cache()
+
+
+def wikitext_stack_probe(device="cuda"):
+    """The comparison cell: the exp002 WIKITEXT-trained relay_pw stack probed
+    on the same registers (overlap prediction) — loaded onto the BASE model it
+    was trained on, prompts rendered with the instruct template regardless
+    (identical probe inputs across stacks)."""
+    from transformers import AutoModelForCausalLM
+    blob = build_caches()
+    model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B",
+                                                 dtype=torch.float32)
+    model.config.use_cache = False
+    for p in model.parameters():
+        p.requires_grad_(False)
+    d = model.config.hidden_size
+    adapters = nn.ModuleList([RelayPatchwork(d) for _ in model.model.layers])
+    ck = torch.load(os.path.join(DATA_ROOT, "qwen_exp002", "q2_relay_pw_s0.pt"),
+                    map_location="cpu", weights_only=True)
+    adapters.load_state_dict(ck["adapters"])
+    model.model.layers = nn.ModuleList(
+        [_QwenBlockWithAdapter(b, a)
+         for b, a in zip(model.model.layers, adapters)])
+    model = model.to(device).eval()
+    sep = register_probe(model, adapters, blob, device=device)
+    ledger = open(os.path.join(EXP3_DIR, "ledger.jsonl"), "a", encoding="utf-8")
+    ledger.write(json.dumps({"exp": "q3", "arm": "wikitext_stack_probe",
+                             "seed": 0, "register_sep": sep}) + "\n")
+    ledger.close()
+    print(f"[q3 wikitext_stack_probe] sep={sep}", flush=True)
+    del model, adapters
+    torch.cuda.empty_cache()
+
+
+def run_exp003(seeds=(0, 1), device="cuda"):
+    train_arm3("frozen", seed=0, device=device)
+    wikitext_stack_probe(device=device)
+    for seed in seeds:
+        for arm in ("relay_pw_task1", "relay_pw_all", "mlp_all"):
+            train_arm3(arm, seed=seed, device=device)
+    print("=== qwen exp003 COMPLETE ===", flush=True)
+
+
+def smoke():
+    ok = parse_tool_json('x <tool_call>\n{"name":"emit_caption_schema",'
+                         '"arguments":{"subjects":[{"name":"cat","attributes":'
+                         '["black"]}],"actions":["sitting"],"setting":"indoor",'
+                         '"style":null,"mood":null}}\n</tool_call>')
+    assert ok and validate("task_1", ok, "A black cat sitting indoor scene.")
+    assert not validate("task_1", ok, "A dog on grass.")       # ungrounded
+    t2 = {"subjects": [{"name": "[pet]", "attributes": ["[color]"]}],
+          "actions": ["[resting]"], "setting": "[indoor]", "style": None,
+          "mood": None}
+    assert validate("task_2", t2, "")
+    t3 = {"subjects": [{"name": "[ENTITY_1]", "attributes": ["[ATTRIBUTE_1]"]}],
+          "actions": ["[ACTION_1]"], "setting": "[INDOOR]", "style": None,
+          "mood": None}
+    assert validate("task_3", t3, "")
+    assert not validate("task_2", t3, "")                      # cross-task
+    r = RelayPatchwork(896)
+    m = MLPAdapter(896, hidden=128)
+    rp = sum(p.numel() for p in r.parameters())
+    mp = sum(p.numel() for p in m.parameters())
+    assert abs(rp - mp) / rp < 0.01, (rp, mp)                  # budget-matched
+    print(f"qwen exp003 smoke passed (validators OK; relay_pw {rp:,} ~ "
+          f"mlp128 {mp:,}; full run needs GPU + instruct download)")
+
+
+def _in_notebook():
+    try:
+        get_ipython()  # type: ignore[name-defined]  # noqa: F821
+        return True
+    except NameError:
+        return False
+
+
+if __name__ == "__main__":
+    smoke() if not _in_notebook() else (smoke(),
+        print("Notebook: run_exp003() on GPU."))

exp007_math/qwen_exp006_story.py

@@ -0,0 +1,470 @@
+"""qwen_exp006_story.py — geolip-aleph-qwen EXPERIMENT 6: story anchors —
+one relay_pw stack, FOUR methods of requesting a story (Phil's overnight
+directive 2026-07-12: "tinystories with multiple methods of requesting
+stories"). The register-differentiation program moves from extraction
+registers (exp003) and composite emission (exp004) to open-ended GENERATION
+with a per-register measurable constraint.
+
+DATA: roneneldan/TinyStories validation parquet (single 10MB file), stories
+40-180 words, ~4.8k after shuffle, story-hash holdout 10%. From EACH story
+four request registers are derived deterministically:
+  cont — continuation: user gives the opening 2 sentences, assistant
+         completes the story (no tools);
+  inst — instruct: "Tell me a short story about {topic}" (topic = 2 salient
+         content words), assistant tells the full story;
+  kw   — keyword-constrained: "Write a short story using these words:
+         w1, w2, w3" (3 content words hash-sampled from the story) — the
+         constraint is CHECKABLE at judge time;
+  json — tool call: emit_story({title, story}) — the structured register.
+
+ARMS: frozen (zero-shot baseline) | relay_pw_story (ONE stack, all four
+registers interleaved — exp003 showed multi-task resolves capture). 3000
+steps, block 512, batch 4, pure Adam lr 1e-3 wd 0.
+
+JUDGES: per-register held-out ppl; GENERATION validity n=12/register
+(terminates before cap + register-specific checks: cont/inst/kw word counts
+and sentence shape, kw = all 3 keywords present, json = parse + keys +
+story length; novelty check — the continuation must not just parrot its
+prompt); the REGISTER PROBE (sign-code separation across the 4 request
+methods, exp003 gauge). Riders: trunk frozen; GPU-only verdicts; Colab-safe;
+step-50 peak_mem telemetry (the WDDM sysmem-spill law).
+Paste order: geolip_vitals -> ar_differentiation_bed ->
+exp013_augmentation_bed -> qwen_exp001_relay -> qwen_exp002_refine ->
+qwen_exp003_instruct -> this file.
+"""
+from __future__ import annotations
+import hashlib
+import json
+import math
+import os
+import re
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+if "parse_tool_json" not in globals():
+    try:
+        from qwen_exp001_relay import _QwenBlockWithAdapter, _batch
+        from qwen_exp002_refine import RelayPatchwork
+        from qwen_exp003_instruct import parse_tool_json, INSTRUCT_MODEL
+    except ImportError:
+        _here = globals().get("__file__")
+        if _here is None:
+            raise ImportError("paste the qwen stack first")
+        import sys, pathlib
+        sys.path.insert(0, str(pathlib.Path(_here).parent))
+        from qwen_exp001_relay import _QwenBlockWithAdapter, _batch
+        from qwen_exp002_refine import RelayPatchwork
+        from qwen_exp003_instruct import parse_tool_json, INSTRUCT_MODEL
+
+DATA_ROOT = os.environ.get("GEOLIP_DATA", "./data")
+EXP6_DIR = os.path.join(DATA_ROOT, "qwen_exp006")
+BLOCK, BATCH, STEPS, LR = 512, 4, 3000, 1e-3
+MAX_NEW, N_EVAL = 320, 12
+REGS = ("cont", "inst", "kw", "json")
+
+STORY_SRC = ("roneneldan/TinyStories",
+             "data/validation-00000-of-00001-869c898b519ad725.parquet")
+
+TOOL = [{
+    "type": "function",
+    "function": {
+        "name": "emit_story",
+        "description": "Return a short story with a title.",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "title": {"type": "string"},
+                "story": {"type": "string"},
+            },
+            "required": ["title", "story"],
+        },
+    },
+}]
+
+SYS = {
+    "cont": "Continue the story the user started. Keep the same characters "
+            "and simple words. Finish the story.",
+    "inst": "You are a storyteller. Tell one short story in simple words. "
+            "Give it a beginning, a middle, and an end.",
+    "kw": "You are a storyteller. Write one short story in simple words "
+          "that uses every word the user requires.",
+    "json": "Return one short story by calling the emit_story tool with a "
+            "title and the story text.",
+}
+
+_STOP = set("the a an and or but so then was were is are had has have he she "
+            "it they them his her its their there this that with very to of "
+            "in on at for said says say not too you i we one day once upon "
+            "time when all out up down back went go going got get".split())
+
+_SENT_RE = re.compile(r"(?<=[.!?])\s+")
+
+
+def _sentences(story: str):
+    return [s.strip() for s in _SENT_RE.split(story.strip()) if s.strip()]
+
+
+def _content_words(story: str):
+    seen, out = set(), []
+    for w in re.findall(r"[A-Za-z']+", story):
+        lw = w.lower()
+        if len(lw) > 3 and lw not in _STOP and lw not in seen:
+            seen.add(lw)
+            out.append(lw)
+    return out
+
+
+def _h(story: str) -> int:
+    return int(hashlib.md5(story.encode()).hexdigest(), 16)
+
+
+def derive_registers(story: str):
+    """Deterministic 4-register derivation for one story; None if the story
+    can't support a register (too few sentences/content words)."""
+    sents = _sentences(story)
+    words = _content_words(story)
+    if len(sents) < 4 or len(words) < 5:
+        return None
+    h = _h(story)
+    opening = " ".join(sents[:2])
+    rest = " ".join(sents[2:])
+    topic = " and ".join(words[:2])
+    kws = [words[(h >> (8 * i)) % len(words)] for i in range(3)]
+    if len(set(kws)) < 3:                     # hash collision -> first three
+        kws = words[:3]
+    title = " ".join(story.split()[:5]) + "..."
+    return {
+        "cont": {"user": opening, "target": rest},
+        "inst": {"user": f"Tell me a short story about {topic}.",
+                 "target": story},
+        "kw": {"user": "Write a short story using these words: "
+                       f"{kws[0]}, {kws[1]}, {kws[2]}.",
+               "target": story, "kws": kws},
+        "json": {"user": f"Tell me a short story about {topic}.",
+                 "target": {"title": title, "story": story}},
+    }
+
+
+def _render(tok, reg, user, target=None, gen_prompt=False):
+    msgs = [{"role": "system", "content": SYS[reg]},
+            {"role": "user", "content": user}]
+    tools = TOOL if reg == "json" else None
+    if not gen_prompt:
+        if reg == "json":
+            msgs.append({"role": "assistant", "tool_calls": [{
+                "type": "function",
+                "function": {"name": "emit_story", "arguments": target}}]})
+        else:
+            msgs.append({"role": "assistant", "content": target})
+    try:
+        return tok.apply_chat_template(msgs, tools=tools, tokenize=False,
+                                       add_generation_prompt=gen_prompt)
+    except Exception:
+        msgs2 = json.loads(json.dumps(msgs))
+        for m in msgs2:
+            for tc in m.get("tool_calls", []) or []:
+                if isinstance(tc["function"]["arguments"], dict):
+                    tc["function"]["arguments"] = json.dumps(
+                        tc["function"]["arguments"])
+        return tok.apply_chat_template(msgs2, tools=tools, tokenize=False,
+                                       add_generation_prompt=gen_prompt)
+
+
+def build_caches():
+    os.makedirs(EXP6_DIR, exist_ok=True)
+    path = os.path.join(EXP6_DIR, "story_cache.pt")
+    if os.path.exists(path):
+        return torch.load(path, map_location="cpu", weights_only=False)
+    from huggingface_hub import hf_hub_download
+    import pyarrow.parquet as pq
+    from transformers import AutoTokenizer
+    tok = AutoTokenizer.from_pretrained(INSTRUCT_MODEL)
+    p = hf_hub_download(STORY_SRC[0], STORY_SRC[1], repo_type="dataset")
+    texts = pq.read_table(p, columns=["text"]).column("text").to_pylist()
+    stories = [t.strip() for t in texts
+               if t and 40 <= len(t.split()) <= 180]
+    g = torch.Generator().manual_seed(0)
+    stories = [stories[i] for i in
+               torch.randperm(len(stories), generator=g).tolist()][:4800]
+    train_txt = {r: [] for r in REGS}
+    hold_txt = {r: [] for r in REGS}
+    eval_rows = {r: [] for r in REGS}
+    n_used = 0
+    for st in stories:
+        d = derive_registers(st)
+        if d is None:
+            continue
+        n_used += 1
+        holdout = _h(st) % 10 == 0
+        for r in REGS:
+            txt = _render(tok, r, d[r]["user"], d[r]["target"])
+            if holdout:
+                hold_txt[r].append(txt)
+                if len(eval_rows[r]) < 48:
+                    eval_rows[r].append({
+                        "user": d[r]["user"],
+                        "gen_prompt": _render(tok, r, d[r]["user"],
+                                              gen_prompt=True),
+                        "kws": d[r].get("kws"),
+                        "opening": d["cont"]["user"] if r == "cont" else None})
+            else:
+                train_txt[r].append(txt)
+    blob = {
+        "train": {r: tok("\n".join(train_txt[r]),
+                         return_tensors="pt").input_ids[0] for r in REGS},
+        "hold": {r: tok("\n".join(hold_txt[r]),
+                        return_tensors="pt").input_ids[0] for r in REGS},
+        "eval_rows": eval_rows,
+        "stats": {"n_stories": n_used,
+                  "n_train": len(train_txt["cont"]),
+                  "n_hold": len(hold_txt["cont"])},
+    }
+    torch.save(blob, path)
+    print(f"[cache q6] {blob['stats']}", flush=True)
+    return blob
+
+
+# ================================ arms ============================================
+def load_arm6(arm: str, seed: int = 0, device="cuda"):
+    from transformers import AutoModelForCausalLM
+    torch.manual_seed(seed)
+    model = AutoModelForCausalLM.from_pretrained(INSTRUCT_MODEL,
+                                                 dtype=torch.float32)
+    model.config.use_cache = False
+    for p in model.parameters():
+        p.requires_grad_(False)
+    adapters = None
+    if arm != "frozen":
+        d = model.config.hidden_size
+        adapters = nn.ModuleList([RelayPatchwork(d)
+                                  for _ in model.model.layers])
+        model.model.layers = nn.ModuleList(
+            [_QwenBlockWithAdapter(b, a)
+             for b, a in zip(model.model.layers, adapters)])
+    return model.to(device), adapters
+
+
+# ================================ judges ==========================================
+@torch.no_grad()
+def ppl_per_reg(model, blob, device="cuda", n=15):
+    g = torch.Generator().manual_seed(0)
+    model.eval()
+    out = {}
+    for r in REGS:
+        ls = []
+        for _ in range(n):
+            x, y = _batch(blob["hold"][r], BATCH, BLOCK, device, g)
+            logits = model(input_ids=x).logits
+            ls.append(F.cross_entropy(logits.reshape(-1, logits.shape[-1]),
+                                      y.reshape(-1)).item())
+        out[r] = round(math.exp(sum(ls) / len(ls)), 3)
+    return out
+
+
+def _word_count(t: str) -> int:
+    return len(re.findall(r"[A-Za-z']+", t))
+
+
+def check_gen(reg: str, row: dict, txt: str, closed: bool):
+    """Register-specific validity for one generation. Returns (ok, reason)."""
+    if not closed:
+        return False, "no_eos"
+    body = txt.split("<|im_end|>")[0].strip()
+    if reg == "json":
+        args = parse_tool_json(body)
+        if args is None:
+            return False, "json_error"
+        if not isinstance(args.get("title"), str) or \
+                not isinstance(args.get("story"), str):
+            return False, "missing_keys"
+        if _word_count(args["story"]) < 30:
+            return False, "too_short"
+        return True, "ok"
+    if _word_count(body) < (40 if reg == "inst" else 30):
+        return False, "too_short"
+    if len(_sentences(body)) < 2:
+        return False, "not_story_shaped"
+    if reg == "kw":
+        low = body.lower()
+        if not all(k in low for k in row["kws"]):
+            return False, "missing_keyword"
+    if reg == "cont":
+        # novelty: the continuation must not just parrot its opening
+        op_words = set(row["opening"].lower().split())
+        new = [w for w in body.lower().split() if w not in op_words]
+        if len(new) < 15:
+            return False, "parrot"
+    return True, "ok"
+
+
+@torch.no_grad()
+def story_validity(model, blob, device="cuda", n_eval=N_EVAL):
+    from transformers import AutoTokenizer
+    tok = AutoTokenizer.from_pretrained(INSTRUCT_MODEL)
+    model.eval()
+    model.config.use_cache = True
+    out = {}
+    for r in REGS:
+        oks, reasons, samples = 0, {}, []
+        rows = blob["eval_rows"][r][:n_eval]
+        for i, row in enumerate(rows):
+            ids = tok(row["gen_prompt"],
+                      return_tensors="pt").input_ids.to(device)
+            gen = model.generate(ids, max_new_tokens=MAX_NEW, do_sample=False,
+                                 pad_token_id=tok.eos_token_id)
+            txt = tok.decode(gen[0][ids.shape[1]:], skip_special_tokens=False)
+            closed = len(gen[0]) - ids.shape[1] < MAX_NEW or \
+                "<|im_end|>" in txt
+            ok, why = check_gen(r, row, txt, closed)
+            oks += int(ok)
+            reasons[why] = reasons.get(why, 0) + 1
+            if i < 2:
+                samples.append(txt[:300])
+        out[r] = {"valid": round(oks / len(rows), 3), "reasons": reasons,
+                  "samples": samples}
+    model.config.use_cache = False
+    return out
+
+
+@torch.no_grad()
+def register_probe6(model, adapters, blob, device="cuda", n=24):
+    """exp003 gauge on the 4 request registers: sign-code separation per
+    relay layer = mean inter-register Hamming - mean intra-register."""
+    from transformers import AutoTokenizer
+    tok = AutoTokenizer.from_pretrained(INSTRUCT_MODEL)
+    model.eval()
+    layer_ids = [0, 8, 16, 23]
+    codes = {r: {i: [] for i in layer_ids} for r in REGS}
+    for r in REGS:
+        prompts = [row["gen_prompt"] for row in blob["eval_rows"][r][:n]]
+        store = {i: [] for i in layer_ids}
+        hooks = []
+
+        def mk(i, a):
+            def h(mod, inp, out):
+                x = inp[0][:, -1:]
+                slots = a.proj(x).view(*x.shape[:-1], a.n_slots, 4)
+                A = F.normalize(a.addr.codebook, dim=-1)
+                cos = F.normalize(slots, dim=-1) @ A.T
+                win = cos.abs().argmax(-1)
+                sgn = (torch.gather(cos, -1, win.unsqueeze(-1))
+                       .squeeze(-1) > 0).long()
+                store[i].append((win * 2 + sgn).reshape(-1).cpu())
+            return h
+        for i in layer_ids:
+            hooks.append(adapters[i].register_forward_hook(
+                mk(i, adapters[i])))
+        for pr in prompts:
+            ids = tok(pr, return_tensors="pt").input_ids.to(device)
+            model(input_ids=ids)
+        for h in hooks:
+            h.remove()
+        for i in layer_ids:
+            codes[r][i] = torch.stack(store[i])
+    sep = {}
+    for i in layer_ids:
+        def ham(a, b):
+            return (a.unsqueeze(1) != b.unsqueeze(0)).float().mean().item()
+        intra = sum(ham(codes[r][i], codes[r][i]) for r in REGS) / len(REGS)
+        pairs = [(a, b) for ai, a in enumerate(REGS) for b in REGS[ai + 1:]]
+        inter = sum(ham(codes[a][i], codes[b][i]) for a, b in pairs) \
+            / len(pairs)
+        sep[f"L{i}"] = {"inter": round(inter, 4), "intra": round(intra, 4),
+                        "sep": round(inter - intra, 4)}
+    return sep
+
+
+# ================================ runner ==========================================
+def run_arm6(arm: str, seed: int = 0, steps=STEPS, device="cuda"):
+    if not torch.cuda.is_available():
+        raise RuntimeError("verdict runs are GPU-only")
+    os.makedirs(EXP6_DIR, exist_ok=True)
+    blob = build_caches()
+    model, adapters = load_arm6(arm, seed=seed, device=device)
+    ledger = open(os.path.join(EXP6_DIR, "ledger.jsonl"), "a",
+                  encoding="utf-8")
+    if arm != "frozen":
+        g = torch.Generator().manual_seed(seed)
+        opt = torch.optim.Adam(adapters.parameters(), lr=LR, weight_decay=0.0)
+        model.train()
+        for step in range(1, steps + 1):
+            r = REGS[int(torch.randint(len(REGS), (1,), generator=g))]
+            x, y = _batch(blob["train"][r], BATCH, BLOCK, device, g)
+            logits = model(input_ids=x).logits
+            loss = F.cross_entropy(logits.reshape(-1, logits.shape[-1]),
+                                   y.reshape(-1))
+            opt.zero_grad(set_to_none=True); loss.backward(); opt.step()
+            if step == 50 or step % 1000 == 0:
+                mem = torch.cuda.max_memory_allocated() / 2**30 \
+                    if device == "cuda" else 0.0
+                print(f"[q6 {arm} s{seed} step {step}] "
+                      f"loss={loss.item():.3f} peak_mem={mem:.1f}GB",
+                      flush=True)
+    rec = {"exp": "q6", "arm": arm, "seed": seed,
+           "steps": steps if arm != "frozen" else 0,
+           "ppl": ppl_per_reg(model, blob, device=device),
+           "validity": story_validity(model, blob, device=device),
+           "register_sep": register_probe6(model, adapters, blob,
+                                           device=device)
+           if adapters is not None else None,
+           "cache_stats": blob["stats"]}
+    ledger.write(json.dumps(rec) + "\n"); ledger.close()
+    v = {r: rec["validity"][r]["valid"] for r in REGS}
+    print(f"[q6 {arm} s{seed}] FINAL ppl={rec['ppl']} valid={v} "
+          f"sep={rec['register_sep']}", flush=True)
+    if adapters is not None:
+        torch.save({"arm": arm, "seed": seed,
+                    "adapters": {k: t.cpu() for k, t in
+                                 adapters.state_dict().items()}},
+                   os.path.join(EXP6_DIR, f"q6_{arm}_s{seed}.pt"))
+    del model, adapters
+    torch.cuda.empty_cache()
+
+
+def run_exp006(device="cuda"):
+    run_arm6("frozen", seed=0, device=device)
+    run_arm6("relay_pw_story", seed=0, device=device)
+    print("=== qwen exp006 COMPLETE ===", flush=True)
+
+
+def smoke():
+    st = ("Once upon a time there was a little dog named Rex. Rex loved to "
+          "play in the park every day. One day Rex found a shiny red ball "
+          "under a big tree. He picked it up and ran to show his friend "
+          "Anna. Anna smiled and they played with the ball until the sun "
+          "went down. Rex was very happy that day.")
+    d = derive_registers(st)
+    assert d is not None
+    assert len(d["kw"]["kws"]) == 3 and len(set(d["kw"]["kws"])) == 3
+    assert d["cont"]["user"].startswith("Once upon")
+    assert d["json"]["target"]["title"].endswith("...")
+    ok, why = check_gen("kw", {"kws": d["kw"]["kws"]},
+                        st + "<|im_end|>", True)
+    assert ok, why
+    ok, why = check_gen("kw", {"kws": ["zebra", "quantum", "xylophone"]},
+                        st + "<|im_end|>", True)
+    assert not ok and why == "missing_keyword"
+    ok, why = check_gen("cont", {"opening": st}, st + "<|im_end|>", True)
+    assert not ok and why == "parrot"
+    ok, why = check_gen("json", {}, "<tool_call>\n" + json.dumps(
+        {"name": "emit_story", "arguments":
+         {"title": "T...", "story": st}}) + "\n</tool_call><|im_end|>", True)
+    assert ok, why
+    assert check_gen("inst", {}, "Hi.<|im_end|>", True)[1] == "too_short"
+    assert check_gen("inst", {}, st, False)[1] == "no_eos"
+    print("qwen exp006 smoke passed (register derivation + validators "
+          "both directions)")
+
+
+def _in_notebook():
+    try:
+        get_ipython()  # type: ignore[name-defined]  # noqa: F821
+        return True
+    except NameError:
+        return False
+
+
+if __name__ == "__main__":
+    smoke() if not _in_notebook() else (smoke(),
+        print("Notebook: run_exp006() on GPU."))

exp007_math/qwen_exp007_math.py

@@ -0,0 +1,395 @@
+"""qwen_exp007_math.py — geolip-aleph-qwen EXPERIMENT 7: math anchors —
+one relay_pw stack, FOUR methods of asking for a computation (Phil's
+overnight directive 2026-07-12: "math functions with multiple methods of
+asking for functions"). The judge splits FORMAT from CORRECTNESS: does the
+anchor paradigm carry COMPUTATION on held-out problems, or only the shape
+of answers? Fully synthetic — no downloads; ground truth computable.
+
+REGISTERS (each problem rendered one way, deterministically assigned):
+  nl   — "What is 37 + 45?"            -> "37 + 45 = 82"
+  json — emit_answer tool call          -> {"expression": "37 + 45",
+                                            "result": 82}
+  code — "def f(x): return 3*x + 2 ... What is f(4)?" -> "f(4) = 14"
+  word — templated word problem         -> "Sam has 82 apples."
+
+PROBLEM SPACE: a+b (1-99), a-b (result>=0), a*b (2-12), f(x)=a*x+b with
+a 2-9, b 0-20, x 2-9. 4,000 problems, problem-hash holdout 10% — held-out
+CORRECTNESS is unseen-instance computation, not recall.
+
+ARMS: frozen | relay_pw_math (all four registers interleaved). 3000 steps,
+block 512, batch 4, pure Adam lr 1e-3 wd 0.
+
+JUDGES: per-register held-out ppl; per-register FORMAT validity and
+CORRECTNESS (n=16/register, both arms — the frozen baseline separates
+"the trunk can already add" from what the anchor adds); the register probe
+(sign-code separation across the 4 ask-methods). Riders: trunk frozen;
+GPU-only verdicts; Colab-safe; step-50 peak_mem telemetry.
+Paste order: geolip_vitals -> ar_differentiation_bed ->
+exp013_augmentation_bed -> qwen_exp001_relay -> qwen_exp002_refine ->
+qwen_exp003_instruct -> qwen_exp006_story -> this file.
+"""
+from __future__ import annotations
+import hashlib
+import json
+import math
+import os
+import re
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+if "parse_tool_json" not in globals():
+    try:
+        from qwen_exp001_relay import _QwenBlockWithAdapter, _batch
+        from qwen_exp002_refine import RelayPatchwork
+        from qwen_exp003_instruct import parse_tool_json, INSTRUCT_MODEL
+        from qwen_exp006_story import load_arm6, register_probe6
+    except ImportError:
+        _here = globals().get("__file__")
+        if _here is None:
+            raise ImportError("paste the qwen stack first")
+        import sys, pathlib
+        sys.path.insert(0, str(pathlib.Path(_here).parent))
+        from qwen_exp001_relay import _QwenBlockWithAdapter, _batch
+        from qwen_exp002_refine import RelayPatchwork
+        from qwen_exp003_instruct import parse_tool_json, INSTRUCT_MODEL
+        from qwen_exp006_story import load_arm6, register_probe6
+
+DATA_ROOT = os.environ.get("GEOLIP_DATA", "./data")
+EXP7_DIR = os.path.join(DATA_ROOT, "qwen_exp007")
+BLOCK, BATCH, STEPS, LR = 512, 4, 3000, 1e-3
+MAX_NEW, N_EVAL = 96, 16
+REGS = ("nl", "json", "code", "word")
+
+TOOL = [{
+    "type": "function",
+    "function": {
+        "name": "emit_answer",
+        "description": "Return the result of a computation.",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "expression": {"type": "string"},
+                "result": {"type": "integer"},
+            },
+            "required": ["expression", "result"],
+        },
+    },
+}]
+
+SYS = {
+    "nl": "Answer the arithmetic question. Reply with the equation and its "
+          "result, like: 12 + 5 = 17",
+    "json": "Compute the expression and return it by calling the "
+            "emit_answer tool.",
+    "code": "The user defines a Python function and asks for its value. "
+            "Reply like: f(3) = 11",
+    "word": "Solve the word problem. Reply with one short sentence "
+            "containing the answer.",
+}
+
+_NAMES = ("Sam", "Anna", "Ben", "Mia", "Tom", "Lily", "Max", "Zoe")
+_THINGS = ("apples", "pens", "books", "coins", "shells", "cards",
+           "marbles", "stickers")
+
+
+def _h(s: str) -> int:
+    return int(hashlib.md5(s.encode()).hexdigest(), 16)
+
+
+def gen_problems(n=4000, seed=0):
+    """Deterministic synthetic problems: (register, user, target, truth)."""
+    g = torch.Generator().manual_seed(seed)
+
+    def ri(lo, hi):
+        return int(torch.randint(lo, hi + 1, (1,), generator=g))
+
+    out, seen = [], set()
+    while len(out) < n:
+        kind = ri(0, 3)
+        if kind == 0:                                          # a + b
+            a, b = ri(1, 99), ri(1, 99)
+            expr, truth = f"{a} + {b}", a + b
+        elif kind == 1:                                        # a - b >= 0
+            a, b = ri(1, 99), ri(1, 99)
+            a, b = max(a, b), min(a, b)
+            expr, truth = f"{a} - {b}", a - b
+        elif kind == 2:                                        # a * b
+            a, b = ri(2, 12), ri(2, 12)
+            expr, truth = f"{a} * {b}", a * b
+        else:                                                  # f(x)=a*x+b
+            a, b, x = ri(2, 9), ri(0, 20), ri(2, 9)
+            expr, truth = f"f{a}x{b}@{x}", a * x + b
+        reg = REGS[len(out) % 4] if kind != 3 else "code"
+        if kind == 3:
+            user = (f"def f(x):\n    return {a} * x + {b}\n\n"
+                    f"What is f({x})?")
+            target = f"f({x}) = {truth}"
+        elif reg == "nl":
+            user = f"What is {expr}?"
+            target = f"{expr} = {truth}"
+        elif reg == "json":
+            user = f"Compute {expr}."
+            target = {"expression": expr, "result": truth}
+        elif reg == "code":                    # non-f(x) problem in code reg
+            user = (f"def f(x):\n    return x {expr.split(' ', 1)[1]}\n\n"
+                    f"What is f({expr.split(' ')[0]})?")
+            target = f"f({expr.split(' ')[0]}) = {truth}"
+        else:                                                  # word
+            nm = _NAMES[_h(expr) % len(_NAMES)]
+            th = _THINGS[_h(expr + "t") % len(_THINGS)]
+            a_, op, b_ = expr.split(" ")
+            if op == "+":
+                user = (f"{nm} has {a_} {th}. {nm} gets {b_} more. "
+                        f"How many {th} does {nm} have now?")
+            elif op == "-":
+                user = (f"{nm} has {a_} {th}. {nm} gives away {b_}. "
+                        f"How many {th} does {nm} have left?")
+            else:
+                user = (f"Each box holds {b_} {th}. There are {a_} boxes. "
+                        f"How many {th} in all?")
+            target = f"{nm} has {truth} {th}." if op != "*" else \
+                f"There are {truth} {th}."
+        key = (reg, user)
+        if key in seen:
+            continue
+        seen.add(key)
+        out.append({"reg": reg, "user": user, "target": target,
+                    "truth": truth})
+    return out
+
+
+def _render(tok, reg, user, target=None, gen_prompt=False):
+    msgs = [{"role": "system", "content": SYS[reg]},
+            {"role": "user", "content": user}]
+    tools = TOOL if reg == "json" else None
+    if not gen_prompt:
+        if reg == "json":
+            msgs.append({"role": "assistant", "tool_calls": [{
+                "type": "function",
+                "function": {"name": "emit_answer", "arguments": target}}]})
+        else:
+            msgs.append({"role": "assistant", "content": target})
+    try:
+        return tok.apply_chat_template(msgs, tools=tools, tokenize=False,
+                                       add_generation_prompt=gen_prompt)
+    except Exception:
+        msgs2 = json.loads(json.dumps(msgs))
+        for m in msgs2:
+            for tc in m.get("tool_calls", []) or []:
+                if isinstance(tc["function"]["arguments"], dict):
+                    tc["function"]["arguments"] = json.dumps(
+                        tc["function"]["arguments"])
+        return tok.apply_chat_template(msgs2, tools=tools, tokenize=False,
+                                       add_generation_prompt=gen_prompt)
+
+
+def build_caches7():
+    os.makedirs(EXP7_DIR, exist_ok=True)
+    path = os.path.join(EXP7_DIR, "math_cache.pt")
+    if os.path.exists(path):
+        return torch.load(path, map_location="cpu", weights_only=False)
+    from transformers import AutoTokenizer
+    tok = AutoTokenizer.from_pretrained(INSTRUCT_MODEL)
+    probs = gen_problems()
+    train_txt = {r: [] for r in REGS}
+    hold_txt = {r: [] for r in REGS}
+    eval_rows = {r: [] for r in REGS}
+    for p in probs:
+        r = p["reg"]
+        txt = _render(tok, r, p["user"], p["target"])
+        if _h(p["user"]) % 10 == 0:
+            hold_txt[r].append(txt)
+            if len(eval_rows[r]) < 48:
+                eval_rows[r].append({
+                    "user": p["user"], "truth": p["truth"],
+                    "gen_prompt": _render(tok, r, p["user"],
+                                          gen_prompt=True)})
+        else:
+            train_txt[r].append(txt)
+    blob = {
+        "train": {r: tok("\n".join(train_txt[r]),
+                         return_tensors="pt").input_ids[0] for r in REGS},
+        "hold": {r: tok("\n".join(hold_txt[r]),
+                        return_tensors="pt").input_ids[0] for r in REGS},
+        "eval_rows": eval_rows,
+        "stats": {r: len(train_txt[r]) for r in REGS},
+    }
+    torch.save(blob, path)
+    print(f"[cache q7] {blob['stats']}", flush=True)
+    return blob
+
+
+# ================================ judges ==========================================
+_NUM_RE = re.compile(r"-?\d+")
+
+
+def check_math(reg: str, row: dict, txt: str, closed: bool):
+    """Returns (format_ok, correct, reason)."""
+    if not closed:
+        return False, False, "no_eos"
+    body = txt.split("<|im_end|>")[0].strip()
+    if reg == "json":
+        args = parse_tool_json(body)
+        if args is None:
+            return False, False, "json_error"
+        if "expression" not in args or "result" not in args:
+            return False, False, "missing_keys"
+        try:
+            return True, int(args["result"]) == row["truth"], "ok"
+        except (TypeError, ValueError):
+            return True, False, "non_int_result"
+    nums = _NUM_RE.findall(body)
+    if not nums:
+        return False, False, "no_number"
+    fmt = ("=" in body) if reg in ("nl", "code") else \
+        (len(body.split()) <= 24)
+    return fmt, int(nums[-1]) == row["truth"], "ok"
+
+
+@torch.no_grad()
+def math_validity(model, blob, device="cuda", n_eval=N_EVAL):
+    from transformers import AutoTokenizer
+    tok = AutoTokenizer.from_pretrained(INSTRUCT_MODEL)
+    model.eval()
+    model.config.use_cache = True
+    out = {}
+    for r in REGS:
+        fmts, cors, samples = 0, 0, []
+        rows = blob["eval_rows"][r][:n_eval]
+        for i, row in enumerate(rows):
+            ids = tok(row["gen_prompt"],
+                      return_tensors="pt").input_ids.to(device)
+            gen = model.generate(ids, max_new_tokens=MAX_NEW, do_sample=False,
+                                 pad_token_id=tok.eos_token_id)
+            txt = tok.decode(gen[0][ids.shape[1]:], skip_special_tokens=False)
+            closed = len(gen[0]) - ids.shape[1] < MAX_NEW or \
+                "<|im_end|>" in txt
+            fmt, cor, _ = check_math(r, row, txt, closed)
+            fmts += int(fmt)
+            cors += int(cor)
+            if i < 2:
+                samples.append(txt[:160])
+        out[r] = {"format": round(fmts / len(rows), 3),
+                  "correct": round(cors / len(rows), 3),
+                  "samples": samples}
+    model.config.use_cache = False
+    return out
+
+
+@torch.no_grad()
+def ppl_per_reg7(model, blob, device="cuda", n=15):
+    g = torch.Generator().manual_seed(0)
+    model.eval()
+    out = {}
+    for r in REGS:
+        ls = []
+        for _ in range(n):
+            x, y = _batch(blob["hold"][r], BATCH, BLOCK, device, g)
+            logits = model(input_ids=x).logits
+            ls.append(F.cross_entropy(logits.reshape(-1, logits.shape[-1]),
+                                      y.reshape(-1)).item())
+        out[r] = round(math.exp(sum(ls) / len(ls)), 3)
+    return out
+
+
+# ================================ runner ==========================================
+def run_arm7(arm: str, seed: int = 0, steps=STEPS, device="cuda"):
+    if not torch.cuda.is_available():
+        raise RuntimeError("verdict runs are GPU-only")
+    os.makedirs(EXP7_DIR, exist_ok=True)
+    blob = build_caches7()
+    model, adapters = load_arm6(arm, seed=seed, device=device)
+    ledger = open(os.path.join(EXP7_DIR, "ledger.jsonl"), "a",
+                  encoding="utf-8")
+    if arm != "frozen":
+        g = torch.Generator().manual_seed(seed)
+        opt = torch.optim.Adam(adapters.parameters(), lr=LR, weight_decay=0.0)
+        model.train()
+        for step in range(1, steps + 1):
+            r = REGS[int(torch.randint(len(REGS), (1,), generator=g))]
+            x, y = _batch(blob["train"][r], BATCH, BLOCK, device, g)
+            logits = model(input_ids=x).logits
+            loss = F.cross_entropy(logits.reshape(-1, logits.shape[-1]),
+                                   y.reshape(-1))
+            opt.zero_grad(set_to_none=True); loss.backward(); opt.step()
+            if step == 50 or step % 1000 == 0:
+                mem = torch.cuda.max_memory_allocated() / 2**30 \
+                    if device == "cuda" else 0.0
+                print(f"[q7 {arm} s{seed} step {step}] "
+                      f"loss={loss.item():.3f} peak_mem={mem:.1f}GB",
+                      flush=True)
+    # the probe reuses exp006's gauge (identical adapter anatomy)
+    import qwen_exp006_story as q6mod
+    q6_regs_save = q6mod.REGS
+    q6mod.REGS = REGS
+    try:
+        sep = register_probe6(model, adapters, blob, device=device) \
+            if adapters is not None else None
+    finally:
+        q6mod.REGS = q6_regs_save
+    rec = {"exp": "q7", "arm": arm, "seed": seed,
+           "steps": steps if arm != "frozen" else 0,
+           "ppl": ppl_per_reg7(model, blob, device=device),
+           "validity": math_validity(model, blob, device=device),
+           "register_sep": sep,
+           "cache_stats": blob["stats"]}
+    ledger.write(json.dumps(rec) + "\n"); ledger.close()
+    v = {r: (rec["validity"][r]["format"], rec["validity"][r]["correct"])
+         for r in REGS}
+    print(f"[q7 {arm} s{seed}] FINAL ppl={rec['ppl']} fmt/cor={v} "
+          f"sep={rec['register_sep']}", flush=True)
+    if adapters is not None:
+        torch.save({"arm": arm, "seed": seed,
+                    "adapters": {k: t.cpu() for k, t in
+                                 adapters.state_dict().items()}},
+                   os.path.join(EXP7_DIR, f"q7_{arm}_s{seed}.pt"))
+    del model, adapters
+    torch.cuda.empty_cache()
+
+
+def run_exp007(device="cuda"):
+    run_arm7("frozen", seed=0, device=device)
+    run_arm7("relay_pw_math", seed=0, device=device)
+    print("=== qwen exp007 COMPLETE ===", flush=True)
+
+
+def smoke():
+    probs = gen_problems(n=400)
+    assert len(probs) == 400
+    regs = {p["reg"] for p in probs}
+    assert regs == set(REGS), regs
+    for p in probs[:50]:
+        if p["reg"] == "json":
+            assert p["target"]["result"] == p["truth"]
+        elif p["reg"] in ("nl", "code"):
+            assert str(p["truth"]) in p["target"]
+    f, c, _ = check_math("nl", {"truth": 82}, "37 + 45 = 82<|im_end|>", True)
+    assert f and c
+    f, c, _ = check_math("nl", {"truth": 82}, "37 + 45 = 83<|im_end|>", True)
+    assert f and not c
+    f, c, _ = check_math("json", {"truth": 14}, "<tool_call>\n" + json.dumps(
+        {"name": "emit_answer",
+         "arguments": {"expression": "3*4+2", "result": 14}}) +
+        "\n</tool_call><|im_end|>", True)
+    assert f and c
+    f, c, _ = check_math("word", {"truth": 42},
+                         "There are 42 pens.<|im_end|>", True)
+    assert f and c
+    f, c, why = check_math("nl", {"truth": 5}, "37 + 45 = 82", False)
+    assert not f and why == "no_eos"
+    print("qwen exp007 smoke passed (problem gen + format/correctness "
+          "judges both directions)")
+
+
+def _in_notebook():
+    try:
+        get_ipython()  # type: ignore[name-defined]  # noqa: F821
+        return True
+    except NameError:
+        return False
+
+
+if __name__ == "__main__":
+    smoke() if not _in_notebook() else (smoke(),
+        print("Notebook: run_exp007() on GPU."))

exp007_math/repro.py

@@ -0,0 +1,23 @@
+"""repro.py — standalone loader/runner for exp007. Code dependencies live in
+THIS folder. python repro.py = CPU smoke; python repro.py --run = full
+campaign (frozen baseline + trained arm; GPU ~1.2h). Data + caches land in
+./data (override with GEOLIP_DATA)."""
+import os
+import sys
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, HERE)
+
+if __name__ == "__main__":
+    import geolip_vitals                       # noqa: F401  (paste order)
+    import ar_differentiation_bed              # noqa: F401
+    import exp013_augmentation_bed             # noqa: F401
+    import qwen_exp001_relay                   # noqa: F401
+    import qwen_exp002_refine                  # noqa: F401
+    import qwen_exp003_instruct                # noqa: F401
+    import qwen_exp007_math as m
+    if "--run" in sys.argv[1:]:
+        m.run_exp007()
+    else:
+        m.smoke()
+        print("repro smoke passed — --run for the campaign (GPU)")

exp007_math/results/ledger.jsonl

@@ -0,0 +1,2 @@
+{"exp": "q7", "arm": "frozen", "seed": 0, "steps": 0, "ppl": {"nl": 2.763, "json": 4.393, "code": 2.591, "word": 3.145}, "validity": {"nl": {"format": 0.312, "correct": 1.0, "samples": ["45<|im_end|>", "The equation for 94 - 27 is 67.<|im_end|>"]}, "json": {"format": 1.0, "correct": 1.0, "samples": ["<tool_call>\n{\"name\": \"emit_answer\", \"arguments\": {\"expression\": \"2 * 12\", \"result\": 24}}\n</tool_call><|im_end|>", "<tool_call>\n{\"name\": \"emit_answer\", \"arguments\": {\"expression\": \"84 + 73\", \"result\": 157}}\n</tool_call><|im_end|>"]}, "code": {"format": 1.0, "correct": 0.688, "samples": ["f(77) = 60<|im_end|>", "f(5) = 15<|im_end|>"]}, "word": {"format": 1.0, "correct": 0.938, "samples": ["Max now has 20 marbles.<|im_end|>", "Zoe has 41 shells left.<|im_end|>"]}}, "register_sep": null, "cache_stats": {"nl": 696, "json": 702, "code": 1496, "word": 718}}
+{"exp": "q7", "arm": "relay_pw_math", "seed": 0, "steps": 3000, "ppl": {"nl": 1.342, "json": 1.061, "code": 1.168, "word": 1.525}, "validity": {"nl": {"format": 1.0, "correct": 1.0, "samples": ["5 * 9 = 45<|im_end|>", "94 - 27 = 67<|im_end|>"]}, "json": {"format": 1.0, "correct": 1.0, "samples": ["<tool_call>\n{\"name\": \"emit_answer\", \"arguments\": {\"expression\": \"2 * 12\", \"result\": 24}}\n</tool_call><|im_end|>", "<tool_call>\n{\"name\": \"emit_answer\", \"arguments\": {\"expression\": \"84 + 73\", \"result\": 157}}\n</tool_call><|im_end|>"]}, "code": {"format": 1.0, "correct": 0.938, "samples": ["f(77) = 6<|im_end|>", "f(5) = 15<|im_end|>"]}, "word": {"format": 1.0, "correct": 1.0, "samples": ["Max has 72 marbles.<|im_end|>", "Zoe has 41 shells.<|im_end|>"]}}, "register_sep": {"L0": {"inter": 0.2563, "intra": 0.0418, "sep": 0.2145}, "L8": {"inter": 0.3293, "intra": 0.0485, "sep": 0.2808}, "L16": {"inter": 0.8737, "intra": 0.4026, "sep": 0.4711}, "L23": {"inter": 0.9304, "intra": 0.4347, "sep": 0.4957}}, "cache_stats": {"nl": 696, "json": 702, "code": 1496, "word": 718}}

exp007_math/results/results.json

@@ -0,0 +1,116 @@
+{
+ "frozen": {
+  "ppl": {
+   "nl": 2.763,
+   "json": 4.393,
+   "code": 2.591,
+   "word": 3.145
+  },
+  "validity": {
+   "nl": {
+    "format": 0.312,
+    "correct": 1.0,
+    "samples": [
+     "45<|im_end|>",
+     "The equation for 94 - 27 is 67.<|im_end|>"
+    ]
+   },
+   "json": {
+    "format": 1.0,
+    "correct": 1.0,
+    "samples": [
+     "<tool_call>\n{\"name\": \"emit_answer\", \"arguments\": {\"expression\": \"2 * 12\", \"result\": 24}}\n</tool_call><|im_end|>",
+     "<tool_call>\n{\"name\": \"emit_answer\", \"arguments\": {\"expression\": \"84 + 73\", \"result\": 157}}\n</tool_call><|im_end|>"
+    ]
+   },
+   "code": {
+    "format": 1.0,
+    "correct": 0.688,
+    "samples": [
+     "f(77) = 60<|im_end|>",
+     "f(5) = 15<|im_end|>"
+    ]
+   },
+   "word": {
+    "format": 1.0,
+    "correct": 0.938,
+    "samples": [
+     "Max now has 20 marbles.<|im_end|>",
+     "Zoe has 41 shells left.<|im_end|>"
+    ]
+   }
+  }
+ },
+ "relay_pw_math": {
+  "steps": 3000,
+  "ppl": {
+   "nl": 1.342,
+   "json": 1.061,
+   "code": 1.168,
+   "word": 1.525
+  },
+  "validity": {
+   "nl": {
+    "format": 1.0,
+    "correct": 1.0,
+    "samples": [
+     "5 * 9 = 45<|im_end|>",
+     "94 - 27 = 67<|im_end|>"
+    ]
+   },
+   "json": {
+    "format": 1.0,
+    "correct": 1.0,
+    "samples": [
+     "<tool_call>\n{\"name\": \"emit_answer\", \"arguments\": {\"expression\": \"2 * 12\", \"result\": 24}}\n</tool_call><|im_end|>",
+     "<tool_call>\n{\"name\": \"emit_answer\", \"arguments\": {\"expression\": \"84 + 73\", \"result\": 157}}\n</tool_call><|im_end|>"
+    ]
+   },
+   "code": {
+    "format": 1.0,
+    "correct": 0.938,
+    "samples": [
+     "f(77) = 6<|im_end|>",
+     "f(5) = 15<|im_end|>"
+    ]
+   },
+   "word": {
+    "format": 1.0,
+    "correct": 1.0,
+    "samples": [
+     "Max has 72 marbles.<|im_end|>",
+     "Zoe has 41 shells.<|im_end|>"
+    ]
+   }
+  },
+  "register_sep": {
+   "L0": {
+    "inter": 0.2563,
+    "intra": 0.0418,
+    "sep": 0.2145
+   },
+   "L8": {
+    "inter": 0.3293,
+    "intra": 0.0485,
+    "sep": 0.2808
+   },
+   "L16": {
+    "inter": 0.8737,
+    "intra": 0.4026,
+    "sep": 0.4711
+   },
+   "L23": {
+    "inter": 0.9304,
+    "intra": 0.4347,
+    "sep": 0.4957
+   }
+  }
+ },
+ "cache_stats": {
+  "nl": 696,
+  "json": 702,
+  "code": 1496,
+  "word": 718
+ },
+ "n_rows": 2
+}