Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,183 @@
+---
+license: mit
+base_model: EleutherAI/pythia-160m
+tags:
+  - sparse-autoencoder
+  - interpretability
+  - deception
+  - sandbagging
+  - sae-lens
+library_name: sae_lens
+---
+
+# V4 Sandbagging SAEs — Pythia-160M
+
+Sparse autoencoders trained on residual-stream activations from
+**EleutherAI/pythia-160m** while the model answered trivia questions under two
+instructions: answer correctly (**genuine**) or answer incorrectly on
+purpose (**sandbag**). Part of the follow-up research program to "The
+Secret Agenda" (arXiv:2509.20393).
+
+## What this repo contains
+
+**6 sparse autoencoders**: 0 JumpReLU (post-STE-fix, arXiv:2407.14435) + 6 TopK (k=64).
+
+Layers covered: L3, L6, L9. Three training conditions per layer per
+architecture:
+
+- **genuine_only** — trained on neutral-prompt activations only
+- **sandbagging_only** — trained on sandbagging-prompt activations only
+- **mixed** — trained on all available activations
+
+For each SAE, the repo includes:
+
+- `{tag}/sae_weights.safetensors` — SAELens-compatible weights
+  (`W_enc`, `W_dec`, `b_enc`, `b_dec`, and `threshold`/`log_threshold` for JumpReLU)
+- `{tag}/cfg.json` — SAELens config (`model_name`, `hook_name`, `hook_layer`,
+  `d_in`, `d_sae`, `architecture`)
+
+Naming: `v4_{model}_{arch}_L{layer}_{condition}`
+e.g. `v4_pythia160_jumprelu_L6_genuine_only`,
+`v4_pythia160_topk_L6_mixed`.
+
+## Architecture and training
+
+All SAEs share:
+
+- `d_in = 768` (residual stream of Pythia-160M)
+- `d_sae = 3072 = 4 × d_in` (expansion factor 4)
+- 300 training epochs, batch 128, lr 3e-4, L1 coefficient 1e-3
+- Decoder normalized per SAELens convention
+- JumpReLU uses the Gaussian-kernel STE from Rajamanoharan et al. 2024
+  (arXiv:2407.14435). Pre-fix SAEs had `threshold=0` and were functionally
+  ReLU; all SAEs in this repo are post-fix.
+- TopK uses hard L0 = 64. Fixed-L0 gives tighter sparsity than JumpReLU;
+  also much faster training (~40s/SAE vs ~500s for JumpReLU).
+
+Training activations source: `llama_sandbagging_dataset.pt` /
+`smollm_sandbagging_dataset.pt` / `tinyllama_sandbagging_dataset.pt` in
+[the research repo](https://github.com/SolshineCode/deception-nanochat-sae-research/tree/main/experiments/v4_sandbagging_concealment/results).
+
+## Key scientific findings
+
+These SAEs were used to validate (and in several cases, **falsify**)
+SAE-mediated behavioral interpretability claims. Summary of what we
+learned using these SAEs (full detail: `RESULTS_INDEX.md` §17-§32 in the
+research repo):
+
+### SAE-level interventions are non-specific
+
+Under adversarial controls E1 (shuffled-label median) and E4
+(reversed-direction / sandbagging median), the E6 full-SAE-swap
+intervention collapses:
+
+- **Specificity gap < 0.15pp across 9 tested model-layer combinations**
+  (all JumpReLU SAEs tested: Llama L4/L8/L12, SmolLM2 L7/L15/L22,
+  TinyLlama L5/L11/L16; §24, §26, §31).
+- The E6 effect (positive or negative) is reproduced by shuffled-label
+  and reversed-direction medians. The SAE intervention is non-specific
+  activation disruption, not feature-specific restoration.
+
+### Circuit-level interventions are head-specific (Llama only)
+
+Ablation of 5 identified suppressor heads in Llama-3.2-1B:
+
+- **+13.71pp restoration** (CI95 [+11.28, +16.24] on n=50 questions; §28).
+- Specificity gap **+14.85pp vs random 5-head ablation** (σ 3.71pp over
+  n_draws=20; §28.4).
+- Cross-model check (§30): only Llama-3.2-1B shows a substantial
+  **sandbagging-specific** portion (+7.72pp = 56% of the circuit effect).
+  TinyLlama-intermediate, Pythia-160M, Pythia-410M all show ~0 or
+  negative sandbagging-specific portions — the Llama heads dampen
+  correctness condition-generally, not only under sandbagging
+  instructions.
+
+### Cross-model circuit finding
+
+Our cross-model suppressor-head search (§13-§14) identified weaker effects
+in TinyLlama (+1.95pp), Pythia-160M (+0.67pp), and Pythia-410M (+4.54pp)
+on the canonical 5-question evaluation. Only Llama-3.2-1B shows circuit-
+level ESH at meaningful magnitude. See the
+[Llama repo](https://huggingface.co/Solshine/deception-v4-saes-llama-3-2-1b)
+for the full circuit writeup.
+
+
+### TopK vs JumpReLU on behavioral separation
+
+Paired Cohen's d comparison on sandbagging-vs-genuine at matched
+(layer, condition) (§32):
+
+- JumpReLU wins d_max in 22/27 pairs (81%).
+- TopK wins in 4 cases, all at mid-to-late layers with **mixed** training.
+- **Standout: Llama L8 mixed TopK d_max = 9.76, vs 4.69 for the
+  matched JumpReLU** — the sharpest single-feature separator observed
+  across all V4 SAEs. Candidate target for interpretability
+  investigation.
+- JumpReLU's d_mean is ~20-50x higher (broader separability across many
+  features); TopK concentrates signal in fewer features.
+
+Use JumpReLU for probing/detection. Use TopK for single-feature
+interpretability targets.
+
+## How to load
+
+With SAELens:
+
+```python
+from sae_lens import SAE
+
+sae, cfg_dict, sparsity = SAE.from_pretrained(
+    release="Solshine/deception-v4-saes-pythia-160m",
+    sae_id="v4_pythia160_jumprelu_L6_genuine_only",
+)
+```
+
+Raw (no SAELens):
+
+```python
+from safetensors.torch import load_file
+from huggingface_hub import hf_hub_download
+import json
+
+tag = "v4_pythia160_topk_L6_mixed"
+weights = load_file(hf_hub_download("Solshine/deception-v4-saes-pythia-160m", f"{tag}/sae_weights.safetensors"))
+cfg = json.loads(open(hf_hub_download("Solshine/deception-v4-saes-pythia-160m", f"{tag}/cfg.json")).read())
+```
+
+Complete list of available tags in this repo:
+
+- `v4_pythia160_topk_L3_genuine_only`
+- `v4_pythia160_topk_L3_mixed`
+- `v4_pythia160_topk_L6_genuine_only`
+- `v4_pythia160_topk_L6_mixed`
+- `v4_pythia160_topk_L9_genuine_only`
+- `v4_pythia160_topk_L9_mixed`
+
+## Citation
+
+```bibtex
+@misc{v4_sandbagging_saes_2026,
+  author = {DeLeeuw, Caleb},
+  title = {V4 Sandbagging SAEs — Pythia-160M},
+  year = {2026},
+  howpublished = {HuggingFace},
+  url = {https://huggingface.co/Solshine/deception-v4-saes-pythia-160m}
+}
+
+@misc{secret_agenda_2025,
+  title={The Secret Agenda: LLMs Strategically Lie Undetected by Current Safety Tools},
+  author={DeLeeuw, Caleb and others},
+  year={2025},
+  howpublished={arXiv:2509.20393},
+  url={https://arxiv.org/abs/2509.20393}
+}
+```
+
+## Source
+
+Research code, full RESULTS_INDEX, and the `papers/specificity_gap/` draft
+manuscript: <https://github.com/SolshineCode/deception-nanochat-sae-research>
+
+## License
+
+MIT. See the source repository for full terms.