Amber knowledge-pack LoRA adapter (mlx_lm, lr 1e-4, 2000 it) + verbatim before/after per-item eval reports

017771e verified 27 days ago

3.28 kB

license: mit
base_model: mlx-community/SmolLM-135M-Instruct-4bit
library_name: mlx
pipeline_tag: text-generation
language:
  - en
datasets:
  - crimson-knight/amber-framework-knowledge-pack
tags:
  - lora
  - mlx
  - knowledge-pack
  - fine-tuning
  - amber
  - crystal

SmolLM-135M-Instruct-4bit + Amber knowledge pack (LoRA adapter)

A tiny demo adapter, published as evidence, not as a product. It is the exact LoRA adapter behind the measured before/after numbers in AgentC-Consulting/knowledge-packs (live-run/RESULTS.md), trained on the Amber Framework knowledge pack dataset.

Base model: mlx-community/SmolLM-135M-Instruct-4bit (135M params, 4-bit, MLX)
Method: LoRA via mlx_lm (rank 8, 8 layers, lr 1e-4, batch 2, 2000 iterations, 62.8 s wall on an M1 Max 32 GB) — full config in adapter_config.json
Data: 471 generated pairs about the Amber web framework (Crystal), deterministic template synthesis from the Amber docs
sha256 of adapters.safetensors: 8f88465b98309f4001d85b53eae959e4f22dae3155fc1e43a60f5b90deba6c76 — matches the hash recorded in the repo's RESULTS.md, so you can verify this is the artifact the numbers were measured on. No --seed was passed during training (mlx-lm default), so bit-exact retraining is not guaranteed; the hash identifies this exact artifact.

Measured results (this adapter, temp 0; counts, not marketing)

Eval set	Metric	Base	This adapter
memorization (trained-fact probe)	recall, normalized substring (20)	1/20	11/20
	multiple choice (30)	2/30	6/30
heldout_sources (control)	recall (20)	2/20	0/20
	multiple choice (30)	7/30	7/30

The per-item reports the table is derived from are included verbatim in this repo: reports/base.eval.json and reports/adapter.eval.json (question, gold answer, full completion, verdict for every one of the 100 items).

Honest read: fact injection is real and measured (recall of trained facts 1/20 → 11/20 under held-out question templates), and the held-out-source control stayed flat — this adapter learned the trained facts, not "the Amber framework." MC accuracy after training (6/30) is at or below the 25% guessing chance; the signal is the recall delta. The eval questions are synthetic and not hand-reviewed. Treat everything here as lab notes, or better: run the harness yourself.

Use it

pip install "mlx-lm" "transformers<5"
python -m mlx_lm generate --model mlx-community/SmolLM-135M-Instruct-4bit \
  --adapter-path . \
  --prompt "In Amber, what does the pipeline block in config/routes.cr do?" \
  --temp 0

Do not expect a reliable framework expert — this is a 135M-parameter 4-bit model that memorized 165 facts. The point is the measurement loop: the long-form story is at https://agentc-consulting.github.io/knowledge-packs/.

Built by the team at AgentC Consulting — https://agentc.consulting?ref=kp-adapter