Subject-Emu-5259/NeuralAI-Nae1

🧠 NeuralAI — Nae1

Trained from scratch. Every weight owned.
A 100M-parameter decoder pretrained end-to-end on FineWeb — no distillation, no fine-tune of someone else's base.

---

🚀 Quick facts (current release: step 80000, v2 lineage)

Property	Value
Architecture	Llama-style decoder — RoPE, GQA, SwiGLU
Parameters	100,111,872 (100.1M)
Layers / hidden / heads	12 layers · 768 hidden · 12 heads · 4 KV heads (GQA)
Vocabulary	32,000 tokens (ByteLevel BPE v2, tokenizer_v2/)
Context	2,048 trained · evaluated on n_ctx=512
Training data	FineWeb CC-MAIN-2013-20 (v2 run; prior v1 run: 768M tokens, 8 shards)
Training schedule	80,000 steps — resumed 60,000 → 80,000 (pretrain-v3 resume run) · Kaggle T4 · peak LR 6e-4, cosine to 10% floor
Held-out perplexity	52.26 (fp32; bf16 serving weights reproduce it to 0.02%; shared 50-doc / 27,198-token FineWeb slice never trained on, n_ctx=512) — down from 57.64 at step 60000 on the same corpus. In-kernel 200k-token slice: 43.09 (different corpus, not comparable — see Evaluation).
Formats in this repo	ckpt_v7_step80000_bf16/ safetensors (400 MB) + tokenizer_v2/ — the serving weights. The root GGUFs are prior v1-lineage (439-token) artifacts, kept for history.
Inference	KV-cached native server, ~4.6 tok/s on shared CPU, streams SSE token-by-token
License	Apache 2.0

Lineage note. The v1 run (439-token vocab, 32k steps, PPL 54.16) is preserved below for history. The v2 run rebuilt the tokenizer at 32k and continued training to step 80000 — the v2 weights are a different lineage, not a continuation of the v1 numbers. Do not compare 54.16 (v1, Q4) against 52.26 (v2, bf16, different corpus).

---

📈 Training

Current release (v2). In-kernel held-out PPL, steps 62000 → 80000 (bf16, 200k-token slice): 47.11 → 43.09, decelerating into the cosine floor — the reason the next run re-warms to peak LR 6e-4 instead of adding more low-LR steps.

v1 lineage (frozen)

Nae1 v1 was pretrained from random initialization in eight cosine cycles: 0 → 4,000 · 4,000 → 8,000 · … · 24,000 → 28,000 · 28,000 → 32,000, each resumed from the previous checkpoint. The chart shows cycle eight — the first cycle trained with corrected next-token labels (an earlier double-shift bug had models predicting token t+2; the held-out gate was always standard next-token, so historical PPL numbers remain valid). Under the corrected objective the logged loss resumes at 28.87, falls to a best of 15.72 (step 31,500) and finishes at 17.93 as the learning rate anneals to zero. Held-out PPL improves 147.97 → 146.17 → 54.16 — a 63% drop from fixing the training objective; a fresh-corpus cross-check (unseen FineWeb CC-MAIN-2014-10: 57.09 vs step-28000's 143.64) confirms the jump is genuine generalization, not eval-set leakage.

• Data pipeline: FineWeb parquet → chunked into 513-token windows served as zero-copy views (peak-RAM-safe at 768M tokens)
• Checkpoints: every 250 steps; v1 release ships the final step-28000 checkpoint

v2 continuation. The v2 run (32k tokenizer, peak LR 6e-4) resumed from step 60000 and trained to step 80000. Gains decelerated into the cosine floor (held-out 57.64 → 52.26 per the shared anchor), motivating the next run's warm-restart recipe (re-warm floor → 6e-4 over 1k steps, then cosine 80k → 100k).

---

📊 Evaluation

Current release (v2). One shared readable corpus anchors all v2 claims — heldout_fineweb.jsonl (2,000 docs from unseen crawl CC-MAIN-2014-10), fp32, n_ctx=512, 50 docs / 27,198 tokens (the bf16 serving weights reproduce it to 0.02%): 57.64 @ step 60000 → 52.26 @ step 80000. The in-kernel 200k-token slice reads 43.09 — a different corpus, not a contradiction. One shared corpus exists precisely so this ambiguity never recurs.

v1 lineage (frozen)

Teacher-forced perplexity on a held-out FineWeb slice (992 documents / 1,027,766 tokens, n_ctx=512), scored token-by-token with exact row alignment — not the echo=True logprob path, which mispairs positions and inflates NLL.

Checkpoint (Q4_K_M)	Held-out PPL
step-2000	192.97
step-4000 (resumed run)	188.89
step-4000 (from scratch)	179.17
step-8000	165.92
step-12000	154.25
step-16000	151.45
step-20000	148.84
step-24000	147.97
step-28000	146.17
step-32000 — v1 release	54.16

Raw report: eval_step32000_q4_n512.json — mean NLL 3.9919 nats/token; fresh-corpus cross-check: eval_fresh_step32000_q4_n512.json / eval_fresh_step28000_q4_n512.json (prior releases kept alongside: eval_step24000_q4_n512.json, eval_step20000_q4_n512.json, eval_step16000_q4_n512.json, eval_step8000_q4_n512.json). Fixed probe ("The quick brown fox", 10 tokens): 174.73 — the 10-token probe is noisy; held-out decides.

⚠️ Perplexity is the trust signal here, not raw capability. At 100M params, expect exploratory, often garbled text — this is a research-lineage model, not a chat assistant. The base model does not follow instructions (no SFT weights yet — sweep in progress as of 2026-10-12).

Logits Parity Notice

Logits parity with the Nae1 native forward pass is not achievable by design: llama.cpp uses RMSNorm (no bias) while the native model uses LayerNorm (with bias), so conversion skips the bias tensors. Tensor integrity is fully verified (109/109 tensors match the source safetensors, max diff 0.00e+00) — small logit drift is a runtime constraint, not a conversion bug.

---

🛠️ Usage

Native server (current weights, recommended)

# weights: ckpt_v7_step80000_bf16/ + tokenizer_v2/ in this repo
python app.py  # see the nae1-inference repo for the SSE server

GET /health → status, step, held-out PPL. POST /chat streams data: {token, text, done, eos, step} frames (KV-cached, ~4.6 tok/s on shared CPU).

llama.cpp (v1-lineage GGUFs)

llama-server -m nae1-llama-Q4_K_M.gguf -c 512 --host 127.0.0.1 --port 8080

llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="nae1-llama-Q4_K_M.gguf",
    n_ctx=512,
    verbose=False,
)

out = llm("The future of AI", max_tokens=32, temperature=0.7)
print(out["choices"][0]["text"])

Files

File	What it is
ckpt_v7_step80000_bf16/	Current serving weights (bf16 safetensors, 400 MB) + config + train_state
tokenizer_v2/	Current 32k tokenizer (vocab + merges + manifest)
ckpt_v7_step60000/ · ckpt_v7_step60000_bf16/	Prior v2 checkpoint (fp32 + bf16), kept for lineage
nae1-llama-Q4_K_M.gguf	4-bit quant, 4.80 BPW, 45 MB — v1 lineage (439-token)
nae1-llama-f32.gguf	Full-precision GGUF, 305 MB — v1 lineage, for requantization / research
config.json	Architecture config as trained (v1)
tokenizer/	Native 439-token vocab + merges + manifest (v1)
eval_step32000_q4_n512.json	The exact v1 evaluation report cited above
eval_fresh_step32000_q4_n512.json	Fresh-corpus cross-check (unseen CC-MAIN-2014-10), v1 release
eval_fresh_step28000_q4_n512.json	Fresh-corpus cross-check, prior release
eval_step28000_q4_n512.json	Prior release (step-28000), kept for lineage
eval_step24000_q4_n512.json	Prior release (step-24000), kept for lineage
eval_step20000_q4_n512.json	Prior release (step-20000), kept for lineage
eval_step16000_q4_n512.json	Prior release (step-16000), kept for lineage
eval_step8000_q4_n512.json	Prior release (step-8000), kept for lineage

---

🧰 What Is NeuralAI?

NeuralAI is a local-first, private generative AI engine built by De'Andrew Preston Harris. The mission: your AI, on your hardware, under your control. Nae1 is the project's first end-to-end pretrained model — trained from random init on public data, converted with verified tensor integrity, and evaluated with a reproducible protocol.

---

⚠️ Limitations

• Scale: 100M parameters is a research checkpoint — long-form reasoning, coding, and factual recall are limited.
• Base model only: no chat template, no instruction tuning — it completes text, it does not follow instructions. An SFT sweep is running (2026-10-12); weights will ship when selected.
• No safety eval: nothing here guards truthfulness, toxicity, or prompt injection. Do not deploy to untrusted users.
• No internet access: pair with a tool layer if you need live data.

---

👤 Who Created NeuralAI?

• Founder & Lead Architect: De'Andrew Preston Harris (D. Harris / Dre)
• Hugging Face: @Subject-Emu-5259
• GitHub: @Subject-Emu-5259
• LinkedIn: linkedin.com/in/deandrewharris94
• Location: Memphis, Tennessee / West Memphis, Arkansas
• Education: AI Software Engineering at Maestro College

NeuralAI was built from resilience, fatherhood, and the belief that personal computing deserves personal intelligence. Every release is handcrafted, iterated, and documented in the open.

---

📖 Citation

@software{neuralai_nae1_2026,
  author       = {Harris, De'Andrew Preston},
  title        = {NeuralAI — Nae1},
  year         = {2026},
  url          = {https://huggingface.co/Subject-Emu-5259/NeuralAI-Nae1},
  version      = {step-80000},
  description  = {A 100M-parameter language model pretrained from scratch on FineWeb}
}

---

Built with discipline by De'Andrew Preston Harris. Maintained in the open. Updated whenever the model, dataset, or project state changes.