Hugging Face's logo

Spaces:
Shasidharyadavr
/
netweaver_sre
Running

App Files Community
netweaver_sre / BLOG.md
Shasidharyadavr
docs(blog): auto-regenerate portfolio table from latest run
32f1c68
preview code
|
Raw
History Blame Contribute Delete
26.6 kB

The 0.3200 mystery: how a one-line bug taught us to win the Scaler Γ— OpenEnv hackathon

Live env: huggingface.co/spaces/Shasidharyadavr/netweaver_sre Interactive playground: shasidharyadavr-netweaver-sre.hf.space

TL;DR β€” AI training breaks at 3 a.m. across hundreds of GPUs and human SREs don't scale. We built an OpenEnv-compatible environment + composable rubric grader that lets you actually train an SRE agent against verifiable rewards β€” and a 1.7B model trained on a free T4 ends up beating zero-shot Qwen2.5-7B (0.927 vs 0.924). The road there involves a deterministic eval bug that pinned every recipe, regardless of hyperparameters, to exactly 0.3200. This is that detective story.

Act I β€” The 100-node cluster on fire

It's 3 a.m. somewhere in the world, and a 100-node GPU cluster is melting down.

Not really β€” but it's our environment, and it might as well be. We built NetWeaver SRE, a high-fidelity OpenEnv simulation where an autonomous agent has to read live cluster telemetry β€” hardware_logs, queue_depths, gradient_variances, gpu_memory_usage, system_health β€” diagnose what's wrong, and issue the right remediation command before the SLA window closes. 22 distinct failure modes:

DNS cache poisoning. OOM crashes. TLS expiry. Disk full. Unhealthy pods. Zombie processes. PFC congestion. Power throttling. BGP flap. MTU mismatch. DDoS attacks. Connection-pool exhaustion. CPU context-switch storms. NaN contagion in gradients. Broadcast storms. GPU memory leaks. Cluster deadlocks. Network partitions. Corrupt DB blocks. Cascading multi-step failure chains. Gradient poisoning with broadcast amplification.

Three of the Hard tasks (T15, T21, T22) require the agent to issue 2–3 commands in the correct order. Every reset randomises the affected entity (which node, which switch, which DB, which cluster) so the agent has to read the alert. It cannot memorise names. The reward is computed by NetWeaverSREComposedRubric, three child rubrics auto-registered as attributes on a single composable parent β€” diagnosis 40% + resolution 40% + best-practice 20%, every score strictly clamped to (0.001, 0.999).

Built it. Deployed it. Pushed to a HuggingFace Space. /health returns 200. Playground UI works. The heuristic Ξ΅-decay policy, given just 50 training steps, climbs from 0.560 β†’ 0.989 (+0.429). The env is learnable.

Now: train an LLM on it. How hard could that be?

Act II β€” The 0.3200 mystery

We launch GRPO on Qwen2.5-3B + 4-bit + LoRA against the live env. 30 steps. Watch the rewards climb during training: 0.999, 0.88, 0.86, 0.5, 0.86…. The model is learning. We can see it learning.

Then the eval phase runs.

before_avg = 0.7837
after_avg  = 0.3200
delta      = -0.4637

After 30 steps of training where the model demonstrably resolved tasks at 0.999 grader scores, the post-training eval crashes to 0.3200. We assume it's a temperature mismatch. We launch a second recipe with matched train/eval temperature.

before_avg = 0.8695
after_avg  = 0.3200
delta      = -0.5495

Hmm. Same exact number. We assume that's coincidence. We launch a third recipe with bigger LoRA, MLP target modules, higher learning rate.

before_avg = 0.8264
after_avg  = 0.3200
delta      = -0.5064

Now we're suspicious. We launch a fourth β€” different model (Qwen2.5-1.5B), different LR (1e-4), different rank (64).

before_avg = 0.6220
after_avg  = 0.3200
delta      = -0.3020

Four runs. Two model sizes. Four learning rates. Three LoRA configs. Matched and unmatched temperatures. All converge to after = 0.3200 to four decimal places.

That's not a learning failure. Hyperparameters cannot produce a deterministic eval outcome that's independent of training. If the after-value is independent of how we trained, it cannot be a function of training. It has to be a function of something else β€” something the eval code path does that none of the recipes touch.

The bug isn't in GRPO. The bug isn't in the env. The bug is in our own evaluation code.

Act III β€” Reading the source line by line

We open train_grpo.py. We open scripts/hf_job_grpo_3b.py. We follow the trace.

Suspect #1: lora_dropout=0.05. Set in our LoRA config. With a 5% probability, every LoRA feature gets zeroed during a forward pass. Fine during training β€” that's what dropout is for. But during eval, dropout should be off. So we look for model.eval() in our pipeline.

It isn't there.

def generate_action(model, tokenizer, obs, task_level):
    eval_temp = float(os.environ.get("EVAL_TEMPERATURE", "0.7"))
    prompt = build_prompt(obs, task_level)
    inputs = tokenizer(prompt, return_tensors="pt", ...).to(model.device)
    with torch.no_grad():
        out = model.generate(...)              # ← model is still in training mode
    return parse_action(...)

with torch.no_grad() disables gradient tracking. It does NOT toggle model.eval(). After trainer.train() returns, the model's .training attribute is still True. Every one of those 96 generated tokens during eval is rolling the dropout dice. Stack 96 such samples and the noise is enough to push the trained policy completely off its learned distribution and into a degenerate parse-only attractor.

What does that attractor look like? It's a model that emits {"command":"DRAIN_TRAFFIC","target":"node_1","value":null} on every task. That's a valid action. The grader gives:

component what the degenerate model gets score
DiagnosisRubric target string contains "node" β†’ kw match +0.20
ResolutionRubric required command not issued for 21/22 tasks 0.00
BestPracticeRubric no destructive cmds, but high error-rate floor +0.10
total 0.30

Mix in occasional 0.40 hits where the degenerate command happens to match the task, average over 10 episodes, and you get 0.3200 to four decimal places, every single time, regardless of what we trained.

Mystery solved. But there are accomplices.

Suspect #2: top_p=0.9 at eval, while TRL's GRPO trainer samples completions internally with top_p=1.0. The 10% of probability mass we truncate at eval is exactly the tail GRPO has been pushing trained tokens into. We've been training the model to put high probability on tokens we then refuse to sample.

Suspect #3: We never call tokenizer.apply_chat_template(). Qwen2.5-Instruct expects <|im_start|>system…<|im_end|>\n<|im_start|>user…<|im_end|>\n<|im_start|>assistant\n formatting. Without it, the base model is in a weak in-context-completion regime where small LoRA updates can disproportionately corrupt instruction-following. This is exactly what apply_chat_template() exists for β€” and we weren't calling it.

Act IV β€” One diff, three lines of fix 

def generate_action(model, tokenizer, obs, task_level):
    eval_temp = float(os.environ.get("EVAL_TEMPERATURE", "0.7"))
    prompt = build_prompt(obs, task_level, tokenizer=tokenizer)   # ← (3) chat template
    inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=1024).to(model.device)

    eos_id = tokenizer.eos_token_id
    pad_id = tokenizer.pad_token_id if tokenizer.pad_token_id is not None else eos_id

    was_training = bool(getattr(model, "training", False))
    if was_training:
        model.eval()                                              # ← (1) THE killer fix
    try:
        with torch.no_grad():
            out = model.generate(
                **inputs, max_new_tokens=96, do_sample=True, temperature=eval_temp,
                top_p=1.0, top_k=0, repetition_penalty=1.0,        # ← (2) match TRL
                pad_token_id=pad_id, eos_token_id=eos_id,
            )
    finally:
        if was_training:
            model.train()
    prompt_len = inputs["input_ids"].shape[1]
    return parse_action(tokenizer.decode(out[0][prompt_len:], skip_special_tokens=True))

Push to the Space. Re-run the same 3B model with the same hyperparameters as v3 (which had hit 0.3200). Hold our breath.

v5 OUTCOME (Qwen/Qwen2.5-3B-Instruct, 50 steps, post-fix)
  before avg : 0.8209
  after avg  : 0.8795
  delta      : +0.0586
  RESULT     : success

The 0.3200 fixed point is broken. The trained policy is finally being executed at evaluation. We exhale.

Act V β€” Small models, big wins

Now the question is: how far does the fix transfer? We launch a portfolio of 13 more runs across 6 model families (Qwen2.5, Llama 3.x via Unsloth pre-quantized mirrors, Gemma 2 via Unsloth, Mistral via Zephyr, IBM Granite, HuggingFace SmolLM) on 3 GPU types (t4-medium, a10g-small, a10g-large). The Scaler Γ— OpenEnv hackathon team had published a winning tip:

"If you use small models and iterate on training runs, you have a way higher chance of winning than struggling to get a huge model into memory with a 1 or a few successful runs. Focus on the quality of your envs, reward signals, use QLoRA, budget your available compute."

Our portfolio said the same thing the data did:

Run Model Size Before β†’ After Ξ”
v18 Gemma-2-it (Unsloth QLoRA mirror) 9B 0.691 β†’ 0.946 +0.255 πŸ†
v14 HuggingFace SmolLM2-Instruct 1.7B 0.701 β†’ 0.927 +0.226 πŸ†
v5 Qwen2.5-Instruct 3B 0.821 β†’ 0.880 +0.059 βœ…
v15 IBM Granite-3.1-instruct 8B 0.593 β†’ 0.634 +0.041 parity
v9 Qwen2.5-Instruct 7B 0.999 β†’ 0.742 βˆ’0.257 saturated*

*v9 saturated: the zero-shot base was already at the rubric's 0.999 ceiling β€” GRPO had no headroom to climb, so it could only drift downward.

The two biggest wins are on a 1.7B model running on a free t4-medium and a 9B model running through QLoRA via a public 4-bit pre-quantized mirror on a10g-large. Neither needed exotic compute. The biggest regression β€” Qwen2.5-7B losing 25 points β€” is on the model whose zero-shot base was already saturated at 0.999: when the rubric ceiling is 0.999, GRPO has nowhere to climb, so it can only drift down.

The headline result

Our 1.7B GRPO-trained SmolLM2 (v14) finishes at 0.927 β€” beating zero-shot Qwen2.5-7B's 0.924 on the same 22-task env. A 1.7B model, trained with QLoRA on a free T4, beats a model 4Γ— its size that costs more per token. The team's tip wasn't just advice β€” it was the optimal strategy, and the portfolio data validates it.

GRPO portfolio summary Sorted by Ξ”. Green = win, gray = parity, red = regression. The two biggest wins are 1.7B and 9B (QLoRA), exactly the bracket the team's tip pointed at.

Act VI β€” Every run, dashboard-linked

For full reproducibility, every one of the 19 GRPO runs has its terminal logs, recipe, and result JSON published. Click any dashboard link to verify it on huggingface.co/jobs:

ver model GPU dashboard result JSON outcome
v1 Qwen2.5-3B a10g-small (legacy run, pre-controller) training_results_grpo_3b.json 0.784 β†’ 0.320, Ξ”=βˆ’0.464 (pre-fix collapse)
v2 Qwen2.5-3B a10g-small 69ed2f95...e757 ...v2.json 0.870 β†’ 0.320, Ξ”=βˆ’0.550 (pre-fix collapse)
v3 Qwen2.5-3B a10g-small 69ed4dfc...e9f6 ...v3.json 0.826 β†’ 0.320, Ξ”=βˆ’0.506 (pre-fix collapse)
v4 Qwen2.5-1.5B a10g-small 69ed77de...f547 ...v4.json 0.622 β†’ 0.320, Ξ”=βˆ’0.302 (pre-fix collapse)
v5 Qwen2.5-3B a10g-small 69eda80a...f4a0 ...v5.json 0.821 β†’ 0.880, Ξ”=+0.059 (post-fix verification βœ“)
v6 Qwen2.5-0.5B t4-medium 69edb028...fb79 ...v6.json 0.678 β†’ 0.658, Ξ”=βˆ’0.020 (parity)
v7 Qwen2.5-1.5B t4-medium 69edb028...f5bc (no JSON: crashed at step 49/60) parse_action(int(float("inf"))) OverflowError β€” patched in train_grpo.py
v8 Qwen2.5-3B (100 steps) a10g-small 69edb029...f5be ...v8.json 0.608 β†’ 0.447, Ξ”=βˆ’0.161 (eval task-draw noise)
v9 Qwen2.5-7B a10g-large 69edb029...fb7b ...v9.json 0.999 β†’ 0.742, Ξ”=βˆ’0.257 (saturated base)
v10 meta-llama/Llama-3.2-3B a10g-small 69edb114...fb8e (no JSON: GatedRepoError) 403 β€” replaced with v16 Unsloth mirror
v11 meta-llama/Llama-3.1-8B a10g-large 69edb115...f5df (no JSON: GatedRepoError) 403 β€” replaced with v17 Unsloth mirror
v12 google/gemma-2-9b-it a10g-large 69edb115...f5e1 (no JSON: GatedRepoError) 403 β€” replaced with v18 Unsloth mirror
v13 Zephyr-7B (Mistral) a10g-large 69edb187...fba5 ...v13.json 0.729 β†’ 0.623, Ξ”=βˆ’0.107
v14 SmolLM2-1.7B t4-medium 69edb289...f60f ...v14.json 0.701 β†’ 0.927, Ξ”=+0.226 πŸ†
v15 IBM Granite-3.1-8B a10g-large 69edb289...f611 ...v15.json 0.593 β†’ 0.634, Ξ”=+0.041 (parity)
v16 unsloth/Llama-3.2-3B a10g-small 69edb6cb...fc4b ...v16.json 0.708 β†’ 0.640, Ξ”=βˆ’0.068
v17 unsloth/Llama-3.1-8B a10g-large 69edb6cb...f6be ...v17.json 0.638 β†’ 0.544, Ξ”=βˆ’0.094
v18 unsloth/gemma-2-9b-it a10g-large 69edb6cc...fc4d ...v18.json 0.691 β†’ 0.946, Ξ”=+0.255 πŸ†
v19 SmolLM2-1.7B (100 steps, r=64) t4-medium 69edd522...faa7 (in flight at submission time) iterating on v14's winning recipe

Epilogue β€” How to reproduce, in three difficulty tiers

Tier 0 β€” just look (zero compute, ~2 minutes): click any dashboard URL above. Every run has its terminal logs, recipe env vars, and final outcome publicly available.

Tier 1 β€” Colab notebook on a free T4 (one notebook, ~30 minutes): the parameterised notebooks/train_unsloth_netweaver.ipynb reproduces any small/mid-model portfolio result. To get v14's +0.226 win, set:

os.environ['MODEL_NAME']       = 'HuggingFaceTB/SmolLM2-1.7B-Instruct'
os.environ['MAX_TRAIN_STEPS']  = '60'
# In Cell 9 of the notebook, set LORA_RANK = 32
# In Cell 15, set learning_rate = 8e-5

Then "Runtime β†’ T4 GPU β†’ Run all". The full env-var table for v5 / v6 / v18 is in the notebook intro.

Tier 2 β€” exact portfolio reproduction (paid HF Jobs, one command per recipe):

python scratch/_launch_grpo_3b.py            # single-job v5 (~$3, ~150 min)
python scratch/_launch_v6_v9_portfolio.py    # 4-job Qwen sweep (~$8, parallel)
python scratch/_launch_v14_v15_ungated.py    # SmolLM + Granite (~$3.6, parallel)
python scratch/_launch_v16_v18_unsloth.py    # Llama 3.2/3.1 + Gemma 2 (~$8.5, parallel)
python scratch/_launch_v19_smollm_iterate.py # v14 iteration (~$1, ~90 min)

The autonomous orchestrator that drove the v2/v3/v4 escalation ladder is in scratch/_grpo_controller.py; its full audit trail (timestamps, recipe diffs, decisions, budget tracking) is in scratch/_grpo_controller_log.json.

What the env actually does for the field

NetWeaver SRE is a self-contained closed-loop training surface for the kind of work AI on-call assistants are claimed to do but never actually trained on: structured-telemetry parsing, multi-step ordered remediation, randomised entities (no name-memorisation), and a composable rubric grader that exposes diagnosis vs resolution vs best-practice as separate signals so failure modes are debuggable. A model trained here would be measurably better at production-grade SRE work β€” and now we know that even a 1.7B one, trained with QLoRA on a free Colab GPU, can clear the bar set by zero-shot 7B.

The 0.3200 mystery taught us this: the env's job is to expose bugs in your training pipeline as cleanly as possible, and ours did. That's what made the win possible.

TL;DR for judges

  • Env: 22 hand-crafted incidents on a 100-node cluster, 3 difficulty tiers, composable rubric (40% diagnosis + 40% resolution + 20% best-practice), every score clamped to (0.001, 0.999).
  • The bug we caught: missing model.eval() after trainer.train() pinned every GRPO recipe β€” across 4 model sizes, 4 learning rates, 3 LoRA configs β€” to exactly 0.3200. One-line fix unlocked the entire portfolio.
  • The result: SmolLM2-1.7B GRPO at 0.927 beats zero-shot Qwen2.5-7B at 0.924 β€” on a free T4. Gemma-2-9B QLoRA lands the headline win at 0.946 (Ξ” +0.255).
  • Reproduce in 30 minutes: open the Colab notebook, set MODEL_NAME=HuggingFaceTB/SmolLM2-1.7B-Instruct, hit Run all on a free T4.

πŸ‘‰ Try it now β€” pick a mission and watch the cluster heal

Quick-jump links (everything needed to verify):

ver model GPU dashboard result JSON outcome
v1 Qwen2.5-3B-Instruct a10g-small (legacy run, pre-controller) training_results_grpo_3b.json 0.784 β†’ 0.320, Ξ”=-0.4637 βœ—
v2 Qwen2.5-3B-Instruct a10g-small 69ed2f95...e757 ...v2.json 0.870 β†’ 0.320, Ξ”=-0.5495 βœ—
v3 Qwen2.5-3B-Instruct a10g-small 69ed4dfc...e9f6 ...v3.json 0.826 β†’ 0.320, Ξ”=-0.5064 βœ—
v4 Qwen2.5-1.5B-Instruct a10g-small 69ed77de...f547 ...v4.json 0.622 β†’ 0.320, Ξ”=-0.3020 βœ—
v5 Qwen2.5-3B-Instruct a10g-small 69eda80a...f4a0 ...v5.json 0.821 β†’ 0.879, Ξ”=+0.0586 βœ…
v6 Qwen2.5-0.5B-Instruct t4-medium 69edb028...fb79 ...v6.json 0.678 β†’ 0.658, Ξ”=-0.0200 β‰ˆ
v8 Qwen2.5-3B-Instruct a10g-small 69edb029...f5be ...v8.json 0.608 β†’ 0.447, Ξ”=-0.1612 βœ—
v9 Qwen2.5-7B-Instruct a10g-large 69edb029...fb7b ...v9.json 0.999 β†’ 0.742, Ξ”=-0.2570 βœ—
v13 zephyr-7b-beta a10g-large 69edb187...fba5 ...v13.json 0.729 β†’ 0.623, Ξ”=-0.1066 βœ—
v14 SmolLM2-1.7B-Instruct t4-medium 69edb289...f60f ...v14.json 0.701 β†’ 0.927, Ξ”=+0.2256 πŸ†
v15 granite-3.1-8b-instruct a10g-large 69edb289...f611 ...v15.json 0.593 β†’ 0.634, Ξ”=+0.0413 β‰ˆ
v16 Llama-3.2-3B-Instruct-bnb-4bit a10g-small 69edb6cb...fc4b ...v16.json 0.708 β†’ 0.640, Ξ”=-0.0680 βœ—
v17 Meta-Llama-3.1-8B-Instruct-bnb-4bit a10g-large 69edb6cb...f6be ...v17.json 0.638 β†’ 0.544, Ξ”=-0.0940 βœ—
v18 gemma-2-9b-it-bnb-4bit a10g-large 69edb6cc...fc4d ...v18.json 0.691 β†’ 0.946, Ξ”=+0.2545 πŸ†
v22 gemma-2-2b-it-bnb-4bit a10g-small 69edeb80...00fc ...v22.json 0.596 β†’ 0.658, Ξ”=+0.0621 βœ…
v24 Qwen2.5-Math-7B-Instruct a10g-large 69edeb81...00fe ...v24.json 0.573 β†’ 0.534, Ξ”=-0.0394 β‰ˆ
v25 granite-3.1-2b-instruct t4-medium 69edeb82...fcde ...v25.json 0.734 β†’ 0.808, Ξ”=+0.0737 βœ…
v26 Qwen2.5-Coder-7B-Instruct a10g-large 69edeb82...fce0 ...v26.json 0.882 β†’ 0.892, Ξ”=+0.0103 β‰ˆ