MK4-Research/LOREA-cyber-v5.9

LOREA-cyber v5.9

Qwen3.6-35B-A3B with a LoRA merged in, 4-bit MLX. Mixture of experts, so about 3B parameters are active per token and it runs faster than the size suggests. No adapter needed at inference.

It reviews code for security problems. Compared to v5.8 it finds more of them, invents fewer, and explains itself more usefully.

Against v5.8

Both models measured in the same session, same harness, same prompt, same settings. This is the comparison that matters; see the note on absolute numbers below.

	v5.8	v5.9
finds planted bugs (n=104)	87.5%	89.4%
full credit: bug, impact and fix	43.3%	51.9%
distracted by planted noise	2.9%	1.9%
claims a bug in correct code (n=65)	21.5%	16.9%

v5.9 is better on all four. That is unusual. The normal trade is recall against false alarms, and earlier versions of this model paid for restraint with misses. This one did not.

Two other checkpoints from the same run were measured, and one of them may suit you better. At 800 iterations the model reaches 7.7% false alarms with zero distraction, at the cost of dropping to 83.7% recall. If you care more about not being sent on wild goose chases than about catching everything, that is the better trade. It is not what is published here; this repo is the 400-iteration checkpoint, which was the strongest all-round result.

About the absolute numbers

They are lower than v5.8's model card reports, and that needs explaining rather than hiding.

v5.8's card quotes 9% false alarms and 91% recall. Those were measured with a different system prompt and a different generation mode. Re-running v5.8 under the harness used here gives 21.5% and 87.5%. Nothing about v5.8 changed; the measurement did. The old and new numbers are not comparable, and mixing them would be misleading in v5.9's favour.

Everything in the table above was produced in one sitting with:

system prompt: "Report any issues you can substantiate. If the code is correct, say so."
temperature 0.0, thinking disabled, max_tokens 460 (recall) / 420 (false alarms)

About the grader

The false-alarm figure is trustworthy: it detects whether a vulnerability was asserted. The companion "correctly cleared" figure is not, and is omitted for that reason. It matches on fixed phrases and misses correct answers phrased differently — it scores "This is a constant-time comparison, which is the correct way to compare secrets" as unclassifiable. An independent re-scoring that asks only whether a vulnerability was asserted puts affirmed-safe at 61.5% for v5.8 and 69.2% for v5.9, but that is a second opinion rather than the benchmark, so it is context here and not a headline.

What was not measured

Stated plainly, because a model card that only lists what went well is not much use.

The reasoning-length statistics were not computed. Reducing runaway reasoning was an explicit goal of this version. Spot checks look right: the fixed Here's a thinking process: opener that dominated v5.8 does not appear in the training data at all, and did not appear in test generations. But "looks right" is not a measurement.

The refusal and over-refusal sets were not re-run. Those were measured for v5.8 and are not reported here, because they were not measured for this model.

Public benchmarks

Twenty-two benchmarks, all run in one session on this model. This was ran without thinking.

MMLU	84.0%	SciQ	98.0%
MMLU-Pro	55.5%	MedMCQA	69.5%
ARC-Challenge	96.0%	RACE-high	89.3%
ARC-Easy	98.5%	BoolQ	94.0%
HellaSwag	91.5%	TruthfulQA (mc1)	77.0%
WinoGrande	79.0%	BBH	85.0%
PIQA	94.0%	GSM8K	82.0%
SIQA	78.0%	HumanEval	90.0%
OpenBookQA	92.0%	MBPP	79.0%
CommonsenseQA	85.0%	SecQA	99.0%
		CyberMetric	93.5%
		cyber MCQ, held out	99.3%

n=200 per benchmark except RACE, GSM8K and the held-out cyber set (150) and HumanEval and MBPP (100). Greedy decoding, single sample, no few-shot examples.

These numbers are not comparable to published leaderboard figures, in either direction. That matters more than any individual score here, so it is worth being specific about why.

Everything above was run with thinking disabled and, for multiple choice, with the model asked for a single letter. That protocol is harder than the usual one on reasoning-heavy benchmarks and easier on some others:

• MMLU-Pro is ten-option and built to require chain-of-thought. Measured with reasoning switched off it lands around 55%, while the base model's published figure is far higher. The fine-tune did not cause that gap; the protocol did. v5.8 scores 55.0% under the same conditions, so the two fine-tunes agree with each other.
• TruthfulQA here is mc1 answered by letter choice. Leaderboards usually score it by log-likelihood ranking over the options, and often report mc2. Letter choice with an instruction-tuned model is the easier task, so 77% is not the same measurement as a 77% on a leaderboard.
• HumanEval and MBPP execute the generated code against the real test suites. Pass or fail, no answer extraction, nothing to argue about. Those two travel better than the rest.

The purpose of this table is to show the fine-tune did not damage general capability, and to give the security scores somewhere to sit. It is not a claim of rank against other models.

HumanEval at 90% is the one that closes a real question: every previous checkpoint in this project paid something on coding for its security gains, and this one does not appear to have.

Prompting

The system prompt moves the false-alarm rate by roughly six times on the same model and the same code. This is not a small effect and it is not optional.

prompt	false alarms
"Identify the security flaw, the invariant it breaks..."	60%
"Report any issues you can substantiate. If the code is correct, say so."	10%

Asking a model to identify the flaw tells it a flaw exists. Give it permission to find nothing.

Suggested:

You are a security code reviewer. Review the code and report any security issues you can
substantiate. If the code is correctly implemented, say so plainly.

Example output

Given a correct function using a parameterized query:

This code is correctly implemented. The query uses parameterized queries with a placeholder (?) and passes the user ID as a parameter, which prevents SQL injection. The result is safely extracted from the fetched row.

Given open('/var/data/' + name).read():

This is a path traversal vulnerability. The function takes a user-controlled name parameter and concatenates it directly to a base directory path without sanitization. An attacker can use ../ sequences to escape the /var/data/ directory and read arbitrary files on the system. A secure implementation would validate that the resolved path remains within the intended directory, for example by using os.path.realpath().

v5.8 frequently named the bug without volunteering the fix. v5.9 usually includes it, which is most of the jump in the full-credit number above.

Running it

python3 -m mlx_lm.chat --model MK4-Research/LOREA-cyber-v5.9

Needs roughly 20 GB of memory at 4-bit.

Training

LoRA, rank 16, learning rate 2e-5, on 4,829 rows. Attention projections only (q_proj, k_proj, v_proj, o_proj).

The change from v5.8 that mattered most was layer coverage. Qwen3.6 is a hybrid architecture: most layers are Mamba-style state-space blocks, and only every fourth layer has real attention. v5.8 was configured for the last ten transformer layers, but only three of those (31, 35, 39) actually have attention projections, so it adapted twelve weight matrices and 1.03M parameters. v5.9 covers all ten full-attention layers (3, 7, 11, 15, 19, 23, 27, 31, 35, 39) for 3.44M parameters.

Do not put LoRA on the expert MLPs. There are 256 of them; including them gives 256M trainable parameters and runs out of memory immediately on a 32 GB machine.

Training ran 1,200 iterations in one uninterrupted cycle with checkpoints every 50. Loss oscillated between 0.78 and 1.03 throughout and never trended toward the collapse that indicates memorisation. Peak memory 25.5 GB. The published checkpoint is iteration 400, chosen on measured results rather than on final loss — later checkpoints traded recall for precision rather than improving.

Data was rebuilt from v5.8's set: 1,240 code-review rows (31.4% of the mix), with additions covering claim verification, concise reasoning, multi-turn tool use, and cases where the honest answer is "I cannot determine this from what is shown". Decontaminated by eight-token overlap against both evaluation suites.

Limits

It is a 4-bit quantized model and it will still be wrong sometimes. It misses about one bug in ten on the internal benchmark, and claims a bug in correct code roughly one time in six.

Its confident answers need checking. An earlier version of this model insisted a Python server's authentication was completely broken because hmac.compare_digest supposedly rejects strings. That is false — it accepts ASCII strings — and the model reasoned through several rounds of self-correction before landing on the wrong answer anyway. Improving claim verification was a goal of this version, but it was not measured in isolation, so treat that failure mode as reduced rather than fixed.

It has not been tested on large real codebases, only on snippets and small multi-file examples. Treat its output as a starting point for review, not a verdict.

Intended for authorized security work: reviewing code you own or have permission to assess, CTF practice, and teaching.

Related

• Previous release: MK4-Research/LOREA-cyber-v5.8 (also has a GGUF build)
• Benchmarks: MK4-Research/VAB-vulnerability-analysis-benchmark
• Training data: MK4-Research/LOREA-cyber-training-data
• Evaluation sets: MK4-Research/LOREA-cyber-eval