arvindcr4/iolai-2026-qwen25-14b-awq-b

IOL-AI 2026 — Qwen2.5-14B-Instruct-AWQ

Submission for the IOL-AI 2026 Linguistics Olympiad Challenge.

The weights are an unmodified copy of Qwen/Qwen2.5-14B-Instruct-AWQ (Apache-2.0, redistributable), shipped in-repo because the evaluation sandbox has no internet access. All of the work is in script.py.

Approach

The eval budget is 30 minutes on a 16 GB T4 for a test set of only ~90 sub-questions, so compute per problem is abundant while reliability is scarce. The script is built around that asymmetry.

1. Alignment first. Each row is a problem block with N numbered items and pred must be a JSON list of exactly N answers in order. A single missing line shifts every later answer and zeroes the whole block on both exact-match and chrF. detect_n_items recovers N from the query — handling numbered lines, (1) blank markers, stated ranges, lettered items, unnumbered one-per-line lists, and the match_letters shape whose items live in the shared context. Measured on the 160 public Linguini problems it puts 98.4% of items in correctly-sized blocks. Model output is then force-fitted to N, preferring the model's own numbering when it supplies it.

2. Never emit an empty answer. The final score is a geometric mean of exact match and chrF, so a blank scores zero on both and is strictly worse than a wrong guess. Every path ends in a non-empty string.

2b. Answer style: a hypothesis that was tested and rejected. Gold answers do follow the conventions of whatever language the answer is in (measured over the 920 public Linguini answers, into-English golds that are full sentences are 99% capitalised, while the 157 that are bare clauses are only 10% capitalised, and the style matches the problem's own glosses in 36/36 measurable cases). Encoding that as prompt guidance nevertheless lowered exact match on the hidden set twice (0.0250 -> 0.0218 -> 0.0000). It is therefore not in the shipped script. The lesson recorded here for anyone rerunning this: a correct statistical description of the gold format did not translate into a better prompt.

3. Monotone improvement under a hard deadline. A complete, correctly-shaped submission.csv is written before the model is loaded, then overwritten after every improvement: greedy pass → each self-consistency pass → explanations. A crash or a timeout leaves the best result reached so far on disk rather than nothing. The script tracks its own remaining budget and stops adding passes when one more would not fit.

4. Greedy-anchored voting. After the greedy pass, sampled passes (T=0.5) run while budget remains, but the greedy answer is the default and sampled answers may only displace it when at least two of them agree on the same normalised form and that form outpolls the greedy one.

The asymmetry is empirical. A symmetric version — majority, else "most central by chrF" — was measurably worse than not voting at all: with only a handful of samples the centrality fallback is ill-defined (with two candidates pairwise chrF is symmetric, so it degenerated into preferring the shorter string) and it swapped the greedy answer for a sampled one about half the time. On the mock set that cost 4x exact match (EM 0.044 -> 0.011). Anchoring makes the procedure monotone: it can only fire on genuine agreement. chrF is implemented inline so the script carries no dependency the sandbox might lack.

5. match_letters as an assignment problem. Free-form generation answers this task type with the identity permutation (A, B, C, ...), which is a valid permutation, so duplicate-repair never fires and it scores ~0. solve_matching instead scores every (item, option) pair from the next-token distribution and takes the optimal one-to-one assignment, enforcing the bijection exactly. Duplicate-repair is retained only as a fallback for when that solver declines.

Human Evaluation Challenge

submission.csv includes an explanation column: a short, human-readable statement of the rules behind each answer (not a raw reasoning trace), generated after the answers are fixed.

Reproducing

python script.py            # reads /tmp/data/test.csv, writes submission.csv

Environment knobs (all optional, defaults match the platform): IOL_TEST_CSV, IOL_OUT_CSV, IOL_MODEL, IOL_TIME_LIMIT, IOL_BATCH, IOL_EXPLAIN.

Revision history (measured on the hidden set, not guessed)

submission	change	score	chrF	exact match
1	symmetric self-consistency vote	0.0686	0.1882	0.0250
2	greedy-anchored voting (vote no longer fires)	0.0712	0.2029	0.0250
3	+ "capitalise English answers" style rule	0.0679	0.2117	0.0218
4	+ mirror-gloss-style, answer normalisation, equation hints	0.0000	0.1527	0.0000
5	revert to 2, plus the match_letters assignment solver
6	+ repetition_penalty=1.0 (the model ships 1.05)

Every layer of prompt/post-processing cleverness measurably hurt. Submission 5 therefore reverts to the configuration of submission 2 and adds exactly one change, motivated by a specific measured failure:

match_letters was being answered with the identity permutation. Replaying seven parser variants over saved raw generations gave exact match 0.0000 for all seven, which exonerates the parser — the model simply was not solving the task, emitting the option labels in order (A, B, C, ...). Because the identity is a valid permutation, repair_bijection never fired. solve_matching replaces free-form generation for this task type: it scores every (item, option) pair from the next-token distribution and takes the optimal one-to-one assignment, so the bijection constraint is enforced exactly rather than hoped for.

The silent decoding bug

Qwen/Qwen2.5-14B-Instruct-AWQ ships generation_config.json containing repetition_penalty: 1.05. Greedy decoding ignores temperature, top_p and top_k — and transformers emits a warning for each of those — but a repetition penalty is applied under greedy decoding, with no warning at all.

That matters here specifically: 34% of the 920 public gold answers repeat some letter three or more times, because these languages are agglutinative and the answers look like ɨmpʼuhurʼu and ɨŋɡɨrʼɨ. A 5% penalty on repeated tokens biases the model away from exactly the strings the task requires. The script now passes repetition_penalty=1.0 explicitly.

NFC normalisation of answers was considered and rejected: 98.15% of public golds are already NFC, but 13 of them are NFD-and-not-NFC, so forcing NFC would break those for an unmeasured gain.

v8 — faithful baseline replication

The organizers' reference script reaches exact match 0.0729 on the hidden set with these exact weights. Our best is 0.0333. Before adding anything further we need to know whether that number is reproducible by us at all, so v8 replicates their script literally — trivial system prompt, no chain-of-thought, batch 1 (no padding at all), naive line split, and no forcing to N answers — changing exactly one thing: repetition_penalty=1.0. Generation is EOS-limited rather than cap-limited: without chain-of-thought the model emits a few short answer lines and stops.

Result: 0.2245 (chrF 0.3150, exact match 0.1600) — first place of 43 teams.

That is 2.7x our best engineered pipeline (0.0830) and +83% on the organizers' own baseline (0.1227), the entire delta over their number being repetition_penalty=1.0.

The lesson is uncomfortable and worth recording plainly: every layer we added on top of the reference structure — chain-of-thought, an ANSWERS: block, answer style rules, output normalisation, forcing exactly N answers — reduced exact match. Submissions 2 -> 3 -> 4 fell 0.0712 -> 0.0679 -> 0.0000 as more engineering went in. The winning move was deleting all of it and fixing one decoding flag.