C5 ties_sum rho=0.5 lambda=0.7 — merge ablation (NOT the recommended checkpoint)
This is a negative result published for completeness. It performs worse than
yungisimon/Qwen2.5-14B-ties-merge-nqa-msq-bcp, which is the checkpoint to use. This one exists to document what was tried and ruled out.
Merges three 14B MEMORY models — each fine-tuned on a different corpus (NarrativeQA, MuSiQue, BrowseComp-Plus) — into one set of weights, entirely in parameter space.
Recipe
python main.py \
--models <nqa> <msq> <bcp> \
--base Qwen/Qwen2.5-14B-Instruct \
--method ties_sum --weight 0.7 0.7 0.7 --density 0.5 0.5 0.5 --raw-weights --anchor-base
Task vectors are computed against the pretrained base, tau_i = phi_i - phi_0.
| alpha = <D_merged, tau_i>/||tau_i||^2 (nqa / msq / bcp) | 0.72 / 0.74 / 0.77 |
| ||D_merged|| / ||phi_0|| | 8.0% |
| single-corpus fine-tune, for reference | 4.8–7.0% |
alpha = 1.0 would mean a corpus's fine-tuning survived the merge at full strength.
Measured result
Single-turn knowledge recall: the MEMORY model is asked corpus questions directly, with no multi-turn protocol, judged against gold answers by Qwen2.5-32B-Instruct. n=80 per corpus, restricted to the subsets each model was actually trained on.
| model | BrowseComp-Plus | MuSiQue |
|---|---|---|
| this checkpoint | 7.5% | 18.8% |
yungisimon/Qwen2.5-14B-ties-merge-nqa-msq-bcp (recommended) |
16.2% | 25.0% |
| per-corpus specialist (ceiling) | ~37% | ~36% |
| untrained Qwen2.5-14B-Instruct (floor) | 0% | ~2.5% |
The reference figures are from the same probe run as this checkpoint, because cross-run noise is about +/-5pp at n=80 (the reference measured 21.2 / 16.2 / 25.0 on BCP across three runs). Differences smaller than ~5pp are not resolvable at this sample size.
The decisive experiment: displacement matched to the reference (8.0% vs 7.9%) with sign-consensus filtering intact, but higher task-vector retention. It loses, which is what rules out retention as the objective.
Why these all failed
The three task vectors are near-orthogonal in aggregate (pairwise cosine 0.002–0.038), which suggests summing them rather than averaging, since orthogonal updates should not interfere. That reasoning is wrong: 75.4% of coordinates carry a sign conflict between the three vectors. Global orthogonality does not imply coordinate-wise compatibility. TIES's magnitude trimming and sign-consensus filtering are therefore doing necessary conflict resolution, not merely attenuating the task vectors — so recipes that preserve more of each task vector (higher alpha) consistently do worse. Six candidates across four families (pure sum, sparse sum, sub-0.3 sparsity, non-uniform weighting) all lost to plain TIES rho=0.3 with uniform averaging.
Caveats
- The probe asks the full question in a single turn, whereas the MEMORY model is trained on decomposed, self-contained reflection pairs and is used at inference on decomposed sub-queries. Absolute numbers are therefore well below published protocol accuracy and are not comparable to it; the probe is a relative screen of knowledge accessibility.
- Judging is a single greedy pass of one local model, not human-validated.
- A NarrativeQA column was collected but discarded: the NQA specialist scored 0% on its own corpus, so that probe measures prompt-style mismatch rather than knowledge.
License
Inherits the license of Qwen/Qwen2.5-14B-Instruct (Apache 2.0).
- Downloads last month
- 24