MohammadKhosravi/llama3.1-8b-prefixmemory-cefr-balanced-topic-aligned

PrefixMemory-Tuning: Decoupled CEFR Baseline

This repository contains the PrefixMemory-Tuning (PMT) baseline developed for controllable CEFR-level text generation.

The model extends Meta Llama-3.1-8B-Instruct with a lightweight external PrefixMemory controller while keeping all base LLM parameters frozen. Only the PrefixMemory controller is optimized during training.

This model serves as the baseline architecture against which all subsequent hierarchical and dual-stream ablation models are compared.

---

Base Model

Foundation Model

• Model: Meta-Llama-3.1-8B-Instruct
• Parameters: 8 Billion
• Backbone: Decoder-only Transformer
• Transformer Layers: 32
• Hidden Dimension: 4096
• Frozen during training: Yes

Only the PrefixMemory controller is trainable.

---

Architecture

The architecture follows the PrefixMemory-Tuning paradigm by attaching a lightweight external memory controller to every transformer attention block.

Figure 1. Overall architecture of the CEFR controller integrated with PrefixMemory..

Controller Pipeline

For a target CEFR level

$$c\in \{A1,A2,B1,B2,C1,C2\}c\; \backslash in\; \backslash \{A1,\; A2,\; B1,\; B2,\; C1,\; C2\backslash \}$$

the controller first retrieves an embedding

$$e_c=\mathrm{E}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{d}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}(c)e\_c\; =\; \backslash mathrm\{Embedding\}(c)$$

which is transformed by a two-layer MLP

$$m=\mathrm{M}\mathrm{L}\mathrm{P}(e_c)m\; =\; \backslash mathrm\{MLP\}(e\_c)$$

The modulation vector is reshaped into

$$m\in {\mathbb{R}}^{32\times r}m\; \backslash in\; \backslash mathbb\{R\}^\{32\; \backslash times\; r\}$$

where

• 32 = transformer layers
• r = PrefixMemory rank (64)

For transformer layer $l$, query states are projected into the low-rank PrefixMemory space

$$Q_l=X{W}_l^{\mathrm{d}\mathrm{o}\mathrm{w}\mathrm{n}}Q\_l\; =\; XW^\{\backslash mathrm\{down\}\}\_l$$

where

• $X$ is the hidden representation.
• $W^{\mathrm{down}} \in \mathbb{R}^{4096 \times 64}$.

The controller applies element-wise modulation

$$\varphi (Q_l)=\mathrm{E}\mathrm{L}\mathrm{U}\text{\hspace{0.17em}⁣}(Q_l\odot m_l)\backslash phi(Q\_l)\; =\; \backslash mathrm\{ELU\}\backslash !\backslash left(Q\_l\; \backslash odot\; m\_l\backslash right)$$

The external PrefixMemory bias becomes

$$B_l=\varphi (Q_l)W_l^{\mathrm{u}\mathrm{p}}B\_l\; =\; \backslash phi(Q\_l)W^\{\backslash mathrm\{up\}\}\_l$$

where

$$W^{\mathrm{u}\mathrm{p}}\in {\mathbb{R}}^{64\times 4096}W^\{\backslash mathrm\{up\}\}\; \backslash in\; \backslash mathbb\{R\}^\{64\; \backslash times\; 4096\}$$

Finally,

$$H_l^{\mathrm{o}\mathrm{u}\mathrm{t}}=H_l^{\mathrm{a}\mathrm{t}\mathrm{t}\mathrm{n}}+\alpha B_{l}H\_l^\{\backslash mathrm\{out\}\}\; =\; H\_l^\{\backslash mathrm\{attn\}\}\; +\; \backslash alpha\; B\_l$$

where

• $H_l^{\mathrm{attn}}$ is the original attention output.
• $\alpha$ is a learnable scaling parameter.

---

Training Objective

Only the PrefixMemory controller parameters are optimized.

The frozen Llama backbone receives the modified hidden representations.

Training minimizes the standard causal language modeling objective

using masked prompt tokens.

Optimization

• Optimizer: AdamW
• Learning rate: $$2 \times 10^{-4}$$
• Batch size: 8
• Epochs: 5

---

Training Dataset

Training uses a balanced subset derived from the EFCAMDAT learner corpus.

Original dataset:

https://ef-lab.mmll.cam.ac.uk/EFCAMDAT.html

Due to the EFCAMDAT data usage policy, the processed training subset cannot be redistributed.

Dataset Columns

Column	Description
topic_title	Writing prompt
clean_text	Human learner response
cefr	Target CEFR level

Distribution

Balanced across all CEFR levels.

CEFR	Samples
A1	1,000
A2	1,000
B1	1,000
B2	1,000
C1	1,000
C2	1,000

Total:

6,000 samples

---

Training Performance

Best checkpoint:

Epoch 2

Validation Perplexity:

7.08

[
  {
    "epoch": 1,
    "train_loss": 2.1147,
    "val_loss": 2.0288,
    "val_ppl": 7.61
  },
  {
    "epoch": 2,
    "train_loss": 1.8625,
    "val_loss": 1.9573,
    "val_ppl": 7.08
  },
  {
    "epoch": 3,
    "train_loss": 1.5514,
    "val_loss": 2.0191,
    "val_ppl": 7.53
  },
  {
    "epoch": 4,
    "train_loss": 1.1617,
    "val_loss": 2.2208,
    "val_ppl": 9.21
  },
  {
    "epoch": 5,
    "train_loss": 0.7790,
    "val_loss": 2.5407,
    "val_ppl": 12.69
  }
]

---

In-Domain Evaluation

Benchmark

Balanced benchmark constructed from held-out EFCAMDAT prompts.

• Total prompts: 702
• Source: EFCAMDAT
• Prompt-only evaluation
• Topic distribution balanced across CEFR levels

Dataset columns

Column
topic_id
topic_title
cefr

Distribution

CEFR	Samples
A1	117
A2	117
B1	117
B2	117
C1	117
C2	117

---

Results

PMT Evaluation Results

============================================================
📊 FINAL COMPILED MACRO-STATISTICS (PREFIXMEMORY-TUNING BASELINE)
============================================================

Total Processed       : 702
Strict Accuracy       : 56.27%
Adjacent Accuracy     : 73.36%
Mean Abs Error (MAE)  : 0.9103
Avg MDD Score         : 1.77
Avg Reading Ease      : 83.51
Avg Sentence Drift    : 2.30 levels
Avg Perplexity        : (Computed externally)

============================================================
📈 PERFORMANCE BREAKDOWN BY CEFR TARGET LEVEL
============================================================

             strict_accuracy  avg_mdd  avg_flesch  avg_drift
target_cefr
-------------------------------------------------------------
A1                     74.36     1.35       89.07       1.01
A2                     69.23     1.67       87.02       1.73
B1                     60.68     1.85       83.37       2.07
B2                     50.43     1.87       82.56       2.94
C1                     45.30     1.93       78.48       2.86
C2                     37.61     1.95       80.55       3.20

============================================================
📝 DETAILED CLASSIFICATION REPORT
============================================================

              precision    recall  f1-score   support

A1               0.56       0.74      0.64       117
A2               0.57       0.69      0.62       117
B1               0.46       0.61      0.53       117
B2               0.48       0.50      0.49       117
C1               0.73       0.45      0.56       117
C2               0.77       0.38      0.51       117

-------------------------------------------------------------

Accuracy                                 0.56       702
Macro Avg         0.60       0.56      0.56       702
Weighted Avg      0.60       0.56      0.56       702

PMT Confusion Matrix

Figure 2. Confusion Matrix of PrefixMemory Tuning on In-Domain Evaluation Dataset (Inference).

Improvement over Base Llama

• +31.63 percentage points Strict Accuracy
• +10.17 percentage points Adjacent Accuracy
• Lower MAE
• Lower syntactic deviation
• Much stronger readability control

Base LLM Evaluation Results

Baseline LLM Evaluation Results

============================================================
📊 FINAL COMPILED MACRO-STATISTICS (BASE LLM)
============================================================

Total Processed       : 690
Strict Accuracy       : 24.64%
Adjacent Accuracy     : 63.19%
Mean Abs Error (MAE)  : 1.3275
Avg MDD Score         : 2.00
Avg Reading Ease      : 49.47
Avg Sentence Drift    : 3.04 levels
Avg Perplexity        : 4.14

============================================================
📈 PERFORMANCE BREAKDOWN BY CEFR TARGET LEVEL
============================================================

             strict_accuracy  avg_mdd  avg_flesch  avg_drift  avg_perplexity
target_cefr
-------------------------------------------------------------------------
A1                     16.24     1.73       82.56       3.25            5.16
A2                     23.08     1.64       85.44       3.27            5.12
B1                     17.39     2.03       57.80       3.11            3.86
B2                     41.74     2.15       34.18       2.97            3.32
C1                     33.63     2.24       18.95       2.94            3.59
C2                     15.93     2.25       15.54       2.69            3.74

============================================================
📝 DETAILED CLASSIFICATION REPORT
============================================================

              precision    recall  f1-score   support

A1               0.28       0.16      0.20       117
A2               0.38       0.23      0.29       117
B1               0.21       0.17      0.19       115
B2               0.22       0.42      0.29       115
C1               0.24       0.34      0.28       113
C2               0.23       0.16      0.19       113

-------------------------------------------------------------

Accuracy                                 0.25       690
Macro Avg         0.26       0.25      0.24       690
Weighted Avg      0.26       0.25      0.24       690

Base LLM Confusion Matrix

Figure 3. Confusion Matrix of Base LLM on In-Domain Evaluation Dataset (Inference).

---

Out-of-Domain Evaluation

Benchmark

MohammadKhosravi/ielts-cefr-benchmark-prompts

A manually curated benchmark containing IELTS-style prompts designed to evaluate prompt generalization.

Dataset:

https://huggingface.co/datasets/MohammadKhosravi/ielts-cefr-benchmark-prompts

Columns

Column
prompt
cefr

Distribution

Balanced

CEFR	Samples
A1	50
A2	50
B1	50
B2	50
C1	50
C2	50

Total:

300 prompts

---

Results

Metric	PrefixMemory	Base Llama
Strict Accuracy	44.00%	18.33%
Adjacent Accuracy	71.67%	54.33%
MAE	0.9933	1.5733
Average MDD	1.96	2.07
Reading Ease	75.43	45.18
Sentence Drift	2.62	2.53

Improvement over Base Llama

• +25.67 percentage points Strict Accuracy
• +17.34 percentage points Adjacent Accuracy
• Lower MAE
• Significantly improved readability control
• Strong transfer to an unseen prompt style

---

Intended Use

This model is intended for research on

• controllable text generation
• CEFR-level writing generation
• educational NLP
• readability control
• parameter-efficient fine-tuning
• PrefixMemory-Tuning

---

Citation

If you use this model, please cite:

• PrefixMemory-Tuning (arXiv:2506.13674)
• Meta Llama-3.1-8B-Instruct
• EFCAMDAT Corpus
• This repository