Microsoft Phi-3-medium-128k-instruct - TevunahAi Ultra-Hybrid GPTQ with EoRA

Model Details

Property	Value
Base Model	microsoft/Phi-3-medium-128k-instruct
Architecture	Phi-3 Dense Decoder-only Transformer
Parameters	14B
Context Length	128,000 tokens
Training Data	4.8T tokens (10% multilingual)
Quantization	TevunahAi Ultra-Hybrid GPTQ + EoRA
Original Size	~28 GB (BF16)
Quantized Size	~8-10 GB
Compression	~65-70% reduction
License	MIT

Architecture Breakdown

Microsoft Phi-3-medium-128k-instruct is a long-context language model trained on 4.8 trillion tokens:

Layer Composition (40 total layers)

40 Transformer Decoder Layers: Dense attention architecture
40 Attention Heads: GQA with 10 KV heads (4:1 ratio)
Hidden Size: 5,120
Intermediate Size: 17,920
Vocab Size: 32,064
SiLU (Swish) Activation: Smooth activation function
RMSNorm + RoPE: theta=10000 with SU scaling for 128K context
Fused Projections: qkv_proj and gate_up_proj for efficiency

Why This Matters

128K token context: Process extremely long documents, entire codebases, books
4.8T training tokens: Extensive pretraining for broad capabilities
10% multilingual: Support for multiple languages
SFT + DPO alignment: Supervised fine-tuning with preference optimization
MIT License: Permissive open source license

Quantization Strategy

TevunahAi Ultra-Hybrid Mixed-Precision with EoRA Error Recovery

This quantization uses EoRA (Error-optimized Low-Rank Adaptation) - NVIDIA's technique for recovering quantization error through learned low-rank adapters applied during the quantization process.

Component	Precision	EoRA Rank	Rationale
Attention qkv_proj (all 40 layers)	INT8	128	Critical for long-context quality
Attention o_proj (all 40 layers)	INT8	128	Output projection quality
MLP gate_up_proj (layers 0-31)	INT4	128	Maximum compression in early/middle layers
MLP down_proj (layers 0-31)	INT4	128	Compression with error recovery
MLP gate_up_proj (layers 32-39)	INT8	128	Higher precision near output
MLP down_proj (layers 32-39)	INT8	128	Output quality critical
Embeddings	FP16	-	Preserved for token accuracy
LM Head	FP16	-	Preserved for output quality

Why INT8 Attention Everywhere?

For a 128K context model, attention quality is paramount:

Long-range dependencies: Attention must work accurately over 128K tokens
INT8 preserves precision: Critical for maintaining context coherence
EoRA-128 recovery: Additional error correction on all attention layers

Why Tiered MLP Precision?

Layers 0-31 (INT4): Early/middle layers are more compressible
Layers 32-39 (INT8): Final 8 layers directly affect output quality
EoRA-128 on all: Error recovery maintains quality across compression levels

Calibration

1,500 samples (6x industry standard of 256)
2,048 sequence length (optimized for 32GB VRAM)
Diverse datasets: Orca-Math, Code-Feedback, UltraChat, SlimOrca
Long-context focus: Calibration designed for extended context use

Performance Benchmarks

Qualitative Tests (7/7 passed)

Test	Result	Details
Basic Inference	✅ PASS	Coherent self-introduction
Math Reasoning	✅ PASS	Train speed problem - 240 miles correct
Code Generation	✅ PASS	Sieve of Eratosthenes - 6/6 elements
Long Context	✅ PASS	Multi-fact retrieval (2/3 correct)
Multilingual	✅ PASS	French + Spanish translations
Logic Puzzle	✅ PASS	Box labeling - correct reasoning
Summarization	✅ PASS	AI impact - 5/5 topics covered

Quantized Model Benchmarks (lm-eval-harness, 0-shot)

Task	Score	Metric	Stderr
HellaSwag	71.00%	acc_norm	±4.56%
Winogrande	76.00%	acc	±4.29%
ARC-Challenge	48.00%	acc_norm	±5.02%
TruthfulQA MC2	50.93%	acc	±4.39%

Quick test with 100 samples per task. Full benchmark recommended for production validation.

Inference Performance

Metric	Value
VRAM Usage	10.22 GB
Generation Speed	10-23 tok/s
Tests Passed	7/7

Note: Speed varies based on response length. No flash attention used during testing.

Usage

GPTQModel (Recommended)

from gptqmodel import GPTQModel
from transformers import AutoTokenizer

model = GPTQModel.from_quantized(
    "TevunahAi/Phi-3-medium-128k-instruct-TevunahAi-GPTQ",
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "TevunahAi/Phi-3-medium-128k-instruct-TevunahAi-GPTQ",
    trust_remote_code=True
)

messages = [
    {"role": "user", "content": "Explain the theory of relativity in simple terms."},
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=500,
    temperature=0.0,
    do_sample=False,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

model = AutoModelForCausalLM.from_pretrained(
    "TevunahAi/Phi-3-medium-128k-instruct-TevunahAi-GPTQ",
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "TevunahAi/Phi-3-medium-128k-instruct-TevunahAi-GPTQ",
    trust_remote_code=True
)

pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
)

messages = [
    {"role": "user", "content": "Write a Python function to calculate fibonacci numbers."},
]

output = pipe(messages, max_new_tokens=500, return_full_text=False)
print(output[0]['generated_text'])

vLLM (Production)

pip install -U vllm

vllm serve TevunahAi/Phi-3-medium-128k-instruct-TevunahAi-GPTQ \
    --max-num-seqs 4 \
    --max-model-len 32768 \
    --trust-remote-code

Chat Format

<|user|>
Your question here<|end|>
<|assistant|>

Installation

pip install gptqmodel transformers>=4.48

Long-Context Capabilities

Phi-3-medium-128k excels at tasks requiring extended context:

Document Analysis

messages = [
    {"role": "user", "content": f"""
Here is a research paper:

{very_long_document}

Please summarize the key findings and methodology.
"""},
]

Multi-Document QA

messages = [
    {"role": "user", "content": f"""
Document 1: {doc1}
Document 2: {doc2}
Document 3: {doc3}

Compare and contrast the approaches described in these documents.
"""},
]

Codebase Analysis

messages = [
    {"role": "user", "content": f"""
Here is the source code for a project:

{entire_codebase}

Explain the architecture and identify potential improvements.
"""},
]

Memory Requirements

Inference (quantized model)

Context Length	VRAM Required
Short (4K)	10-11 GB
Medium (16K)	12-14 GB
Long (32K)	16-20 GB
Extended (64K)	24-32 GB
Full (128K)	40+ GB

Tested on: RTX 5000 Ada (32GB) - 10.22 GB active VRAM during inference

Quantization (reproduction)

GPU: RTX 5000 Ada 32GB
CPU: Dual Xeon (224 cores for Hessian computation)
RAM: 64GB+ recommended
Method: GPU forward passes + CPU Hessian

Quantization Details

Specification	Value
Method	GPTQ + Ultra-Hybrid + EoRA
Quantizer	GPTQModel
EoRA Rank	128 (all layers)
Calibration Samples	1,500 (6x industry standard)
Sequence Length	2,048 tokens
Group Size	128
desc_act	False
sym	True (symmetric quantization)
Bits (default)	4
Layer Rules	160 custom precision rules

Use Cases

Ideal for:

📄 Long document analysis (128K context)
📚 Book summarization and QA
💻 Codebase understanding
🔬 Research paper analysis
🌍 Multilingual tasks (10% multilingual training)
🧮 Mathematical reasoning
🔧 Resource-constrained deployment (28GB → 10GB)

Technical Specifications

Specification	Value
Model Family	Microsoft Phi-3
Variant	medium-128k-instruct
Total Parameters	14B
Total Layers	40
Hidden Size	5,120
Intermediate Size	17,920
Attention Heads	40
KV Heads	10 (GQA)
Activation	SiLU (Swish)
Normalization	RMSNorm
Position Encoding	RoPE (theta=10000, SU scaling)
Context Length	128,000
Vocab Size	32,064
Training Tokens	4.8T
Multilingual	10%
Post-training	SFT + DPO

Expected Performance

Based on the precision strategy:

Task Type	Expected Retention
Long-context QA	97-99%
Math Reasoning	97-99%
Code Generation	96-98%
Multilingual	96-98%
General Chat	98-99%

Acknowledgments

Microsoft Research for developing the Phi-3 model family
NVIDIA for the EoRA (Error-optimized Low-Rank Adaptation) technique used in this quantization
GPTQModel team for the excellent quantization framework

License

MIT License - Permissive open source license allowing commercial use, modification, and distribution.

Citation

@software{phi3_medium_128k_gptq_2025,
  title = {Microsoft Phi-3-medium-128k-instruct - TevunahAi Ultra-Hybrid GPTQ with EoRA},
  author = {TevunahAi},
  year = {2025},
  note = {Ultra-Hybrid GPTQ with EoRA-128 for long-context quality retention},
  url = {https://huggingface.co/TevunahAi/Phi-3-medium-128k-instruct-TevunahAi-GPTQ}
}

@misc{phi3_2024,
  title = {Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone},
  author = {Microsoft},
  year = {2024},
  url = {https://huggingface.co/microsoft/Phi-3-medium-128k-instruct}
}

https://huggingface.co/TevunahAi