Hugging Face's logo

TevunahAi
/
Phi-3-medium-128k-instruct-TevunahAi-GPTQ

Safetensors
phi3
custom_code
4-bit precision
gptq
Model card Files
xet
 Community
Phi-3-medium-128k-instruct-TevunahAi-GPTQ / README.md
rockylynnstein's picture
rockylynnstein
Update README.md
e18b8b5 verified
preview code
|
Raw
History Blame Contribute Delete
10.4 kB
metadata 
license: mit
base_model:
  - microsoft/Phi-3-medium-128k-instruct
base_model_relation: quantized

Microsoft Phi-3-medium-128k-instruct - TevunahAi Ultra-Hybrid GPTQ with EoRA

Model Details

Property Value
Base Model microsoft/Phi-3-medium-128k-instruct
Architecture Phi-3 Dense Decoder-only Transformer
Parameters 14B
Context Length 128,000 tokens
Training Data 4.8T tokens (10% multilingual)
Quantization TevunahAi Ultra-Hybrid GPTQ + EoRA
Original Size ~28 GB (BF16)
Quantized Size ~8-10 GB
Compression ~65-70% reduction
License MIT

Architecture Breakdown

Microsoft Phi-3-medium-128k-instruct is a long-context language model trained on 4.8 trillion tokens:

Layer Composition (40 total layers)

  • 40 Transformer Decoder Layers: Dense attention architecture
  • 40 Attention Heads: GQA with 10 KV heads (4:1 ratio)
  • Hidden Size: 5,120
  • Intermediate Size: 17,920
  • Vocab Size: 32,064
  • SiLU (Swish) Activation: Smooth activation function
  • RMSNorm + RoPE: theta=10000 with SU scaling for 128K context
  • Fused Projections: qkv_proj and gate_up_proj for efficiency

Why This Matters

  • 128K token context: Process extremely long documents, entire codebases, books
  • 4.8T training tokens: Extensive pretraining for broad capabilities
  • 10% multilingual: Support for multiple languages
  • SFT + DPO alignment: Supervised fine-tuning with preference optimization
  • MIT License: Permissive open source license

Quantization Strategy

TevunahAi Ultra-Hybrid Mixed-Precision with EoRA Error Recovery

This quantization uses EoRA (Error-optimized Low-Rank Adaptation) - NVIDIA's technique for recovering quantization error through learned low-rank adapters applied during the quantization process.

Component Precision EoRA Rank Rationale
Attention qkv_proj (all 40 layers) INT8 128 Critical for long-context quality
Attention o_proj (all 40 layers) INT8 128 Output projection quality
MLP gate_up_proj (layers 0-31) INT4 128 Maximum compression in early/middle layers
MLP down_proj (layers 0-31) INT4 128 Compression with error recovery
MLP gate_up_proj (layers 32-39) INT8 128 Higher precision near output
MLP down_proj (layers 32-39) INT8 128 Output quality critical
Embeddings FP16 - Preserved for token accuracy
LM Head FP16 - Preserved for output quality

Why INT8 Attention Everywhere?

For a 128K context model, attention quality is paramount:

  • Long-range dependencies: Attention must work accurately over 128K tokens
  • INT8 preserves precision: Critical for maintaining context coherence
  • EoRA-128 recovery: Additional error correction on all attention layers

Why Tiered MLP Precision?

  • Layers 0-31 (INT4): Early/middle layers are more compressible
  • Layers 32-39 (INT8): Final 8 layers directly affect output quality
  • EoRA-128 on all: Error recovery maintains quality across compression levels

Calibration

  • 1,500 samples (6x industry standard of 256)
  • 2,048 sequence length (optimized for 32GB VRAM)
  • Diverse datasets: Orca-Math, Code-Feedback, UltraChat, SlimOrca
  • Long-context focus: Calibration designed for extended context use

Performance Benchmarks

Qualitative Tests (7/7 passed)

Test Result Details
Basic Inference ✅ PASS Coherent self-introduction
Math Reasoning ✅ PASS Train speed problem - 240 miles correct
Code Generation ✅ PASS Sieve of Eratosthenes - 6/6 elements
Long Context ✅ PASS Multi-fact retrieval (2/3 correct)
Multilingual ✅ PASS French + Spanish translations
Logic Puzzle ✅ PASS Box labeling - correct reasoning
Summarization ✅ PASS AI impact - 5/5 topics covered

Quantized Model Benchmarks (lm-eval-harness, 0-shot)

Task Score Metric Stderr
HellaSwag 71.00% acc_norm ±4.56%
Winogrande 76.00% acc ±4.29%
ARC-Challenge 48.00% acc_norm ±5.02%
TruthfulQA MC2 50.93% acc ±4.39%

Quick test with 100 samples per task. Full benchmark recommended for production validation.

Inference Performance

Metric Value
VRAM Usage 10.22 GB
Generation Speed 10-23 tok/s
Tests Passed 7/7

Note: Speed varies based on response length. No flash attention used during testing.

Usage

GPTQModel (Recommended) 

from gptqmodel import GPTQModel
from transformers import AutoTokenizer

model = GPTQModel.from_quantized(
    "TevunahAi/Phi-3-medium-128k-instruct-TevunahAi-GPTQ",
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "TevunahAi/Phi-3-medium-128k-instruct-TevunahAi-GPTQ",
    trust_remote_code=True
)

messages = [
    {"role": "user", "content": "Explain the theory of relativity in simple terms."},
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=500,
    temperature=0.0,
    do_sample=False,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Transformers 

from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

model = AutoModelForCausalLM.from_pretrained(
    "TevunahAi/Phi-3-medium-128k-instruct-TevunahAi-GPTQ",
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "TevunahAi/Phi-3-medium-128k-instruct-TevunahAi-GPTQ",
    trust_remote_code=True
)

pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
)

messages = [
    {"role": "user", "content": "Write a Python function to calculate fibonacci numbers."},
]

output = pipe(messages, max_new_tokens=500, return_full_text=False)
print(output[0]['generated_text'])

vLLM (Production) 

pip install -U vllm

vllm serve TevunahAi/Phi-3-medium-128k-instruct-TevunahAi-GPTQ \
    --max-num-seqs 4 \
    --max-model-len 32768 \
    --trust-remote-code

Chat Format 

<|user|>
Your question here<|end|>
<|assistant|>

Installation 

pip install gptqmodel transformers>=4.48

Long-Context Capabilities

Phi-3-medium-128k excels at tasks requiring extended context:

Document Analysis 

messages = [
    {"role": "user", "content": f"""
Here is a research paper:

{very_long_document}

Please summarize the key findings and methodology.
"""},
]

Multi-Document QA 

messages = [
    {"role": "user", "content": f"""
Document 1: {doc1}
Document 2: {doc2}
Document 3: {doc3}

Compare and contrast the approaches described in these documents.
"""},
]

Codebase Analysis 

messages = [
    {"role": "user", "content": f"""
Here is the source code for a project:

{entire_codebase}

Explain the architecture and identify potential improvements.
"""},
]

Memory Requirements

Inference (quantized model)

Context Length VRAM Required
Short (4K) 10-11 GB
Medium (16K) 12-14 GB
Long (32K) 16-20 GB
Extended (64K) 24-32 GB
Full (128K) 40+ GB

Tested on: RTX 5000 Ada (32GB) - 10.22 GB active VRAM during inference

Quantization (reproduction)

  • GPU: RTX 5000 Ada 32GB
  • CPU: Dual Xeon (224 cores for Hessian computation)
  • RAM: 64GB+ recommended
  • Method: GPU forward passes + CPU Hessian

Quantization Details

Specification Value
Method GPTQ + Ultra-Hybrid + EoRA
Quantizer GPTQModel
EoRA Rank 128 (all layers)
Calibration Samples 1,500 (6x industry standard)
Sequence Length 2,048 tokens
Group Size 128
desc_act False
sym True (symmetric quantization)
Bits (default) 4
Layer Rules 160 custom precision rules

Use Cases

Ideal for:

  • 📄 Long document analysis (128K context)
  • 📚 Book summarization and QA
  • 💻 Codebase understanding
  • 🔬 Research paper analysis
  • 🌍 Multilingual tasks (10% multilingual training)
  • 🧮 Mathematical reasoning
  • 🔧 Resource-constrained deployment (28GB → 10GB)

Technical Specifications

Specification Value
Model Family Microsoft Phi-3
Variant medium-128k-instruct
Total Parameters 14B
Total Layers 40
Hidden Size 5,120
Intermediate Size 17,920
Attention Heads 40
KV Heads 10 (GQA)
Activation SiLU (Swish)
Normalization RMSNorm
Position Encoding RoPE (theta=10000, SU scaling)
Context Length 128,000
Vocab Size 32,064
Training Tokens 4.8T
Multilingual 10%
Post-training SFT + DPO

Expected Performance

Based on the precision strategy:

Task Type Expected Retention
Long-context QA 97-99%
Math Reasoning 97-99%
Code Generation 96-98%
Multilingual 96-98%
General Chat 98-99%

Acknowledgments

  • Microsoft Research for developing the Phi-3 model family
  • NVIDIA for the EoRA (Error-optimized Low-Rank Adaptation) technique used in this quantization
  • GPTQModel team for the excellent quantization framework

License

MIT License - Permissive open source license allowing commercial use, modification, and distribution.

Citation 

@software{phi3_medium_128k_gptq_2025,
  title = {Microsoft Phi-3-medium-128k-instruct - TevunahAi Ultra-Hybrid GPTQ with EoRA},
  author = {TevunahAi},
  year = {2025},
  note = {Ultra-Hybrid GPTQ with EoRA-128 for long-context quality retention},
  url = {https://huggingface.co/TevunahAi/Phi-3-medium-128k-instruct-TevunahAi-GPTQ}
}

@misc{phi3_2024,
  title = {Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone},
  author = {Microsoft},
  year = {2024},
  url = {https://huggingface.co/microsoft/Phi-3-medium-128k-instruct}
}

https://huggingface.co/TevunahAi