README.md

---
license: mit
base_model:
- microsoft/Phi-3-medium-128k-instruct
base_model_relation: quantized
---
# Microsoft Phi-3-medium-128k-instruct - TevunahAi Ultra-Hybrid GPTQ with EoRA

## Model Details

| Property | Value |
|----------|-------|
| **Base Model** | microsoft/Phi-3-medium-128k-instruct |
| **Architecture** | Phi-3 Dense Decoder-only Transformer |
| **Parameters** | 14B |
| **Context Length** | 128,000 tokens |
| **Training Data** | 4.8T tokens (10% multilingual) |
| **Quantization** | TevunahAi Ultra-Hybrid GPTQ + EoRA |
| **Original Size** | ~28 GB (BF16) |
| **Quantized Size** | ~8-10 GB |
| **Compression** | ~65-70% reduction |
| **License** | MIT |

## Architecture Breakdown

Microsoft Phi-3-medium-128k-instruct is a long-context language model trained on 4.8 trillion tokens:

### Layer Composition (40 total layers)

- **40 Transformer Decoder Layers**: Dense attention architecture
- **40 Attention Heads**: GQA with 10 KV heads (4:1 ratio)
- **Hidden Size**: 5,120
- **Intermediate Size**: 17,920
- **Vocab Size**: 32,064
- **SiLU (Swish) Activation**: Smooth activation function
- **RMSNorm + RoPE**: theta=10000 with SU scaling for 128K context
- **Fused Projections**: qkv_proj and gate_up_proj for efficiency

### Why This Matters

- **128K token context**: Process extremely long documents, entire codebases, books
- **4.8T training tokens**: Extensive pretraining for broad capabilities
- **10% multilingual**: Support for multiple languages
- **SFT + DPO alignment**: Supervised fine-tuning with preference optimization
- **MIT License**: Permissive open source license

## Quantization Strategy

### TevunahAi Ultra-Hybrid Mixed-Precision with EoRA Error Recovery

This quantization uses **EoRA (Error-optimized Low-Rank Adaptation)** - NVIDIA's technique for recovering quantization error through learned low-rank adapters applied during the quantization process.

| Component | Precision | EoRA Rank | Rationale |
|-----------|-----------|-----------|-----------|
| Attention qkv_proj (all 40 layers) | INT8 | 128 | Critical for long-context quality |
| Attention o_proj (all 40 layers) | INT8 | 128 | Output projection quality |
| MLP gate_up_proj (layers 0-31) | INT4 | 128 | Maximum compression in early/middle layers |
| MLP down_proj (layers 0-31) | INT4 | 128 | Compression with error recovery |
| MLP gate_up_proj (layers 32-39) | INT8 | 128 | Higher precision near output |
| MLP down_proj (layers 32-39) | INT8 | 128 | Output quality critical |
| Embeddings | FP16 | - | Preserved for token accuracy |
| LM Head | FP16 | - | Preserved for output quality |

### Why INT8 Attention Everywhere?

For a 128K context model, attention quality is paramount:
- **Long-range dependencies**: Attention must work accurately over 128K tokens
- **INT8 preserves precision**: Critical for maintaining context coherence
- **EoRA-128 recovery**: Additional error correction on all attention layers

### Why Tiered MLP Precision?

- **Layers 0-31 (INT4)**: Early/middle layers are more compressible
- **Layers 32-39 (INT8)**: Final 8 layers directly affect output quality
- **EoRA-128 on all**: Error recovery maintains quality across compression levels

### Calibration

- **1,500 samples** (6x industry standard of 256)
- **2,048 sequence length** (optimized for 32GB VRAM)
- **Diverse datasets**: Orca-Math, Code-Feedback, UltraChat, SlimOrca
- **Long-context focus**: Calibration designed for extended context use

## Performance Benchmarks

### Qualitative Tests (7/7 passed)

| Test | Result | Details |
|------|--------|---------|
| Basic Inference | ✅ PASS | Coherent self-introduction |
| Math Reasoning | ✅ PASS | Train speed problem - 240 miles correct |
| Code Generation | ✅ PASS | Sieve of Eratosthenes - 6/6 elements |
| Long Context | ✅ PASS | Multi-fact retrieval (2/3 correct) |
| Multilingual | ✅ PASS | French + Spanish translations |
| Logic Puzzle | ✅ PASS | Box labeling - correct reasoning |
| Summarization | ✅ PASS | AI impact - 5/5 topics covered |

### Quantized Model Benchmarks (lm-eval-harness, 0-shot)

| Task | Score | Metric | Stderr |
|------|-------|--------|--------|
| HellaSwag | **71.00%** | acc_norm | ±4.56% |
| Winogrande | **76.00%** | acc | ±4.29% |
| ARC-Challenge | **48.00%** | acc_norm | ±5.02% |
| TruthfulQA MC2 | **50.93%** | acc | ±4.39% |

*Quick test with 100 samples per task. Full benchmark recommended for production validation.*

### Inference Performance

| Metric | Value |
|--------|-------|
| **VRAM Usage** | 10.22 GB |
| **Generation Speed** | 10-23 tok/s |
| **Tests Passed** | 7/7 |

*Note: Speed varies based on response length. No flash attention used during testing.*

## Usage

### GPTQModel (Recommended)

```python
from gptqmodel import GPTQModel
from transformers import AutoTokenizer

model = GPTQModel.from_quantized(
    "TevunahAi/Phi-3-medium-128k-instruct-TevunahAi-GPTQ",
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "TevunahAi/Phi-3-medium-128k-instruct-TevunahAi-GPTQ",
    trust_remote_code=True
)

messages = [
    {"role": "user", "content": "Explain the theory of relativity in simple terms."},
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=500,
    temperature=0.0,
    do_sample=False,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### Transformers

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

model = AutoModelForCausalLM.from_pretrained(
    "TevunahAi/Phi-3-medium-128k-instruct-TevunahAi-GPTQ",
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "TevunahAi/Phi-3-medium-128k-instruct-TevunahAi-GPTQ",
    trust_remote_code=True
)

pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
)

messages = [
    {"role": "user", "content": "Write a Python function to calculate fibonacci numbers."},
]

output = pipe(messages, max_new_tokens=500, return_full_text=False)
print(output[0]['generated_text'])
```

### vLLM (Production)

```bash
pip install -U vllm

vllm serve TevunahAi/Phi-3-medium-128k-instruct-TevunahAi-GPTQ \
    --max-num-seqs 4 \
    --max-model-len 32768 \
    --trust-remote-code
```

### Chat Format

```
<|user|>
Your question here<|end|>
<|assistant|>
```

### Installation

```bash
pip install gptqmodel transformers>=4.48
```

## Long-Context Capabilities

Phi-3-medium-128k excels at tasks requiring extended context:

### Document Analysis
```python
messages = [
    {"role": "user", "content": f"""
Here is a research paper:

{very_long_document}

Please summarize the key findings and methodology.
"""},
]
```

### Multi-Document QA
```python
messages = [
    {"role": "user", "content": f"""
Document 1: {doc1}
Document 2: {doc2}
Document 3: {doc3}

Compare and contrast the approaches described in these documents.
"""},
]
```

### Codebase Analysis
```python
messages = [
    {"role": "user", "content": f"""
Here is the source code for a project:

{entire_codebase}

Explain the architecture and identify potential improvements.
"""},
]
```

## Memory Requirements

### Inference (quantized model)

| Context Length | VRAM Required |
|----------------|---------------|
| Short (4K) | **10-11 GB** |
| Medium (16K) | 12-14 GB |
| Long (32K) | 16-20 GB |
| Extended (64K) | 24-32 GB |
| Full (128K) | 40+ GB |

**Tested on**: RTX 5000 Ada (32GB) - 10.22 GB active VRAM during inference

### Quantization (reproduction)

- **GPU**: RTX 5000 Ada 32GB
- **CPU**: Dual Xeon (224 cores for Hessian computation)
- **RAM**: 64GB+ recommended
- **Method**: GPU forward passes + CPU Hessian

## Quantization Details

| Specification | Value |
|--------------|-------|
| Method | GPTQ + Ultra-Hybrid + EoRA |
| Quantizer | GPTQModel |
| EoRA Rank | 128 (all layers) |
| Calibration Samples | 1,500 (6x industry standard) |
| Sequence Length | 2,048 tokens |
| Group Size | 128 |
| desc_act | False |
| sym | True (symmetric quantization) |
| Bits (default) | 4 |
| Layer Rules | 160 custom precision rules |

## Use Cases

**Ideal for:**

- 📄 **Long document analysis** (128K context)
- 📚 **Book summarization and QA**
- 💻 **Codebase understanding**
- 🔬 **Research paper analysis**
- 🌍 **Multilingual tasks** (10% multilingual training)
- 🧮 **Mathematical reasoning**
- 🔧 **Resource-constrained deployment** (28GB → 10GB)

## Technical Specifications

| Specification | Value |
|--------------|-------|
| Model Family | Microsoft Phi-3 |
| Variant | medium-128k-instruct |
| Total Parameters | 14B |
| Total Layers | 40 |
| Hidden Size | 5,120 |
| Intermediate Size | 17,920 |
| Attention Heads | 40 |
| KV Heads | 10 (GQA) |
| Activation | SiLU (Swish) |
| Normalization | RMSNorm |
| Position Encoding | RoPE (theta=10000, SU scaling) |
| Context Length | 128,000 |
| Vocab Size | 32,064 |
| Training Tokens | 4.8T |
| Multilingual | 10% |
| Post-training | SFT + DPO |

## Expected Performance

Based on the precision strategy:

| Task Type | Expected Retention |
|-----------|-------------------|
| Long-context QA | 97-99% |
| Math Reasoning | 97-99% |
| Code Generation | 96-98% |
| Multilingual | 96-98% |
| General Chat | 98-99% |

## Acknowledgments

- **Microsoft Research** for developing the Phi-3 model family
- **NVIDIA** for the EoRA (Error-optimized Low-Rank Adaptation) technique used in this quantization
- **GPTQModel team** for the excellent quantization framework

## License

**MIT License** - Permissive open source license allowing commercial use, modification, and distribution.

## Citation

```bibtex
@software{phi3_medium_128k_gptq_2025,
  title = {Microsoft Phi-3-medium-128k-instruct - TevunahAi Ultra-Hybrid GPTQ with EoRA},
  author = {TevunahAi},
  year = {2025},
  note = {Ultra-Hybrid GPTQ with EoRA-128 for long-context quality retention},
  url = {https://huggingface.co/TevunahAi/Phi-3-medium-128k-instruct-TevunahAi-GPTQ}
}

@misc{phi3_2024,
  title = {Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone},
  author = {Microsoft},
  year = {2024},
  url = {https://huggingface.co/microsoft/Phi-3-medium-128k-instruct}
}
```



https://huggingface.co/TevunahAi