README.md

---
license: other
language:
- en
- es
- fr
- de
- it
- pt
- ru
- ar
- hi
- ko
- zh
library_name: transformers
base_model:
- arcee-ai/Trinity-Nano-Base-Pre-Anneal
license_link: LICENSE
license_name: openmdw-1.1
---
<div align="center">
  <picture>
    <img
      src="https://cdn-uploads.huggingface.co/production/uploads/6435718aaaef013d1aec3b8b/i-v1KyAMOW_mgVGeic9WJ.png"
      alt="Arcee Trinity Mini"
      style="max-width: 100%; height: auto;"
    >
  </picture>
</div>

# Trinity Nano Base

Trinity Nano is an Arcee AI 6B MoE model with 1B active parameters. It is the small-sized model in our new Trinity family, a series of open-weight models for enterprise and tinkerers alike.

This base model *pre* fine tuning, and so is not suitable for chatting, and should be trained for your specific domain before use.

Trinity Nano is trained on 10T tokens gathered and curated through a key partnership with [Datology](https://www.datologyai.com/), building upon the excellent dataset we used on [AFM-4.5B](https://huggingface.co/arcee-ai/AFM-4.5B) with additional math and code.

Training was performed on a cluster of 512 H200 GPUs powered by [Prime Intellect](https://www.primeintellect.ai/) using HSDP parallelism.

More details, including key architecture decisions, can be found on our blog [here](https://www.arcee.ai/blog/the-trinity-manifesto)

***

## Model Details

* **Model Architecture:** AfmoeForCausalLM
* **Parameters:** 6B, 1B active
* **Experts:** 128 total, 8 active, 1 shared
* **Context length:** 128k
* **Training Tokens:** 10T
* **License:** [OpenMDW-1.1](https://huggingface.co/arcee-ai/Trinity-Nano-Base#license)

## Benchmarks

### 🔢 Math & Reasoning
| Benchmark | Score |
|-----------|-------|
| GSM8K | 58.4% |
| Minerva Math 500 | 36.0% |
| DROP (0-shot) | 4.5% |
| DROP (5-shot) | 63.6% |

### 💻 Code Generation
| Benchmark | Pass@1 | Pass@10 |
|-----------|--------|---------|
| HumanEval (3-shot, bpb) | 36.3% (bpb) | - |
| HumanEval+ (temp 0.8) | 31.7% | - |
| MBPP+ | 44.7% | - |

### 🧠 Knowledge & Reasoning
| Benchmark | 5-shot | 0-shot |
|-----------|---------|--------|
| ARC-Challenge | 84.0% | 78.2% |
| ARC-Easy | 94.8% | 91.2% |
| CommonsenseQA | 74.9% | 62.7% |
| OpenBookQA | 82.2% | 75.2% |
| WinoGrande | 72.8% | 68.0% |
| MMLU | 67.7% | 64.2% |
| MMLU Pro | 35.8% | 27.7% |
| AGI Eval (English) | 51.8% | - |
| BBH (CoT) | 50.4% | 7.6% |

### 📘 Understanding & QA
| Benchmark | Score |
|-----------|-------|
| BoolQ (5-shot) | 84.3% |
| HellaSwag (5-shot) | 77.4% |
| PIQA (5-shot) | 82.2% |
| SciQ (5-shot) | 93.2% |
| Social IQA (5-shot) | 73.0% |

<div align="center">
  <picture>
      <img src="https://cdn-uploads.huggingface.co/production/uploads/6435718aaaef013d1aec3b8b/sSVjGNHfrJKmQ6w8I18ek.png" style="background-color:ghostwhite;padding:5px;" width="17%" alt="Powered by Datology">
  </picture>
</div>

## License

Trinity-Mini-Base is released under the OpenMDW-1.1 license.