Instructions to use OrisTeam/Koliber-1.3-BeLooped with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use OrisTeam/Koliber-1.3-BeLooped with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="OrisTeam/Koliber-1.3-BeLooped", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("OrisTeam/Koliber-1.3-BeLooped", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use OrisTeam/Koliber-1.3-BeLooped with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "OrisTeam/Koliber-1.3-BeLooped" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "OrisTeam/Koliber-1.3-BeLooped", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/OrisTeam/Koliber-1.3-BeLooped
- SGLang
How to use OrisTeam/Koliber-1.3-BeLooped with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "OrisTeam/Koliber-1.3-BeLooped" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "OrisTeam/Koliber-1.3-BeLooped", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "OrisTeam/Koliber-1.3-BeLooped" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "OrisTeam/Koliber-1.3-BeLooped", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use OrisTeam/Koliber-1.3-BeLooped with Docker Model Runner:
docker model run hf.co/OrisTeam/Koliber-1.3-BeLooped
Koliber-1.3-BeLooped
Koliber-1.3-BeLooped is an experimental Polish decoder-only language model developed by OrisTeam as part of its architecture model zoo and private model research.
The model explores weight-shared recurrent Transformer layers, configurable computational depth (x1/x2/x3), and alternating sliding-window and full causal attention.
Koliber v1.3 is an architectural experiment rather than a direct continuation of the Koliber v1.2 design. Its purpose is to investigate alternative ways of allocating parameters and computational depth within the approximately 125M-parameter model class.
This release contains the base pretrained model, without instruction tuning or preference optimization.
Benchmarks
Koliber 1.3 at its default depth (x2) matches Koliber 1.2 on Polish grammar, and at x2 and x3 it has the lowest WikiText-2 perplexity of the Koliber models. Like every Koliber, it was trained on Polish text only, so on English reasoning and knowledge benchmarks it stays at the level of chance.
English results are given for reference only. No Koliber was trained on English text. The English reasoning and knowledge results are noise: they do not measure the models' ability, and the differences between them carry no meaning. English language modeling (WikiText-2, LAMBADA) does separate the models, but measures them on a language they were not trained on.
| Benchmark | Baseline (chance or constant) |
Koliber 1.3 x2 (default) |
Koliber 1.3 x1 |
Koliber 1.3 x3 |
Koliber 1.2 |
Koliber 1.1-Preview |
Koliber 1.0 |
|---|---|---|---|---|---|---|---|
| Polish | |||||||
| MultiBLiMP (Polish), accuracy | 50.0 | 95.7 | 94.7 | 95.7 | 95.6 | 94.0 | 88.2 |
| English: language modeling (reference) | |||||||
| WikiText-2, byte perplexity โ | โ | 2.905 | 2.988 | 2.902 | 3.033 | 3.740 | 3.757 |
| LAMBADA (OpenAI), accuracy | 0.3 | 9.5 | 8.7 | 9.1 | 7.0 | 7.4 | 8.0 |
| English: reasoning and knowledge (reference only: noise level, see note) | |||||||
| ARC-Easy, acc | 26.6 | 26.8 | 26.3 | 27.3 | 29.3 | 26.9 | 25.3 |
| ARC-Challenge, acc_norm | 26.5 | 22.2 | 22.3 | 22.3 | 23.2 | 22.1 | 22.3 |
| PIQA, acc_norm | 50.5 | 52.3 | 52.0 | 52.3 | 52.6 | 51.9 | 52.9 |
| WinoGrande, acc | 50.4 | 49.0 | 49.6 | 49.4 | 50.7 | 50.7 | 50.3 |
| WSC273, acc | 50.2 | 46.5 | 49.5 | 48.0 | 51.6 | 49.8 | 49.8 |
| Social IQa, acc | 33.6 | 33.3 | 33.5 | 33.1 | 34.1 | 34.2 | 34.1 |
| MathQA, acc_norm | 22.6 | 20.7 | 20.7 | 21.0 | 20.6 | 21.9 | 21.2 |
| MedMCQA, acc | 32.2 | 32.0 | 32.2 | 32.0 | 32.1 | 32.2 | 32.2 |
| PubMedQA, acc | 55.2 | 56.6 | 54.0 | 54.6 | 33.6 | 33.8 | 33.8 |
The highlighted value is the best of the Koliber models in each row; ties are all highlighted. Accuracies are in percent. On WikiText-2, lower is better. HellaSwag has not been run yet.
Polish. MultiBLiMP (Polish) asks the model to prefer the grammatical sentence of a minimal pair. Every Koliber is far above chance (50), and the scores grow from 88.2 (1.0) to 95.7 (1.3 x2 and x3). The standard error is about 0.4 points. The other Polish benchmarks of the suite are not yet available in its runner.
English (reference only). No Koliber had English text in its training mixture, so results near chance were expected on English benchmarks, and that is what they show. The reasoning and knowledge rows are noise and are listed for reference, not for comparison. The baseline is the higher of random choice and a constant answer.
- Across the nine English reasoning and knowledge benchmarks, only one result is above the baseline: Koliber 1.2 on ARC-Easy, at +2.7 points (a probable difference, not a certain one).
- Several results are below the baseline:
- on ARC-Challenge, every model is about 4 points under a constant answer;
- on PubMedQA, Koliber 1.0-1.2 nearly always give an answer other than the most common one, about 21 points under;
- on WSC273, Koliber 1.3 x2 is 3.7 points under.
- On MedMCQA, the models give mostly the same answer to every question.
The highlighted values in these rows are therefore not meaningful wins.
Language modeling does separate the models. On WikiText-2, Koliber 1.3 at x2 and x3 has the lowest byte perplexity. On LAMBADA, every model is clearly above the near-zero baseline, but low in absolute terms.
Recurrent depth. On the benchmarks that separate the models (MultiBLiMP, WikiText-2, LAMBADA), x2 and x3 score better than x1, and x3 adds little over x2. These differences between depths, and those between Koliber versions, have not been tested pairwise.
Protocol
All results come from the benchmark suite of SlayerLab / Fabryka AI (CodeSOTA), run with its runner python -m codesota.runner at commit a295888. Every benchmark uses the suite's full test set at a pinned dataset revision, 0-shot log-likelihood scoring through lm-evaluation-harness 0.4.9.1, and the same settings for every model:
| Setting | Value |
|---|---|
| Precision | float32 (the BF16 weights are loaded in float32) |
| Batch size | 1 |
| BOS | one leading BOS token per request (the suite's default) |
| Hardware | NVIDIA GeForce RTX 5060 Ti (16 GB), Windows 11 |
| Software | Python 3.12.2, PyTorch 2.8.0+cu128, Transformers 4.57.6, Datasets 3.6.0, Tokenizers 0.22.2 |
| Model code | trust_remote_code (the repository's modeling_koliber.py, reviewed before the runs) |
Commands, from benchmarks/ of the suite at that commit:
python -m codesota.runner --benchmarks @small,@medium --model OrisTeam/Koliber-1.3-BeLooped \
--revision c384ec7ffb1efbe0ddc539478fe6986b7084bd95 --model-id OrisTeam/Koliber-1.3-BeLooped@x2 \
--trust-remote-code --out runs/
Deviations, each recorded in the run's manifest:
- x1 and x3 run from a local copy of this repository whose
config.jsonsetsrecurrence_stepsto 1 or 3. The weights and code are unchanged. - WikiText-2 for Koliber 1.3 uses a 2048-token window (
--max-length 2048). With the configured 131,072 positions, the evaluation's windows exceed 2048 tokens, and the model's code requires flash-attn for those, which is not available on this machine. Koliber 1.0-1.2 use their configured 1536 tokens. Perplexity depends on the window, so the 1.3 and older WikiText-2 values are not strictly comparable. - Koliber 1.0-1.2 run from local copies whose
tokenizer_config.jsonnames the tokenizer classPreTrainedTokenizerFastinstead ofTokenizersBackend, which Transformers 4.57.6 does not know. The tokenizer file is unchanged, and the token IDs were checked to be identical.
Validation. Every result in the table comes from a run directory that passes the suite's result validator (python -m codesota.results, all 72 runs behind this table). The validator checks the run's structure, checksums, per-question records, and provenance fields. It does not check model quality. The runner itself was verified against the suite's reference pipeline: on Pythia-1B, it reproduces the November 2026 full run on all 13 benchmarks (verification).
Architecture
| Property | Configuration |
|---|---|
| Parameters | 125,459,712 |
| Architecture | Decoder-only, recurrent Transformer |
| Unique decoder layers | 15 |
| Layer distribution | 5 input / 6 recurrent / 4 output |
| Execution depth | x1: 15 / x2: 21 / x3: 27 |
| Hidden size | 768 |
| FFN dimension | 2048 |
| Attention | GQA, 16 query heads / 4 KV heads |
| Head dimension | 64 |
| Q projection dimension | 1024 |
| Attention pattern | Alternating Full / Sliding Window |
| Sliding window | 512 tokens |
| Normalization | Pre-RMSNorm |
| Positional encoding | RoPE |
| Activation | SwiGLU |
| Linear biases | Disabled |
| Embeddings | 32,768 rows, tied input/output |
| Tokenizer | Aster v1.05, 32,000 tokens |
| Training sequence length | 16,384 tokens |
| Configured maximum positions | 131,072 |
| Training stage | Base pretraining |
Recurrent execution
The architecture consists of three stages:
Input (5 layers) โ Recurrent (6 layers ร N) โ Output (4 layers)
The recurrent block reuses the same learned parameters across successive passes. Its execution count can be configured independently of the number of unique model parameters.
| Mode | Executed layers | Full attention | Sliding-window attention |
|---|---|---|---|
| x1 | 15 | 8 | 7 |
| x2 | 21 | 11 | 10 |
| x3 | 27 | 14 | 13 |
The architecture combines local attention with periodic access to the complete preceding sequence. The final decoder layer uses full attention.
These execution modes provide different computational budgets while preserving the same model weights. The benchmarks above give a first comparison of their quality; their computational efficiency has not been measured.
Training
The model was pretrained from scratch using the SlayerLab GoLLeM Polish corpus, with a configured mixture of web-derived and curated Polish-language material.
| Property | Training configuration |
|---|---|
| Processed tokens | 6,000,017,408 |
| Sequence length | 16,384 |
| Effective batch | 65,536 tokens/update |
| Optimizer | AdamW |
| Training precision | FP32 weights with BF16 autocast |
| Recurrence | Mixed x1/x2/x3 |
| Data sampling configuration | 60% web-derived / 40% curated |
| Hardware | NVIDIA RTX PRO 6000 Blackwell |
Training used document packing with explicit document boundaries. The reported token count describes processed training exposure, not the number of unique tokens in the underlying corpus.
Context length
The 16,384-token training sequence was used partly to stress-test the training implementation, extended attention, recurrent execution, and the data pipeline.
The model configuration reserves a maximum positional range of 131,072 tokens for future experimentation.
These values must not be interpreted as validated effective context lengths. The ability to retrieve, preserve, or combine information across long sequences has not been established through dedicated evaluation.
The Koliber line
Koliber remains an experimental line of models around 125M parameters, used to try out architectures, training decisions, and data within a fixed, small budget. The versions differ in architecture and in training exposure, so the benchmarks compare models, not controlled changes. In particular, they do not account for inference FLOPs, which grow with the recurrent depth.
The architecture, training decisions, and experimental observations will be discussed in a separate technical note.
Checkpoints and implementation
The repository includes the final model and intermediate training milestones.
| Location | Contents |
|---|---|
| Repository root | Final 6B-token checkpoint, BF16 |
fp32/ |
Final 6B-token checkpoint, FP32 |
milestones/1B/ |
Intermediate checkpoint, BF16 |
milestones/2B/ |
Intermediate checkpoint, BF16 |
milestones/3B/ |
Intermediate checkpoint, BF16 |
milestones/4B/ |
Intermediate checkpoint, BF16 |
milestones/5B/ |
Intermediate checkpoint, BF16 |
training/ |
Training implementation |
The Hugging Face implementation provides configurable recurrent execution through recurrence_steps.
The final FP32 export was checked against the original training checkpoint, including weight integrity and numerical logit comparisons for x1, x2, and x3. The BF16 export was separately checked for loading and numerical consistency with FP32 on short test sequences. The benchmarks above use the BF16 export from the repository root.
These implementation checks are not model-quality benchmarks.
Intended use and limitations
Koliber v1.3-BeLooped is intended for architecture research, language-model experimentation, continued pretraining, and downstream fine-tuning.
It is a base language model, not an instruction-following assistant. It has not undergone instruction tuning, preference optimization, or dedicated safety alignment.
The model may produce incorrect information, repetition, incoherent continuations, or misleading text. Reliable factual behavior and long-context capabilities have not been demonstrated.
This release should be treated as an experimental checkpoint of the Koliber architecture family.
License & Credits
Model weights: CC BY-SA 4.0. Source corpus licensing and original text rights are separate from the model-weight license; users are responsible for checking applicable upstream restrictions.
Data source: SlayerLab GoLLeM Polish corpus, prepared by SlayerLab / Arkadiusz Sลota and contributors.
Development: Aleksander Ogrodzki / OrisTeam.
@misc{ogrodzki2026koliber-13,
author = {Ogrodzki, Aleksander},
title = {Koliber-1.3-BeLooped},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/OrisTeam/Koliber-1.3-BeLooped}
}
- Downloads last month
- 16