Instructions to use jomangbp/seldonium-2x7b-MoE-v0.1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jomangbp/seldonium-2x7b-MoE-v0.1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="jomangbp/seldonium-2x7b-MoE-v0.1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("jomangbp/seldonium-2x7b-MoE-v0.1")
model = AutoModelForCausalLM.from_pretrained("jomangbp/seldonium-2x7b-MoE-v0.1", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use jomangbp/seldonium-2x7b-MoE-v0.1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jomangbp/seldonium-2x7b-MoE-v0.1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jomangbp/seldonium-2x7b-MoE-v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/jomangbp/seldonium-2x7b-MoE-v0.1

SGLang

How to use jomangbp/seldonium-2x7b-MoE-v0.1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "jomangbp/seldonium-2x7b-MoE-v0.1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jomangbp/seldonium-2x7b-MoE-v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "jomangbp/seldonium-2x7b-MoE-v0.1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jomangbp/seldonium-2x7b-MoE-v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use jomangbp/seldonium-2x7b-MoE-v0.1 with Docker Model Runner:
```
docker model run hf.co/jomangbp/seldonium-2x7b-MoE-v0.1
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

seldonium-2x7b-MoE-v0.1

seldonium-2x7b-MoE-v0.1-coder-logic is a Mixture of Experts (MoE) model that combines the capabilities of two specialized language models:

Locutusque/Hercules-4.0-Mistral-v0.2-7B: A 7B parameter model focused on programming tasks, such as writing functions, implementing algorithms, and working with data structures.

Open-Orca/Mistral-7B-OpenOrca: A 7B parameter model focused on logical reasoning and analysis, including solving logic problems, evaluating arguments, and assessing the validity of statements.

This MoE model was created using the LazyMergekit colab, which allows for efficient combination of specialized models to produce a more capable and efficient overall model. The seldonium-2x3b-MoE-v0.1 can be used for a variety of natural language processing tasks that benefit from the complementary strengths of its expert components.

🧩 Configuration

base_model: NousResearch/Hermes-2-Pro-Mistral-7B
gate_mode: cheap_embed  # Use hidden state representations to determine MoE gates
dtype: bfloat16  # Output data type
experts_per_token: 2  # Number of experts per token
experts:
  - source_model: Locutusque/Hercules-4.0-Mistral-v0.2-7B
    positive_prompts:
      - "Write a Python function to calculate the factorial of a number."
      - "Implement a quicksort algorithm to sort a list of integers."
      - "Design a Python class to represent a binary search tree."
  
  - source_model: Open-Orca/Mistral-7B-OpenOrca
    positive_prompts:
      - "Solve the logic puzzle: 'If Ann is older than Belinda, and Belinda is younger than Cathy, who is the oldest?'"
      - "Analyze the argument: 'All cats are animals. Some animals are pets. Therefore, all cats are pets.' Determine if the conclusion follows logically from the premises."
      - "Evaluate the validity of the statements: 'A is true. A is false.'"

💻 Usage

!pip install -qU transformers bitsandbytes accelerate

from transformers import AutoTokenizer
import transformers
import torch

model = "jomangbp/seldonium-2x3b-MoE-v0.1"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": True},
)

messages = [{"role": "user", "content": "Explain what a Mixture of Experts is in less than 100 words."}]
prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])

Downloads last month: 8

Safetensors

Model size

13B params

Tensor type

BF16

Model tree for jomangbp/seldonium-2x7b-MoE-v0.1

Locutusque/Hercules-4.0-Mistral-v0.2-7B

Open-Orca/Mistral-7B-OpenOrca

Merge model

this model

Quantizations

2 models