Instructions to use jatayulabs/Qwen3-32B-ToolCall-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use jatayulabs/Qwen3-32B-ToolCall-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf jatayulabs/Qwen3-32B-ToolCall-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf jatayulabs/Qwen3-32B-ToolCall-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf jatayulabs/Qwen3-32B-ToolCall-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf jatayulabs/Qwen3-32B-ToolCall-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf jatayulabs/Qwen3-32B-ToolCall-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf jatayulabs/Qwen3-32B-ToolCall-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf jatayulabs/Qwen3-32B-ToolCall-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf jatayulabs/Qwen3-32B-ToolCall-GGUF:Q4_K_M

Use Docker

docker model run hf.co/jatayulabs/Qwen3-32B-ToolCall-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use jatayulabs/Qwen3-32B-ToolCall-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jatayulabs/Qwen3-32B-ToolCall-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jatayulabs/Qwen3-32B-ToolCall-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/jatayulabs/Qwen3-32B-ToolCall-GGUF:Q4_K_M

Ollama
How to use jatayulabs/Qwen3-32B-ToolCall-GGUF with Ollama:
```
ollama run hf.co/jatayulabs/Qwen3-32B-ToolCall-GGUF:Q4_K_M
```

Unsloth Studio

How to use jatayulabs/Qwen3-32B-ToolCall-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for jatayulabs/Qwen3-32B-ToolCall-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for jatayulabs/Qwen3-32B-ToolCall-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for jatayulabs/Qwen3-32B-ToolCall-GGUF to start chatting

How to use jatayulabs/Qwen3-32B-ToolCall-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf jatayulabs/Qwen3-32B-ToolCall-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "jatayulabs/Qwen3-32B-ToolCall-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use jatayulabs/Qwen3-32B-ToolCall-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf jatayulabs/Qwen3-32B-ToolCall-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default jatayulabs/Qwen3-32B-ToolCall-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use jatayulabs/Qwen3-32B-ToolCall-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf jatayulabs/Qwen3-32B-ToolCall-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "jatayulabs/Qwen3-32B-ToolCall-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use jatayulabs/Qwen3-32B-ToolCall-GGUF with Docker Model Runner:
```
docker model run hf.co/jatayulabs/Qwen3-32B-ToolCall-GGUF:Q4_K_M
```

Lemonade

How to use jatayulabs/Qwen3-32B-ToolCall-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull jatayulabs/Qwen3-32B-ToolCall-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3-32B-ToolCall-GGUF-Q4_K_M

List all available models

lemonade list

Qwen3-32B ToolCall — GGUF

The Jatayu Labs agentic tool-calling model, packaged to run anywhere llama.cpp runs.
One 48 GB GPU. One command. A full OpenAI-compatible tool-calling API.

🧠 Source adapter + checkpoints · ⚡ Serve in 60 seconds · 🪶 Jatayu Labs

🚀 What this model does

Give it a goal and a toolbox — it plans the sequence, calls the right tools with the right arguments, reads every result, recovers from failures, and reports back in the user's own language: English, हिंदी, or Hinglish.

Verified orchestration flow

Every transcript below was generated by this exact GGUF file, served with llama-server on 2× NVIDIA A40, temperature 0.1, unedited.

Episode 1 — end-to-end order (English)

USER: Order 2 packets of Amul milk and 1 brown bread on Zepto. Pay cash on delivery.

[1] → search_multiple_products({"queries": ["Amul milk", "brown bread"]})
[2] → update_cart({"items": [{"product_id": "ZP-MILK-001", "quantity": 2},
                             {"product_id": "ZP-BREAD-007", "quantity": 1}]})
[3] → get_payment_methods({})
[4] → create_order({"payment_method": "COD"})

MODEL: Your order has been confirmed! 🛒
       Order ID: ZO-88121
       Items: 2 × Amul Taaza Toned Milk 500ml (₹29 each)
              1 × Britannia Brown Bread 400g (₹50)
       Total: ₹108 | Payment: Cash on Delivery
       Estimated delivery: 9 minutes.

One request → four correct tool calls → a confirmation where every number (order ID, prices, ETA) comes from tool results. Nothing invented.

Episode 2 — same task, in Hinglish

USER: yaar zepto se 2 amul doodh aur ek brown bread mangwa de, cash on delivery rakhna

[1] → search_multiple_products({"queries": ["Amul doodh", "brown bread"]})
[2] → update_cart({...})  [3] → get_payment_methods({})  [4] → create_order({"payment_method": "COD"})

MODEL: Your order has been confirmed! 🛒  Order ID: ZO-88121 · Total ₹108 ·
       COD · delivery in ~9 minutes.

Casual romanized Hindi is a first-class input — the model even carried the user's own word ("Amul doodh") into the search query.

Episode 3 — Hindi in, Hindi out

USER: कृपया Zepto से 2 अमूल दूध और 1 ब्राउन ब्रेड ऑर्डर करें। भुगतान कैश ऑन डिलीवरी होगा।

[1-4] → identical four-step tool plan

MODEL: आपका ऑर्डर स्वीकृत हो गया है!
       ऑर्डर ID: ZO-88121
       कुल राशि: ₹108 · भुगतान विधि: कैश ऑन डिलीवरी (COD)
       अनुमानित डिलीवरी समय: 9 मिनट
       हम जल्द से जल्द आपके पास डिलीवर करेंगे! 🚚

Devanagari in → Devanagari out. No language flag, no extra prompting.

Episode 4 — error recovery

USER: Order 2 Amul milk on Zepto, cash on delivery.

[3] → create_order({"payment_method": "cod"})
      ← {"status": "error", "reason": "COD not available for this store. Available: UPI, Zepto Cash"}

MODEL: The Cash on Delivery (COD) option is not available for this store. You can
       proceed using UPI or Zepto Cash. Would you like to update your payment preference?

A payment failure is surfaced and handed back to the user — no silent retries, no fake success.

Measured results (this file, this hardware)

What we measured	Result
Multi-step episodes completed correctly	4 / 4 (English, Hinglish, Hindi, error-injection)
Tool selection across all steps	13 / 13 calls correct
Argument validity (schema-conformant JSON)	13 / 13 calls
Language fidelity (reply matches user's language)	3 / 3
Hallucinated values in final answers	0
Sustained throughput (2× NVIDIA A40, `-ngl 99`)	~26.7 tok/s across all 15 generations

⚡ Serve it in 60 seconds

hf download jatayulabs/Qwen3-32B-ToolCall-GGUF qwen3-32b-toolcall-Q4_K_M.gguf --local-dir models

llama-server -m models/qwen3-32b-toolcall-Q4_K_M.gguf \
    --host 0.0.0.0 --port 8000 --jinja -ngl 99 -c 8192

--jinja activates the embedded tool-calling chat template. You now have a standard OpenAI-compatible endpoint:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [{"role": "user", "content": "What is the weather in Delhi right now?"}],
  "tools": [{"type": "function", "function": {"name": "get_weather",
    "description": "Get current weather for a city",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"}},
                   "required": ["city"]}}}]
}'
# → finish_reason: "tool_calls", get_weather({"city": "Delhi"})

Works with any OpenAI SDK — point base_url at your server and pass tools as usual.

File

File	Quant	Size	Fits on
`qwen3-32b-toolcall-Q4_K_M.gguf`	Q4_K_M (4.82 BPW)	19.8 GB	1× 48 GB GPU (comfortable, 8–16k ctx) · 1× 24 GB (short ctx) · 2× 24 GB

🔬 How we trained it

Trained on NVIDIA A100 GPUs with LLaMA-Factory, using LoRA on top of Qwen3-32B. This GGUF is the step-600 release checkpoint (best train loss, 0.070) merged into the base weights.

Training loss curve

Epoch 1 learns the format of structured tool calls (plateau ~0.45), a sharp drop at the epoch boundary marks format mastery (0.44 → 0.23), epoch 2 refines planning, and the model locks in at 0.070 by step 600. Step 800 showed early overfitting (0.077), so step 600 is the release.

Setting	Value
Base model	Qwen3-32B (32.8B params)
Method	LoRA — rank 64, alpha 128, all attention + MLP projections
Trainable parameters	537M (1.6% of total)
Optimizer	LoRA+ (AdamW, lr-ratio 16) · lr 1e-4 cosine, 10% warmup
Effective batch size	16 · sequence length 2,048
Training data	5,000 curated multi-step tool-calling conversations — quick-commerce, payments, messaging, search — with English, Hindi, and Hinglish coverage
Final train loss	0.070 (step 600, epoch 2.1)

🪶 About Jatayu Labs

Jatayu Labs is an Indian AI lab building agentic intelligence for a billion users — in their own languages.

🔧 Fine-tuned LLMs for agentic tool calling and orchestration
🎯 Specialised small language models for focused production tasks
🎙️ Voice-modality models — speech in, speech out
🇮🇳 Indic languages first — Hindi, Hinglish, and regional languages as first-class citizens

Jatayu — the bird of Indian epic who flew fastest and acted first.

License: Apache 2.0 · Contact: via the Community tab

Downloads last month: 75

GGUF

Model size

33B params

Architecture

qwen3

Hardware compatibility

4-bit

Model tree for jatayulabs/Qwen3-32B-ToolCall-GGUF

Base model

Qwen/Qwen3-32B

Adapter

jatayulabs/Qwen3-32B-ToolCall

Quantized

(1)

this model