Yeah honestly haven't run that test yet, but you're right that's the real one to run.
Idea is โ same multi-hop Hindi prompts, run once with think forced open, once closed, see if accuracy actually moves or if that English reasoning was just... there, not really doing anything.
For data, skipping MILU (it's recall, not real reasoning) and MGSM doesn't even have Hindi. So just hand-checking a small GSM8K-hi slice myself instead of auto-translating โ didn't want noise messing with the results.
And yeah will track think-block token count too. If closed keeps the same accuracy as open, that's basically free speed, no tradeoff. Will post numbers once I run it.
๐๏ธ Building on HF
Pankaj Pandey
pankajpandey-dev
AI & ML interests
Natural Language Processing, Text Generation, Large Language Models, Quantization, Fine-Tuning, RLHF, Model Merging.
Organizations
replied to their post 2 months ago
replied to their post 2 months ago
Fair callout โ I don't actually know yet. My checks were informal, not a real multi-step eval.
Hunch: for simple factual/instructional prompts (most of my training data), the English trace was probably just habit, not load-bearing. For chained reasoning (math, multi-hop), it might matter more โ I didn't test that specifically.
Planning to run a small multi-step Hindi eval (with/without forced-open think block) and post numbers. Open to using a benchmark if you have one you trust.
reacted to their post with ๐ฅ๐ค๐๐คฏ๐ค๐โค๏ธ๐๐ 2 months ago
Post
4144
๐ฎ๐ณ Qwen3.5-9B Hindi Instruct โ it stops thinking in English
Ask base Qwen3.5-9B a question in Hindi and it burns hundreds of tokens thinking in English inside its think block before a single Devanagari word appears โ then code-switches in the answer. I fine-tuned it to close the think block instantly and reply in pure, native Hindi.
โ Model (16-bit): pankajpandey-dev/qwen3.5-9b-hindi-instruct
โ GGUF (Q4/Q5/Q8): pankajpandey-dev/qwen3.5-9b-hindi-instruct-GGUF
โ Try it in the browser: pankajpandey-dev/qwen3.5-9b-hindi-demo
Recipe: Unsloth + LoRA (r=16, response-only loss) on 12.9k Hindi pairs โ AI4Bharat anudesh + dolly-hi + wikiHow-hi + Aya Hindi (human-written). The Q4_K_M is 5.4 GB and runs on a plain laptop CPU.
New in this run vs my earlier models: mixed in long-form native sources (wikiHow) after my last eval showed the fine-tune traded detail for conciseness โ this one keeps answers detailed and native.
Part of my weekly ๐ฎ๐ณ Hindi LLM Series. Feedback welcome ๐
#Hindi #IndicNLP #Qwen #GGUF #LocalLLM #Unsloth
Ask base Qwen3.5-9B a question in Hindi and it burns hundreds of tokens thinking in English inside its think block before a single Devanagari word appears โ then code-switches in the answer. I fine-tuned it to close the think block instantly and reply in pure, native Hindi.
โ Model (16-bit): pankajpandey-dev/qwen3.5-9b-hindi-instruct
โ GGUF (Q4/Q5/Q8): pankajpandey-dev/qwen3.5-9b-hindi-instruct-GGUF
โ Try it in the browser: pankajpandey-dev/qwen3.5-9b-hindi-demo
Recipe: Unsloth + LoRA (r=16, response-only loss) on 12.9k Hindi pairs โ AI4Bharat anudesh + dolly-hi + wikiHow-hi + Aya Hindi (human-written). The Q4_K_M is 5.4 GB and runs on a plain laptop CPU.
New in this run vs my earlier models: mixed in long-form native sources (wikiHow) after my last eval showed the fine-tune traded detail for conciseness โ this one keeps answers detailed and native.
Part of my weekly ๐ฎ๐ณ Hindi LLM Series. Feedback welcome ๐
#Hindi #IndicNLP #Qwen #GGUF #LocalLLM #Unsloth
- 4 replies
posted an update 2 months ago
Post
4144
๐ฎ๐ณ Qwen3.5-9B Hindi Instruct โ it stops thinking in English
Ask base Qwen3.5-9B a question in Hindi and it burns hundreds of tokens thinking in English inside its think block before a single Devanagari word appears โ then code-switches in the answer. I fine-tuned it to close the think block instantly and reply in pure, native Hindi.
โ Model (16-bit): pankajpandey-dev/qwen3.5-9b-hindi-instruct
โ GGUF (Q4/Q5/Q8): pankajpandey-dev/qwen3.5-9b-hindi-instruct-GGUF
โ Try it in the browser: pankajpandey-dev/qwen3.5-9b-hindi-demo
Recipe: Unsloth + LoRA (r=16, response-only loss) on 12.9k Hindi pairs โ AI4Bharat anudesh + dolly-hi + wikiHow-hi + Aya Hindi (human-written). The Q4_K_M is 5.4 GB and runs on a plain laptop CPU.
New in this run vs my earlier models: mixed in long-form native sources (wikiHow) after my last eval showed the fine-tune traded detail for conciseness โ this one keeps answers detailed and native.
Part of my weekly ๐ฎ๐ณ Hindi LLM Series. Feedback welcome ๐
#Hindi #IndicNLP #Qwen #GGUF #LocalLLM #Unsloth
Ask base Qwen3.5-9B a question in Hindi and it burns hundreds of tokens thinking in English inside its think block before a single Devanagari word appears โ then code-switches in the answer. I fine-tuned it to close the think block instantly and reply in pure, native Hindi.
โ Model (16-bit): pankajpandey-dev/qwen3.5-9b-hindi-instruct
โ GGUF (Q4/Q5/Q8): pankajpandey-dev/qwen3.5-9b-hindi-instruct-GGUF
โ Try it in the browser: pankajpandey-dev/qwen3.5-9b-hindi-demo
Recipe: Unsloth + LoRA (r=16, response-only loss) on 12.9k Hindi pairs โ AI4Bharat anudesh + dolly-hi + wikiHow-hi + Aya Hindi (human-written). The Q4_K_M is 5.4 GB and runs on a plain laptop CPU.
New in this run vs my earlier models: mixed in long-form native sources (wikiHow) after my last eval showed the fine-tune traded detail for conciseness โ this one keeps answers detailed and native.
Part of my weekly ๐ฎ๐ณ Hindi LLM Series. Feedback welcome ๐
#Hindi #IndicNLP #Qwen #GGUF #LocalLLM #Unsloth
- 4 replies
reacted to their post with ๐ 2 months ago
Post
7855
๐ฎ๐ณ New in my Hindi LLM Series: Gemma-4 E4B, fine-tuned for Hindi โ and it runs on your laptop's CPU.
I fine-tuned Google's new Gemma-4 E4B on ~10k Hindi instruction pairs (AI4Bharat: anudesh + dolly) using Unsloth + LoRA, on a single L4 GPU.
Then I ran an honest side-by-side eval: base Gemma-4 vs my fine-tune, across 25 Hindi prompts. The results were interesting ๐
โ My fine-tune is more concise โ ask for "3 tips" and it gives exactly 3. Base writes a 1,200-character essay.
โ Pure native Hindi โ base keeps slipping into English ("เคธเคเคคเฅเคฒเคฟเคค เคเคนเคพเคฐ (Eat a Balanced Diet)", "เคคเคพเคฐเคพ (Star)"). My fine-tune stays in clean Hindi.
โ Tighter instruction-following โ ask for a "short message" and it gives one, not a menu of options.
โ๏ธ And to be honest: base Gemma-4 is more detailed and comprehensive. I didn't build a "smarter" model โ I built a focused, Hindi-native, edge-friendly one that runs as a 5GB GGUF (Q4) on CPU.
๐ Try it:
Live demo (CPU): pankajpandey-dev/gemma-4-e4b-hindi-demo
GGUF (Ollama/llama.cpp): pankajpandey-dev/gemma-4-e4b-hindi-instruct-GGUF
16-bit model: pankajpandey-dev/gemma-4-e4b-hindi-instruct
Built with @unsloth ยท Data by @ai4bharat ๐
#Hindi #LLM #Gemma #Unsloth #IndicNLP #GGUF
I fine-tuned Google's new Gemma-4 E4B on ~10k Hindi instruction pairs (AI4Bharat: anudesh + dolly) using Unsloth + LoRA, on a single L4 GPU.
Then I ran an honest side-by-side eval: base Gemma-4 vs my fine-tune, across 25 Hindi prompts. The results were interesting ๐
โ My fine-tune is more concise โ ask for "3 tips" and it gives exactly 3. Base writes a 1,200-character essay.
โ Pure native Hindi โ base keeps slipping into English ("เคธเคเคคเฅเคฒเคฟเคค เคเคนเคพเคฐ (Eat a Balanced Diet)", "เคคเคพเคฐเคพ (Star)"). My fine-tune stays in clean Hindi.
โ Tighter instruction-following โ ask for a "short message" and it gives one, not a menu of options.
โ๏ธ And to be honest: base Gemma-4 is more detailed and comprehensive. I didn't build a "smarter" model โ I built a focused, Hindi-native, edge-friendly one that runs as a 5GB GGUF (Q4) on CPU.
๐ Try it:
Live demo (CPU): pankajpandey-dev/gemma-4-e4b-hindi-demo
GGUF (Ollama/llama.cpp): pankajpandey-dev/gemma-4-e4b-hindi-instruct-GGUF
16-bit model: pankajpandey-dev/gemma-4-e4b-hindi-instruct
Built with @unsloth ยท Data by @ai4bharat ๐
#Hindi #LLM #Gemma #Unsloth #IndicNLP #GGUF
- 12 replies
replied to their post 2 months ago
Thanks for sharing, Divyanshu! I appreciate it. It's encouraging to see more open-source work focused on Hindi and its dialects. I'll explore SASTRI and see how it performs on our evaluation benchmarks. If you have any recommended benchmark datasets or evaluation methodology, I'd love to hear about them
reacted to their post with โค๏ธ๐ 2 months ago
Post
7855
๐ฎ๐ณ New in my Hindi LLM Series: Gemma-4 E4B, fine-tuned for Hindi โ and it runs on your laptop's CPU.
I fine-tuned Google's new Gemma-4 E4B on ~10k Hindi instruction pairs (AI4Bharat: anudesh + dolly) using Unsloth + LoRA, on a single L4 GPU.
Then I ran an honest side-by-side eval: base Gemma-4 vs my fine-tune, across 25 Hindi prompts. The results were interesting ๐
โ My fine-tune is more concise โ ask for "3 tips" and it gives exactly 3. Base writes a 1,200-character essay.
โ Pure native Hindi โ base keeps slipping into English ("เคธเคเคคเฅเคฒเคฟเคค เคเคนเคพเคฐ (Eat a Balanced Diet)", "เคคเคพเคฐเคพ (Star)"). My fine-tune stays in clean Hindi.
โ Tighter instruction-following โ ask for a "short message" and it gives one, not a menu of options.
โ๏ธ And to be honest: base Gemma-4 is more detailed and comprehensive. I didn't build a "smarter" model โ I built a focused, Hindi-native, edge-friendly one that runs as a 5GB GGUF (Q4) on CPU.
๐ Try it:
Live demo (CPU): pankajpandey-dev/gemma-4-e4b-hindi-demo
GGUF (Ollama/llama.cpp): pankajpandey-dev/gemma-4-e4b-hindi-instruct-GGUF
16-bit model: pankajpandey-dev/gemma-4-e4b-hindi-instruct
Built with @unsloth ยท Data by @ai4bharat ๐
#Hindi #LLM #Gemma #Unsloth #IndicNLP #GGUF
I fine-tuned Google's new Gemma-4 E4B on ~10k Hindi instruction pairs (AI4Bharat: anudesh + dolly) using Unsloth + LoRA, on a single L4 GPU.
Then I ran an honest side-by-side eval: base Gemma-4 vs my fine-tune, across 25 Hindi prompts. The results were interesting ๐
โ My fine-tune is more concise โ ask for "3 tips" and it gives exactly 3. Base writes a 1,200-character essay.
โ Pure native Hindi โ base keeps slipping into English ("เคธเคเคคเฅเคฒเคฟเคค เคเคนเคพเคฐ (Eat a Balanced Diet)", "เคคเคพเคฐเคพ (Star)"). My fine-tune stays in clean Hindi.
โ Tighter instruction-following โ ask for a "short message" and it gives one, not a menu of options.
โ๏ธ And to be honest: base Gemma-4 is more detailed and comprehensive. I didn't build a "smarter" model โ I built a focused, Hindi-native, edge-friendly one that runs as a 5GB GGUF (Q4) on CPU.
๐ Try it:
Live demo (CPU): pankajpandey-dev/gemma-4-e4b-hindi-demo
GGUF (Ollama/llama.cpp): pankajpandey-dev/gemma-4-e4b-hindi-instruct-GGUF
16-bit model: pankajpandey-dev/gemma-4-e4b-hindi-instruct
Built with @unsloth ยท Data by @ai4bharat ๐
#Hindi #LLM #Gemma #Unsloth #IndicNLP #GGUF
- 12 replies
replied to their post 2 months ago
Fire in, energy out ๐ฅ โ โก
reacted to their post with ๐ฅ 2 months ago
Post
7855
๐ฎ๐ณ New in my Hindi LLM Series: Gemma-4 E4B, fine-tuned for Hindi โ and it runs on your laptop's CPU.
I fine-tuned Google's new Gemma-4 E4B on ~10k Hindi instruction pairs (AI4Bharat: anudesh + dolly) using Unsloth + LoRA, on a single L4 GPU.
Then I ran an honest side-by-side eval: base Gemma-4 vs my fine-tune, across 25 Hindi prompts. The results were interesting ๐
โ My fine-tune is more concise โ ask for "3 tips" and it gives exactly 3. Base writes a 1,200-character essay.
โ Pure native Hindi โ base keeps slipping into English ("เคธเคเคคเฅเคฒเคฟเคค เคเคนเคพเคฐ (Eat a Balanced Diet)", "เคคเคพเคฐเคพ (Star)"). My fine-tune stays in clean Hindi.
โ Tighter instruction-following โ ask for a "short message" and it gives one, not a menu of options.
โ๏ธ And to be honest: base Gemma-4 is more detailed and comprehensive. I didn't build a "smarter" model โ I built a focused, Hindi-native, edge-friendly one that runs as a 5GB GGUF (Q4) on CPU.
๐ Try it:
Live demo (CPU): pankajpandey-dev/gemma-4-e4b-hindi-demo
GGUF (Ollama/llama.cpp): pankajpandey-dev/gemma-4-e4b-hindi-instruct-GGUF
16-bit model: pankajpandey-dev/gemma-4-e4b-hindi-instruct
Built with @unsloth ยท Data by @ai4bharat ๐
#Hindi #LLM #Gemma #Unsloth #IndicNLP #GGUF
I fine-tuned Google's new Gemma-4 E4B on ~10k Hindi instruction pairs (AI4Bharat: anudesh + dolly) using Unsloth + LoRA, on a single L4 GPU.
Then I ran an honest side-by-side eval: base Gemma-4 vs my fine-tune, across 25 Hindi prompts. The results were interesting ๐
โ My fine-tune is more concise โ ask for "3 tips" and it gives exactly 3. Base writes a 1,200-character essay.
โ Pure native Hindi โ base keeps slipping into English ("เคธเคเคคเฅเคฒเคฟเคค เคเคนเคพเคฐ (Eat a Balanced Diet)", "เคคเคพเคฐเคพ (Star)"). My fine-tune stays in clean Hindi.
โ Tighter instruction-following โ ask for a "short message" and it gives one, not a menu of options.
โ๏ธ And to be honest: base Gemma-4 is more detailed and comprehensive. I didn't build a "smarter" model โ I built a focused, Hindi-native, edge-friendly one that runs as a 5GB GGUF (Q4) on CPU.
๐ Try it:
Live demo (CPU): pankajpandey-dev/gemma-4-e4b-hindi-demo
GGUF (Ollama/llama.cpp): pankajpandey-dev/gemma-4-e4b-hindi-instruct-GGUF
16-bit model: pankajpandey-dev/gemma-4-e4b-hindi-instruct
Built with @unsloth ยท Data by @ai4bharat ๐
#Hindi #LLM #Gemma #Unsloth #IndicNLP #GGUF
- 12 replies
posted an update 2 months ago
Post
7855
๐ฎ๐ณ New in my Hindi LLM Series: Gemma-4 E4B, fine-tuned for Hindi โ and it runs on your laptop's CPU.
I fine-tuned Google's new Gemma-4 E4B on ~10k Hindi instruction pairs (AI4Bharat: anudesh + dolly) using Unsloth + LoRA, on a single L4 GPU.
Then I ran an honest side-by-side eval: base Gemma-4 vs my fine-tune, across 25 Hindi prompts. The results were interesting ๐
โ My fine-tune is more concise โ ask for "3 tips" and it gives exactly 3. Base writes a 1,200-character essay.
โ Pure native Hindi โ base keeps slipping into English ("เคธเคเคคเฅเคฒเคฟเคค เคเคนเคพเคฐ (Eat a Balanced Diet)", "เคคเคพเคฐเคพ (Star)"). My fine-tune stays in clean Hindi.
โ Tighter instruction-following โ ask for a "short message" and it gives one, not a menu of options.
โ๏ธ And to be honest: base Gemma-4 is more detailed and comprehensive. I didn't build a "smarter" model โ I built a focused, Hindi-native, edge-friendly one that runs as a 5GB GGUF (Q4) on CPU.
๐ Try it:
Live demo (CPU): pankajpandey-dev/gemma-4-e4b-hindi-demo
GGUF (Ollama/llama.cpp): pankajpandey-dev/gemma-4-e4b-hindi-instruct-GGUF
16-bit model: pankajpandey-dev/gemma-4-e4b-hindi-instruct
Built with @unsloth ยท Data by @ai4bharat ๐
#Hindi #LLM #Gemma #Unsloth #IndicNLP #GGUF
I fine-tuned Google's new Gemma-4 E4B on ~10k Hindi instruction pairs (AI4Bharat: anudesh + dolly) using Unsloth + LoRA, on a single L4 GPU.
Then I ran an honest side-by-side eval: base Gemma-4 vs my fine-tune, across 25 Hindi prompts. The results were interesting ๐
โ My fine-tune is more concise โ ask for "3 tips" and it gives exactly 3. Base writes a 1,200-character essay.
โ Pure native Hindi โ base keeps slipping into English ("เคธเคเคคเฅเคฒเคฟเคค เคเคนเคพเคฐ (Eat a Balanced Diet)", "เคคเคพเคฐเคพ (Star)"). My fine-tune stays in clean Hindi.
โ Tighter instruction-following โ ask for a "short message" and it gives one, not a menu of options.
โ๏ธ And to be honest: base Gemma-4 is more detailed and comprehensive. I didn't build a "smarter" model โ I built a focused, Hindi-native, edge-friendly one that runs as a 5GB GGUF (Q4) on CPU.
๐ Try it:
Live demo (CPU): pankajpandey-dev/gemma-4-e4b-hindi-demo
GGUF (Ollama/llama.cpp): pankajpandey-dev/gemma-4-e4b-hindi-instruct-GGUF
16-bit model: pankajpandey-dev/gemma-4-e4b-hindi-instruct
Built with @unsloth ยท Data by @ai4bharat ๐
#Hindi #LLM #Gemma #Unsloth #IndicNLP #GGUF
- 12 replies
reacted to their post with ๐ฅ 3 months ago
Post
922
๐ฎ๐ณ Gemma-3-1B Hindi Instruct โ a Hindi LLM that runs fully offline, anywhere.
Last week I shipped Qwen3-4B Hindi. This week I went the other direction: how tiny can a useful Hindi model get? So I fine-tuned Gemma-3-1B on quality-filtered Hindi instruction data and shipped the full GGUF ladder.
โ Fine-tune (16-bit): pankajpandey-dev/gemma-3-1b-hindi-instruct
โ GGUF (Q4/Q5/Q8): pankajpandey-dev/gemma-3-1b-hindi-instruct-GGUF
Runs in Ollama, llama.cpp, and LM Studio. The Q4_K_M is just 806 MB โ runs on CPU, a cheap laptop, even a Raspberry Pi.
What I tried this round: chrF-filtered the training data to drop weak translations, and used response-only loss so the model learns how to answer, not how to repeat prompts.
Honest note: at 1B, Hindi fluency is strong but coherence is bounded by size โ it's a lightweight/edge experiment, not a 4B replacement. Gemma-3-4B Hindi is next.
Part of my Hindi LLM Series โ openly-licensed Indic models for local & edge use. Feedback welcome ๐
#Hindi #IndicNLP #GGUF #LocalLLM #Gemma #EdgeAI
Last week I shipped Qwen3-4B Hindi. This week I went the other direction: how tiny can a useful Hindi model get? So I fine-tuned Gemma-3-1B on quality-filtered Hindi instruction data and shipped the full GGUF ladder.
โ Fine-tune (16-bit): pankajpandey-dev/gemma-3-1b-hindi-instruct
โ GGUF (Q4/Q5/Q8): pankajpandey-dev/gemma-3-1b-hindi-instruct-GGUF
Runs in Ollama, llama.cpp, and LM Studio. The Q4_K_M is just 806 MB โ runs on CPU, a cheap laptop, even a Raspberry Pi.
What I tried this round: chrF-filtered the training data to drop weak translations, and used response-only loss so the model learns how to answer, not how to repeat prompts.
Honest note: at 1B, Hindi fluency is strong but coherence is bounded by size โ it's a lightweight/edge experiment, not a 4B replacement. Gemma-3-4B Hindi is next.
Part of my Hindi LLM Series โ openly-licensed Indic models for local & edge use. Feedback welcome ๐
#Hindi #IndicNLP #GGUF #LocalLLM #Gemma #EdgeAI