Any updates to the Gemma model or are you fully switching to Qwen?

#1
by kabachuha - opened

Hello!

I'm loving the Gemma-Deckard version, given the core Gemma's already vast knowledge and RP/Writing capability. Is in your plans to enhance it even further or you are switching to Qwen? I'd greatly enjoy an even more advanced Gemma4-Deckard version :)

Owner

I will be coming back to Gemmas shortly.

There is an issue with "expanding" Gemmas (IE: from 31B to 43B ) that is halting further tuning in this regard.
Gemma 4s seem to be "LAYER LOCKED", which means when you add extra layers the model "explodes".
Even when you follow Gemma 4's arch of 6 layer-blocks.
The "E" series is even more difficult to work with.

I have built larger Gemma 4s ( IE 43B, E8B ) ; but the road to bring these to "functional" requires a lot more work.
Gemma 3s do not have the same issue.

Note the Qwens - including 3.5/3.6 - do not have these kinds of issues.

is it possible to tune 26b models?

Owner

@maxwell6778

Yes, but a lot more work - and these are more difficult to train in general.
Add to the fact these must be trained at BF16 [can't compress to 4bit]; the vram [60-80GB] / time [due to all the expert training] is considerable.

all have to be on gpu or can half be on ram?

Owner

GPU all ; cpu/ram offloading is terrible and glitch-y.
Does not work ; plus add it is a sparse moe and Gemma [special requirements]...

thank you for telling me the info

Sign up or log in to comment