@Banaxi-Tech @vovaRL does 250-350M parameters still count as a SLM?
nice
@ProCreations who won the thing? will the results be open source?
@juiceb0xc0de One other kid also has an account. He joined huggingface last week @MUK-IS-GOAT
nice
@juiceb0xc0de no. i goto QASMT in qld. and the kids there just like this kinda stuff lol
@ProCreations Also i have an update on my entry, the code now features custom Triton kernels i have testes in my molab thing and im happy to say it doubles speed!
@juiceb0xc0de correction on that i have managed to create my own triton kernels with my friends just then at like lunch so we got from 3K tokens per second ===> 7K per second! so probably a lot faster.
@Banaxi-Tech Out of curiosity, do you usually write your own triton kernels for the training? i do it but im not sure if other ai devs do the same lol
sometimes crazy ones fail but they might end up being very good i usualy just screw around till one works lol
lol nice
@ProCreations Have a lot of people entered?
nice 10000000 B200s lol
The variance is due to the nature of the novel design with heterogenous sized experts.
@ProCreations The active parameter count is small like 350-400Million ish. its quite easy to run
@ProCreations Custom MoE 4.02 Billion parameters but only a few active like 300M active.
@ProCreations I entered an entry! Where am i supposed to put my code for the entry? Or is that later
@ProCreations You mean RTX 6000 Pro Blackwell? Thats what im using but my model is large and novel architecure
@juiceb0xc0de This time we have heterogenous MoE sizes as an experiment. To try and learn patterns smaller experts cannot. We also have shrunk the average size of experts to smaller to generalise better. It will take about 3 months as it stands!
