NVFP4
#2
by Skillerok - opened
I’ve looked into the project, and it seems very interesting for the 5090—especially given the use of hardware support for NVFP4, unlike in llama.cpp, which lacks it. Is there any chance this model will be released in NVFP4 format for the Nifer engine?
I think this is technically possible, if one was to convert the model to nvfp4 first THEN run the convert script in ninfer, since ninfer has an nvfp4 mode if it detects the model as such. Right now its groupwise-int I believe, I might get around to it later, no promises