Upload Flux2-Klein-9B-True-V3-INT8-ConvRot.safetensors
INT8 quantized model with ConvRot (requires the nodes https://github.com/BobJohnson24/ComfyUI-INT8-Fast/) is much faster than the FP8 model on my RTX 4070 with 12GB VRAM; I believe it is approximately twice as fast.
The model was quantized using the same tools provided by the node developer on GitHub
con
| Configuration | Time per Iteration | Speedup |
|---|---|---|
| INT8 ConvRot + SageAttention | 2.26 s/it | 🚀 2.48x faster |
| FP8 + SageAttention | 5.60 s/it | 1.00x (baseline) |
Note that it does not work well with torch compile (if you are using it, it might be better to disable it or check the author's GitHub)
As for whether it loses quality or not, I believe I understood that it even has better quality as long as it is ConvRot when compared to FP8. I don't know, I don't understand any of this technically; I compared it visually in ComfyUI and didn't see any appreciable loss or notable gain, things look very similar.
Could you add the file or convert it yourself? And also add it to the readme.md so people know they can use it; it supposedly works very well on older GPUs like the 3000 and 2000 series. Thank you very much for everything!!
Thanks! I will try and test it, I know Silveroxides's tool, it is ComfyUI offical used. if it working better, I will upload INT8 quantized version.
On latest Comfy: ComfyUI execution error: Unknown quantization format for layer double_blocks.0.img_attn.qkv
On latest Comfy:
ComfyUI execution error: Unknown quantization format for layer double_blocks.0.img_attn.qkv
I had tested two Int8 model on ComfyUI just now, Everything works fine, no problem. for "Flux2-Klein-9B-True-V3-INT8-ConvRot.safetensors", you need install ComfyUI-INT8-Fast plugin first and load model by it's node; and for "Flux2-Klein-9B-True-V3-int8mixedrow.safetensors", you can load it normal as diffusion_model node by ComfyUI offical.
Supposedly, official support for int8 ConvRot has already been added, but I haven't tested it myself, so I don't know how mature or stable it is: https://github.com/Comfy-Org/ComfyUI/releases/tag/v0.27.0
For the moment, I will continue using the https://github.com/BobJohnson24/ComfyUI-INT8-Fast/ node, which I consider to be safer, and then once everything stabilizes, I might switch to the official one. Keep in mind that they have support for LoRA and other things that I'm not sure have been implemented well in the official version.
If you want to test it on the official version, make sure it is updated to version v0.27.0 or nightly @jtreminio
I'm also getting the same error:
ComfyUI Error Report
Error Details
- Node ID: 33
- Node Type: UNETLoader
- Exception Type: ValueError
- Exception Message: ValueError: Unknown quantization format for layer double_blocks.0.img_attn.qkv
comfyUI version: 0.27.0
comfy-kitchen version: 0.2.16
comfyui-frontend-package version: 1.45.20
What I understand is, if using the latest Comfyui, this case 0.27.0, then the custom node "ComfyUI-INT8-Fast" should not be needed anymore as the default loader should now work.
And yes, the default loader is working because I have tested a few other int8 convrot models from winnougan (https://huggingface.co/Winnougan) like ZIT and Krea2 and its working as expected, inference is fast and no errors.
I'm also getting the same error:
ComfyUI Error Report
Error Details
- Node ID: 33
- Node Type: UNETLoader
- Exception Type: ValueError
- Exception Message: ValueError: Unknown quantization format for layer double_blocks.0.img_attn.qkv
comfyUI version: 0.27.0
comfy-kitchen version: 0.2.16
comfyui-frontend-package version: 1.45.20What I understand is, if using the latest Comfyui, this case 0.27.0, then the custom node "ComfyUI-INT8-Fast" should not be needed anymore as the default loader should now work.
And yes, the default loader is working because I have tested a few other int8 convrot models from winnougan (https://huggingface.co/Winnougan) like ZIT and Krea2 and its working as expected, inference is fast and no errors.
Yeah? That's strange! How about we run a test? Could you convert the normal Bf16 mode to FP8 using the ComfyUI-INT8-Fast nodes or silveroxiodes' script, and after that, try running it without those nodes, using the default ComfyUI one? Let us know what happens. If it throws an error, it most likely has something to do with the ComfyUI implementation.
Give us more details: what command are you using to launch ComfyUI, what versions of PyTorch and Python are you using, share your workflow with us, and are you using SageAttention, Flash, or anything else?
@rontonie Don't worry about testing it, I already did it for you. I have the same error with my ComfyUI nightly; it appears to be an error due to a poor implementation in ComfyUI. I recommend you use the author's nodes at https://github.com/BobJohnson24/ComfyUI-INT8-Fast/, who made sure to create a good implementation. For now, all I can do is report this to ComfyUI, but first I'm going to test the normal model in convrot mode to see what happens.
Error (this only happens when using the official comfy nodes)
File "C:\Users\User\Documents\TEO\ComfyUI\.venv\Lib\site-packages\torch\nn\modules\module.py", line 2603, in load
load(child, child_state_dict, child_prefix) # noqa: F821
~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\User\Documents\TEO\ComfyUI\.venv\Lib\site-packages\torch\nn\modules\module.py", line 2603, in load
load(child, child_state_dict, child_prefix) # noqa: F821
~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Previous line repeated 1 more time]
File "C:\Users\User\Documents\TEO\ComfyUI\.venv\Lib\site-packages\torch\nn\modules\module.py", line 2586, in load
module._load_from_state_dict(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
local_state_dict,
^^^^^^^^^^^^^^^^^
...<5 lines>...
error_msgs,
^^^^^^^^^^^
)
^
File "C:\Users\User\Documents\TEO\ComfyUI\comfy\ops.py", line 1194, in _load_from_state_dict
_load_quantized_module(self, super()._load_from_state_dict, *args, load_extra_params=True)
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\User\Documents\TEO\ComfyUI\comfy\ops.py", line 1072, in _load_quantized_module
raise ValueError(f"Unknown quantization format for layer {layer_name}")
ValueError: Unknown quantization format for layer double_blocks.0.img_attn.qkv
Solution
If you are reading this and experiencing the same issue, for the time being, the solution is to use the author's original code at https://github.com/BobJohnson24/ComfyUI-INT8-Fast because the official implementation appears to be buggy.
Update
In fact, it is a recognized and officially reported bug; it seems they do not have support for everyone, read here https://github.com/Comfy-Org/ComfyUI/issues/14722
Awesome. Regarding the "ComfyUI-INT8-Fast" nodes, can it be used to convert other finedtuned fp8 models to int8 convrot?
Awesome. Regarding the "ComfyUI-INT8-Fast" nodes, can it be used to convert other finedtuned fp8 models to int8 convrot?
Yes, but the recommendation is to try doing it from bf16 so that it preserves better precision. As I understand it, theoretically you can do it from int8 to int8, but you could lose some precision; if you have the bf16, it's better to use that. I tested it by doing them with the bf16 and it is very fast (I thought it was going to be a slow headache due to the low VRAM issues of my GPU). You can do it from the author's nodes in the int8-fast GitHub link, or there is also a conversion module by silveroxides right there.
I've finished comparing everything. My "int8mixedrow" quantization(Silveroxides's tool) is exactly the same as ComfyUI's official "int8_convrot" quantization. The only difference is that the shape of my .comfy_quant is [46], while the official one is [72]. This weight difference is one ten-thousandth. The differences in layers and step sampling accumulate and show up in the image's pixel texture, but to the naked eye, there's almost no difference. So, you can just use my quantized version; it's fully compatible with the official loading node. Here's a comparison of the images: