File size: 2,514 Bytes
91cc0c7 2d49e60 91cc0c7 8e463d5 91cc0c7 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 |
---
tags:
- text-to-image
- lora
- diffusers
- template:diffusion-lora
license: apache-2.0
base_model:
- Tongyi-MAI/Z-Image-Turbo
---
# Z-Image-Turbo Training Adapter
This is a training adapter designed to be used for fine-tuning [Tongyi-MAI/Z-Image-Turbo](https://huggingface.co/Tongyi-MAI/Z-Image-Turbo).
It was made for use with [AI Toolkit](https://github.com/ostris/ai-toolkit) but could potentially be used in other trainers as well.
If you are implementing it into training code and have questions. I am always heppy to help. Just reach out. It can
also be used as a general de-distillation LoRA for inference to remove the "Turbo" from "Z-Image-Turbo".
### Why is it needed?
When you train directly on a step distilled model, the distillation breaks down very quickly. This results in losing the step distillation
in an unpredictable way. A de-distill training adapter slows this process down significantly allowing you to do short training runs while
preserving the step distillation (speed).
### What is the catch?
This is really just a hack to significantly slow down the distillation when fine-tuning a distilled model. The distillation will
still be broken down over time. What that means is, this adapter will work great for shorter runs such as styles, concepts, and
characters. However, doing a long training run will likely lead to the distillation breaking down to a point where artifacts
will be produced when the adapter is removed.
### How was it made?
I generated thousands of images at various sizes and aspect ratios using
[Tongyi-MAI/Z-Image-Turbo](https://huggingface.co/Tongyi-MAI/Z-Image-Turbo). Then I simply trained a LoRA on those images at a low learning
rate (1e-5). This allowed the distillation to break down while preserving the model's existing knowledge.
### How does it work?
Since this adapter has broken down the distillation, if you train a LoRA on top of it, the distillation will no longer break down in
your new LoRA, since this adapter has de-distilled the model. Your LoRA will now only learn the subject you are training. When
it comes time to run inference / sampling, we remove this training adapter which leaves your new information on the distilled model
allowing your new information to run at distilled speeds. Attached, is an example of a short training run on a character with and without
this adapter
|