AryamaR/Frog_in_yarn_art_SDXL_Dreambooth_Lora

Model Card for Model ID

This model is finetuned from SDXL base model. The aim is to build character consistency over a multi prompt comic panels. This model is fine tuned with reference class image of a Frog.

 ```from diffusers import DiffusionPipeline, StableDiffusionXLPipeline, DPMSolverMultistepScheduler
import torch
from IPython.display import Image, display

#loading vae path
from diffusers import AutoencoderKL
vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)

device="cuda"

#loading SDXl pipeline
pipe = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    vae=vae,
    torch_dtype=torch.float16,
    variant="fp16",
    use_safetensors=True).to(device)

lora_path = "iamno-one/sdxl_dreambooth_lora_frog_v2"

pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)

## adding Lora weight
pipe.load_lora_weights(lora_path)

pipe.to("cuda")

##Multi Prompt
prompts = [
    "Tok, A frog, in a colorful yarn world, eating fly, yarn style",
    "The Frog is jumping across yarn hills, whimsical, wide shot, in yarn art style",
    "The Frog is holding a yarn cup, cozy lighting, yarn art style",
    "The Frog is watching stars from a yarn hill at night, dreamy, in yarn art style"
]
generated_images = []

for prompt in prompts:
    image = pipe(
        prompt=prompt,
        negative_prompt="low quality, blurry, deformed, extra limbs, text, watermark, realistic image",
        num_inference_steps=30,
        guidance_scale=7.5,
    ).images[0]
    generated_images.append(image)  ``` 

Sample Comic with hard coded multiprompt in text bubble for testing