I have been keep testing different scenarios with OneTrainer for Fine-Tuning SDXL on my relatively bad dataset. My training dataset is deliberately bad so that you can easily collect a better one and surpass my results. My dataset is bad because it lacks expressions, different distances, angles, different clothing and different backgrounds.

Used base model for tests are Real Vis XL 4 :

Here below used training dataset 15 images:

 None of the images that will be shared in this article are cherry picked. They are grid generation with SwarmUI. Head inpainted automatically with segment:head - 0.5 denoise.

Full SwarmUI tutorial :

The training models can be seen as below :

If you are a company and want to access models message me

• BS1

• BS15_scaled_LR_no_reg_imgs

• BS1_no_Gradient_CP

• BS1_no_Gradient_CP_no_xFormers

• BS1_no_Gradient_CP_xformers_on

• BS1_yes_Gradient_CP_no_xFormers

• BS30_same_LR

• BS30_scaled_LR

• BS30_sqrt_LR

• BS4_same_LR

• BS4_scaled_LR

• BS4_sqrt_LR

• Best

• Best_8e_06

• Best_8e_06_2x_reg

• Best_8e_06_3x_reg

• Best_8e_06_no_VAE_override

• Best_Debiased_Estimation

• Best_Min_SNR_Gamma

• Best_NO_Reg

Based on all of the experiments above, I have updated our very best configuration which can be found here :

It is slightly better than what has been publicly shown in below masterpiece OneTrainer full tutorial video (133 minutes fully edited):

I have compared batch size effect and also how they scale with LR. But since batch size is usually useful for companies I won't give exact details here. But I can say that Batch Size 4 works nice with scaled LR.

Here other notable findings I have obtained. You can find my testing prompts at this post that is suitable for prompt grid :

Check attachments (test_prompts.txt, prompt_SR_test_prompts.txt) of above post to see 20 different unique prompts to test your model training quality and overfit or not.

All comparison full grids 1 (12817x20564 pixels) :

All comparison full grids 2 (2567x20564 pixels) :

Using xFormers vs not using xFormers

xFormers on vs xFormers off full grid :

xformers definitely impacts quality and slightly reduces it

Example part (left xformers on right xformers off) :

Using regularization (also known as classification) images vs not using regularization images

Full grid here :

This is one of the biggest impact making part. When reg images are not used the quality degraded significantly

I am using 5200 ground truth unsplash reg images dataset from here :

Example of reg images dataset all preprocessed in all aspect ratios and dimensions with perfect cropping

 Example case reg images off vs on :

Left 1x regularization images used (every epoch 15 training images + 15 random reg images from 5200 reg images dataset we have) - right no reg images used only 15 training images

The quality difference is very significant when doing OneTrainer fine tuning

Loss Weight Function Comparisons

I have compared min SNR gamma vs constant vs Debiased Estimation. I think best performing one is min SNR Gamma then constant and worst is Debiased Estimation. These results may vary based on workflows but for my Adafactor workflow this is the case

Here full grid comparison :

Here example case (left ins min SNR Gamma right is constant ):

VAE Override vs Using Embedded VAE

We already know that custom models are using best fixed SDXL VAE but I still wanted to test this. Literally no difference as expected

Full grid :

Example case:

1x vs 2x vs 3x Regularization / Classification Images Ratio Testing

Since using ground truth regularization images provides far superior results, I decided to test what if we use 2x or 3x regularization images.

This means that in every epoch 15 training images and 30 reg images or 45 reg images used.

I feel like 2x reg images very slightly better but probably not worth the extra time.

Full grid :

Example case (1x vs 2x vs 3x) :

I also have tested effect of Gradient Checkpointing and it made 0 difference as expected.

Old Best Config VS New Best Config

After all findings here comparison of old best config vs new best config. This is for 120 epochs for 15 training images (shared above) and 1x regularization images at every epoch (shared above).

Full grid :

Example case (left one old best right one new best) :

New best config :

I'm experimenting with making a simple model of Brad Pitt, but this result doesn't look quite write. I'm wondering if this is an over/undertraining issue, or something else. I personally think it's undertrained, but I'd like professional input. Thanks!

Friends, the training is flawless, but the results are always like this.

I did the following examples with epicrealismieducation. I tried others as well, same result. I am missing something but I couldn't find it. Does anyone have an idea? I make all kinds of realistic realistic entries in the prompts.

It also looks normal up to 100%, it becomes like this at 100%. In other words, those hazy states look normal. It suddenly takes this form in its final state. I tried all the Sampling methods. I also tried it with different models like epicrealism, dreamshaper. I tried it with different photos and numbers.

For the life of me I can’t seem to add a model for Lora and dreambooth training. I only ever see original options included in Kohya. Can’t find a video about it either. How might I add a model from civitai or hugging face to train with? In other words, instead of selecting sdxl base for training, how might I add Juggernaut or Real Vis XL

Hello everyone,

I know why CLIP skip is used for inference, especially when using fine-tuned models. However, I am using Dreambooth (via kohya_ss) and was wondering when to use CLIP skip values greater than 0 when training.

From what I know, assuming no gradients are calculated for the CLIP layers that are skipped during training, a greater CLIP skip value should reduce VRAM utilization. Can someone tell me if that assumption is reasonable?

Then, what difference will it make during inference? Since the last X-amount of CLIP layers are practically frozen during training, they remain the same as they were in the base model. What would happen if a CLIP-skip > 0 trained model would be inferenced with CLIP skip = 0?

But the more important question: Why would someone choose to CLIP skip during training? I noticed that there is a lack of documentation and discussions on the topic of CLIP skip during training. It would be great if someone could enlighten me!

I'm creating a discord community called AIBuilders Community AIBC for GenAI Reseacher where I'm inviting people who like to contribute, Learn, generate and build with community

Who can join?

• Building GenAI And vision model mini Projects or MVP.

• Maintain projects on GitHub, hugging face son on.

• Testing github Projects, goggle collab, Kaggle, huggingface models, etc.

• Testing ComfiUI Workflow,

• Testing LLMs, SLM, VLLM so on.

• Want to create resources around GenAI and Vision models such as Reseacher Interview, Github Project or ComfiUI workflow discuss, Live project showcase, Finetuneting models, training dreambooth, lora, so on.

• Want to contribute to open source GenAI Newsletter.

• If you have idea to grow GenAI community together.

Everything will be Opensource on GitHub and I like to invite you to be the part of it.

Kindely DM me for the discord link.

Thank you

Does anyone know how I can use bounding boxes with Dreambooth or the correct format to do so when uploading captions? Every time I try to do so, it says my json schema is not correct.

Hi everyone,

I'm currently working on my thesis where I need to fine-tune DeepFloyd using the , aiming to generate images of a single turbine blade. However, I'm running into an issue where the model keeps generating the entire turbine instead of just one blade.

Here's what I've done so far:

• Increased training steps.

• Increased image number.

• Tried various text prompts ("a photo of a sks detached turbine-blade", "a photo of a sks singleaero-engine-blade" and similar), but none have yielded the desired outcome. I always get the whole tubine as an output and not just single blades as you can see in the attached image.

I’m hoping to get some advice on:

1. Best practices for fine-tuning DeepFloyd specifically to generate a single turbine blade.

2. Suggestions for the most effective text prompts to achieve this.

Has anyone encountered a similar problem or have any tips or insights to share? Your help would be greatly appreciated!

Thanks in advance!

Hi everyone, I started to do some dreambooth training on my dogs and I wanted to give a try with sdxl on colab, but what I am seeing confuse me, I always see dreambooth lora for sdxl, (for ex: ) and I thought that dreambooth and lora were 2 distincts techniques to fine tune your model, am I missing something ? ( maybe it is just about combining both ?). And a last question, kohya_ss is a UI with some scripts ? I mean it seems everyone (or almost) is using it, can I just go with the diffusers script, what koya brings in more ?

thanks

Hey everyone !

I wrote a tutorial about AI for some friends who are into it, and I've got a section that's specifically about training models and LoRAs.

It's actually part of a bigger webpage with other "tutorials" about things like UIs, ComfyUI and what not. If you guys think it's interesting enough I might post the entire thing (at this point it's become a pretty handy starting guide!)

I'm wondering where I could get some constructive criticism from smarter people than me, regarding the training pages ? I thought I'd ask here!

Cheers!!

Does anyone know if this is possible? And is dreambooth what im looking for? Its starting to seem that people dont consider dreambooth a lora maker but on youtube they act like thats all it is. Can anyone help me? Im super noob to yhis

edit beta full! Thanks everyone who volunteered!

———-

Hi all, a while back I published a couple articles about cutting dreambooth training costs with interruptible instances (i.e. spot instances or community cloud)

My employer let me build that out into an actual training service that runs on our community cloud, and here it is:

There's also a tutorial here:

I’ve been in image generation for a while, but my expertise is more in distributed systems than in stable diffusion training specifically, so I’d love feedback on how it can be more useful. It is based on the diffusers implementation (), and it saves the lora weights in both diffusers and webui/kohya formats.

I’m looking for 5 beta testers to use it for free (on credits) for a week to help iron out bugs and make improvements. DM me once you’ve got a salad account set up so I load up your credits.

I have about 9000 images (essentially black and white drawings of the same subject done in Ms paint) . I'm hoping to train a model and have stable diffusion create another 9000 drawings of its own (same basic style and same subject). Am I on the right path thinking that DreamBooth can help me? I'm not interested in having SD draw anything else. Can someone suggest a good strategy for this that I can start looking into? Thanks!

Hi, I'm trying to train a model on multiple concepts at once, mostly about a specific drawing style, and a person (eventually multiples but starting with just one). The goal of this experiment is to see if I can get a better version of that person in that specific drawing style than just stacking two LoRa's or one FT + one LoRa.

Has anyone experience regarding this kind of experiment they could share? (mostly regarding using small or large batch, and dataset weighting and captionning)?

Hello,

I'm currently using Dreambooth LoRA advanced in Colab ( ) and I'm looking for advice on an ideal or at least a good starting point for style training. The results I'm getting are not great, and I'm not sure what I'm missing.

I've generated captions for each image, but for some reason, in huggingface, I can only see the generated images from the validation prompt. Is this normal?

I tested the LoRA, but the results are far from what I was hoping for.

Any help would be greatly appreciated!

here are my current settings:

!accelerate launch train_dreambooth_lora_sdxl_advanced.py \
  --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" \
  --pretrained_vae_model_name_or_path="madebyollin/sdxl-vae-fp16-fix" \
  --dataset_name="./my_folder" \
  --instance_prompt="$instance_prompt" \
  --validation_prompt="$validation_prompt" \
  --output_dir="$output_dir" \
  --caption_column="prompt" \
  --mixed_precision="bf16" \
  --resolution=1024 \
  --train_batch_size=3 \
  --repeats=1 \
  --report_to="wandb"\
  --gradient_accumulation_steps=1 \
  --gradient_checkpointing \
  --learning_rate=1.0 \
  --text_encoder_lr=1.0 \
  --adam_beta2=0.99 \
  --optimizer="prodigy"\
  --train_text_encoder_ti\
  --train_text_encoder_ti_frac=0.5\
  --snr_gamma=5.0 \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --rank="$rank" \
  --max_train_steps=1000 \
  --checkpointing_steps=2000 \
  --seed="0" \
  --push_to_hub

Thanks,
Spyros