Icon

Updated weekly

OW

OW

Icon

Updated weekly

OW

Output Weekly

Output Weekly

Tools, workflows and honest rankings for people who make things with AI

Tools, workflows and honest rankings for people who make things with AI

Images

How to train your own LoRA

Ines Cardoso

A close up of a computer motherboard in the dark

Key takeaways

  • A LoRA trains on a small, varied image set, and AWS's FLUX.2 Klein guide recommends 20 to 50 images for most subjects.

  • Open-source trainers such as ai-toolkit, kohya_ss, OneTrainer, and the Diffusers scripts all run on your own hardware, and the example configs from ai-toolkit, AWS, and Diffusers all use a 1e-4 learning rate.

  • Save a checkpoint every few hundred steps and test several of them at different strengths, since the last one is often the most overfit.

  • Noisy or degraded output usually means too many steps or too high a learning rate, and the fix is an earlier checkpoint or a lower setting.

  • The base model's license follows your LoRA, and training on a real person's face needs that person's consent.

How do you train your own LoRA?

Collect 20 to 50 varied images of one subject, caption each one, pick a base model, and run an open-source trainer for roughly 1,500 to 2,000 steps at a learning rate near 1e-4. Then test the saved checkpoints at several strengths and keep the earliest one that holds your subject.

Time varies a lot with the model and the card. AWS puts a FLUX.2 Klein 4B run at 15 to 45 minutes on a single GPU, while the Diffusers docs say a full SD 1.5 run of 15,000 steps takes about 5 hours on an 11GB 2080 Ti. The rest of this guide follows the order of the work, and our plain explainer on LoRAs covers what the file is.

What you need before you start

You need a capable GPU, a base model, and a trainer. The ai-toolkit README lists an "Nvidia GPU with enough ram to do what you need" and gives no figure of its own, but its example FLUX.1 config is named for 24GB cards. AWS recommends 13GB or more of VRAM for FLUX.2 Klein 4B and suggests 512-pixel training on a 24GB GPU. The Diffusers SDXL guide says you can train SDXL LoRAs with less than 16GB of VRAM by turning on gradient checkpointing, an 8-bit Adam optimizer, memory-efficient attention, and fp16. Our VRAM guide for AI image and video covers generation, which needs less than training.

Pick the base model first, because a LoRA only works on the family it was trained for. ai-toolkit lists FLUX.1, FLUX.2, SDXL, and SD 1.5 among its supported image models, and kohya's scripts list SD 1.x and 2.x, SDXL, SD3, and FLUX.1. If you want an SDXL fine-tune such as Pony or Illustrious, our comparison of the three explains why a LoRA trained on one doesn't always behave on another. New to the model family itself? Start with what Stable Diffusion is.

Prepare the dataset

For a subject, AWS's guide says to use 20 to 50 high-quality images "with varied poses, lighting, and backgrounds." The Diffusers SDXL guide describes DreamBooth LoRA as working from "just a few (3~5) images," so treat that as a floor for a quick test and not a target.

On file handling, ai-toolkit supports jpg, jpeg, and png and says webp "currently has issues." It also says you don't need to crop or resize, since images are downscaled into buckets and never upscaled. Kohya's dataset config has its own resolution and bucket settings. Remove duplicates, blurry frames, and anything with a stray face or logo you don't want learned.

A character LoRA wants one subject in many settings, a style LoRA wants many subjects drawn or shot in one style, and a concept LoRA wants one thing, such as an object or effect, in many settings. The steps below are the same for all three. If the goal is a recurring character, write the reference details down first with our character bible guide, and read what identity drift is to see what the LoRA is meant to prevent.

Caption the images

Most trainers read a text file with the same name as each image. ai-toolkit says the file should contain only the caption, and kohya's dataset config takes a caption extension such as .txt. AWS notes that captions are optional in its workflow but that custom ones "produce higher-quality results" because they teach the model more specific links between your trigger word and what it sees.

Choose a trigger word that doesn't already mean something to the model. AWS suggests "ohwx," which "doesn't conflict with existing model knowledge." In ai-toolkit you can write a [trigger] placeholder in each caption and set the trigger word once in the config. You can also caption by machine, and the Diffusers LoRA guide trains on a dataset named for BLIP-generated captions. Ai-toolkit's FLUX example also drops the caption on 5% of training steps.

Choose the main settings

The trainers' own docs and examples agree more than you might expect.

Setting

Documented range or example

Where it comes from

Rank

4 to 128 commonly used, 16 as the example default, 32 for complex concepts

Kohya docs, AWS, ai-toolkit

Alpha

About half the rank, or equal to it

Kohya docs, AWS, ai-toolkit

Learning rate

1e-4 in the examples, 1e-4 to 1e-3 often used when alpha is 1

AWS, ai-toolkit, Diffusers, kohya

Steps

500 to 4,000 as a good range, with 500, 1,500, and 2,000 in the examples

ai-toolkit, AWS, Diffusers

Batch size

1 in every example, with gradient accumulation of 4 in the Diffusers SDXL command

ai-toolkit, Diffusers

Resolution

512 on 24GB GPUs, 1,024 for SDXL

AWS, Diffusers

Kohya's LoRA training doc says rank values "between 4 and 128 are commonly used" and recommends an alpha of about half the rank. It also says a text encoder learning rate of "a smaller value than that for U-Net is recommended," and its sample command uses 1e-5. Rank is how much the LoRA can learn, so a bigger value means a bigger file and more room to overfit. AWS says 16 is a good default and 32 suits complex or abstract concepts.

Kohya lets you set epochs instead of steps, and if both are given it uses the epoch count. It also names the memory savers, which are gradient checkpointing, mixed precision at fp16 or bf16, and the AdamW8bit optimizer.

Run the training

The trainers differ in how you drive them and not in what they do.

  • ai-toolkit has a web interface at localhost:8675 where you start, stop, and monitor jobs, and its example YAML configs hold the values above.

  • kohya_ss is a Gradio-based GUI for Kohya's scripts, and its README lists LoRA, LoHa, LoKr, DreamBooth, and fine-tuning support.

  • OneTrainer's README lists full fine-tuning, LoRA, and embeddings across models from SD 1.5 to FLUX.2 and Qwen Image.

  • The Diffusers SDXL script is a command line run through Accelerate, and its example uses 1,024-pixel images, a 1e-4 learning rate, and 500 steps.

Whichever you choose, set a validation prompt or sample interval so the trainer renders test images as it goes. You'll want them in the next step.

Check your checkpoints

Don't wait for the final file. Ai-toolkit's FLUX example saves every 250 steps, keeps the last four, and renders samples every 250 steps. AWS's guide saves checkpoints every 300 steps, and kohya can save every set number of epochs.

Line up the samples from early to late. Early ones usually look generic, and later ones look more like your subject until a point where detail turns noisy or every image starts to look the same. The best file is normally just before that point, so keep two or three candidates and test each one.

Test strength in a prompt

Load each candidate on the same base model with a fixed prompt and seed, then vary the strength. In Diffusers, a weight of 0 uses only the base model and 1 applies the LoRA fully, and the docs' two-LoRA example uses 0.7 and 0.8. Our LoRA explainer lists the ranges other interfaces use.

Run three checks. Use the trigger word alone to see if the LoRA fires. Put the subject in a scene that isn't in the dataset to see if it adapts. Leave the trigger word out to see if the LoRA leaks into unrelated images. A good checkpoint holds the subject at a moderate strength and still follows the prompt.

Fix the common failures

Overfitting is the usual one. Hugging Face's DreamBooth write-up says the method "tends to overfit quickly," that "high learning rates and too many training steps will lead to overfitting," and that noisy or degraded images are a likely sign. Pick an earlier checkpoint first, and if every checkpoint looks degraded, retrain with a lower learning rate or fewer steps.

Underfitting is the opposite problem. AWS says to increase the step count if outputs don't resemble your subject. Faces are harder than objects. The same Hugging Face write-up found faces needed about 1,200 steps where objects needed 400, though those figures come from full DreamBooth runs with their own learning rates, so don't copy them into a LoRA config.

Identity bleed, where the subject's look turns up in images where it shouldn't, shows up when the leave-the-trigger-out test fails. Two documented levers help. One is a varied dataset, as AWS advises. The other is prior preservation, which the Diffusers DreamBooth guide describes as using the model's own generated class images to help it "retain what it has learned about the class." Lowering the strength at generation time also weakens the effect.

Licensing and consent

The base model's terms come with the LoRA. The FLUX.1 [dev] license allows you to create Derivatives "for Non-Commercial Purposes" only, though it says you may use Output "for any purpose (including for commercial purposes)." The SDXL license is CreativeML Open RAIL++-M, which says use-based restrictions "MUST be included as an enforceable provision" when you distribute derivatives, and that the licensor "claims no rights in the Output." FLUX.2 [klein] 4B Base is listed on its Hugging Face card under Apache 2.0.

SDXL's use restrictions include harassing or defaming others and exploiting minors. Consent is a separate question. Train on photos of a real person only with that person's permission, and only on images you took or have the right to use. The FTC says the TAKE IT DOWN Act requires platforms to remove nonconsensual intimate images, "including those that are AI-generated," within 48 hours of a request.

How to train a LoRA FAQ

How many images do I need to train a LoRA?

AWS recommends 20 to 50 high-quality images with varied poses, lighting, and backgrounds for most subjects. The Diffusers SDXL guide shows DreamBooth LoRA working from 3 to 5 images, which is a quick-test floor.

How long does LoRA training take?

AWS puts FLUX.2 Klein 4B at 15 to 45 minutes of training on one GPU. The Diffusers docs say a full SD 1.5 run of 15,000 steps takes about 5 hours on an 11GB 2080 Ti.

How much VRAM do I need to train a LoRA?

It depends on the model. The Diffusers SDXL guide says less than 16GB works with gradient checkpointing, 8-bit Adam, memory-efficient attention, and fp16, and AWS recommends 13GB or more for FLUX.2 Klein 4B.

What rank and learning rate should I start with?

Start with rank 16 and a 1e-4 learning rate, since AWS and ai-toolkit examples both use those values. Kohya says ranks from 4 to 128 are commonly used, and AWS suggests 32 for complex concepts.

Why do my LoRA images look noisy or degraded?

That usually means overfitting, from too high a learning rate or too many steps. Test an earlier checkpoint, lower the strength, or retrain with fewer steps.

Can I sell a LoRA I trained on FLUX.1 [dev]?

Not under the license's terms. It limits Derivatives to non-commercial use, though it allows commercial use of the images you generate.