Used https://civitai.com/user/seruva19 's Ghibli dataset. 120 images, batch size 4, 100 epochs, 3000 steps. Learning rate: 1e-5. 13 hours running on 2 x rtx 5000 ada gpus. I used diffusion-pipeline since it allows distributing loads across multiple gpu's.
*Update V2.0 I doubled the lr rate to 2e-5 to make a more potent LORA.
Image2Image guidance: If you want to do img2img, I recommend setting sampler = lcm and scheduler = linear_quadratic. The denoise will be between .3 - .4 and steps around 5-10. Will need to adjust on a case by case basis.