The first version was trained on SD15 and didn't capture the real style I was going for. That being said it did turn out nice, just different.
So now I used the same training data and trained it on Illustrious-XL - v0.1.
The result worked pretty well given I only used 50 images as training data.
The images where made:
CFG: 5
Steps: 25
Sampler: Euler
Strength: 0.9
Have not had the time to experiment yet.