TL;DR: Our primary data center is pulling GPU support out from under us to switch to crypto mining, and we got less than 48 hours of notice. Generation still works, but it will be slower while we stand up a new home for our hardware. Expect image and video generation for models like Illustrious, Pony, and Anima to take longer. We may toggle specific models on and off, and some models may be Members-only during the worst of it. We will keep this article updated as things change.
What's happening
Yesterday we were told our primary data center is retracting support for us and switching their capacity over to crypto mining. Effective today. We got less than 48 hours of notice.
That is the bulk of the GPUs we use to serve open source model generation. Losing them on this kind of timeline is about as abrupt as it gets.
We have spent every hour since scrambling for replacement GPUs and a new data center. The catch: the same mining fever that took our capacity has gobbled up most of the spare GPUs on the market. We have a new home lined up, but it takes time to get ready. It is not going to be instant.
What this means for you
Generation is not going away. But while we move, it is going to be slower and lumpier than you are used to.
Slower images. Models that we serve like Illustrious, Pony, Anima, and more may take a while, especially if you're requesting an uncommon LoRA. We are already scaling out on other services to absorb what we can, but there is only so much headroom.
Some models may go on and off. Over the next ten days or so we may have to turn specific models off and back on as we balance load against the hardware we actually have.
Some models may be Members-only for stretches. During the tightest periods, generation on certain models may be limited to Members so we can keep things moving. If that happens, it is temporary.
What we're doing about it
The team is heads-down on this. The short version:
Securing a new data center and getting our hardware moved in as fast as it can physically happen.
Scaling out on the services we can scale out on right now to take pressure off.
Watching load in real time and making calls on a model-by-model basis so the experience stays as steady as we can keep it.
We are not happy about this. We did not get a say in the timeline, and we would much rather not be writing this. But either way this was going to be one that the community felt.
About the timeline
When we say ten days, that is the plan as it stands right now, assuming our new home comes together on schedule. We are also chasing a few other venues in parallel, so there is a real chance it lands sooner. There is also a chance it runs longer if the new data center hits delays getting ready. We genuinely do not know yet. What we can promise is that we are pushing on every option at once, and we will update this article the moment the picture gets clearer.
Updates
Updates will be added here as things change.
June 5th 8:30pm UTC: Generation with the following models are turned on for all users: SD 1.5, SDXL, Pony, Illustrious, Flux1, NoobAI, Z-Image, and HiDream. Additional resource limits have been temporarily reduced (Free Users 9->3, Members 12->8) due to limited bandwidth on our temporary hardware. We are continuing to monitor the situation and hope to be able to increase limits and re-open additional models like Anima and Flux Klein within the next few hours.
Thank you
You have ridden out bumpy stretches with us before, and it never goes unnoticed. We will get through this one the same way: heads-down, transparent about where things stand, and moving as fast as the hardware will let us.
Hang in there. We are on it. More soon.
Mandatory P.S. Yes, we're still working on other issues and bugs! We even made a whole new page so you can keep track of ones we're aware of, check out known issues.