Previous incidents

This has been resolved and predictions should be handled normally.

We've fixed the issue and predictions on trained versions are running again.

Things are running normally as of about 15 minutes ago.

Service has been restored. Thanks for your patience!

The model is processing predictions properly again, and the queue is empty.

Service has been restored as of a few minutes ago.

Model version publishing is now working as expected.

The issues with inference was limited to select LLM models. At this time the problematic code has been rolled back and all inference should be operating normally at this time.

All outstanding issues have been resolved. Model boots and setups should be functioning normally again.

All queues have been dealt with, and predictions and trainings are running smoothly once again.

Backlogs have been cleared and all models are now running smoothly.

The source of the problem appears to have been that our API was unable to connect to one of its underlying data stores, most likely due to a networking interruption. This has recovered as of 00:02 UTC and traffic is being served normally once again. We will continue to monitor.

Webhooks for Dreambooth trainings are working again.