Previous incidents

Service has been restored as of a few minutes ago.

Model version publishing is now working as expected.

The issues with inference was limited to select LLM models. At this time the problematic code has been rolled back and all inference should be operating normally at this time.

All outstanding issues have been resolved. Model boots and setups should be functioning normally again.

All queues have been dealt with, and predictions and trainings are running smoothly once again.

Backlogs have been cleared and all models are now running smoothly.

The source of the problem appears to have been that our API was unable to connect to one of its underlying data stores, most likely due to a networking interruption. This has recovered as of 00:02 UTC and traffic is being served normally once again. We will continue to monitor.

Webhooks for Dreambooth trainings are working again.

At this time service has been restored. All inference (prediction serving) and model instance starts have returned to normal.

Our systems indicate that the problem has been resolved. We will continue monitoring the situation.