Previous incidents

All outstanding issues have been resolved. Model boots and setups should be functioning normally again.

All queues have been dealt with, and predictions and trainings are running smoothly once again.

Backlogs have been cleared and all models are now running smoothly.

The source of the problem appears to have been that our API was unable to connect to one of its underlying data stores, most likely due to a networking interruption. This has recovered as of 00:02 UTC and traffic is being served normally once again. We will continue to monitor.

Webhooks for Dreambooth trainings are working again.

At this time service has been restored. All inference (prediction serving) and model instance starts have returned to normal.

Our systems indicate that the problem has been resolved. We will continue monitoring the situation.

All but a very small slice of our A40 hardware is back online, and Replicate workloads are processing normally. We again thank you for your patience.

Workloads across all regions are now running normally. We apologise for the disruption, and will working to better improve our ability to shift load between providers in situations like this one.

We identified excessive load on our database. Shortly after the root cause was isolated, our engineering team disabled the problematic code paths.

Degradation of database responsiveness resulted in the general service outage beginning at approximately 6:23UTC.

At this time API, Inference, and Web are fully functional. Any predictions that resulted in errors or were marked as failed during this window can be safely re-run.