Previous incidents

Thank you for your patience. At this time all hung workloads targeted for the T4 hardware should no longer be stuck in starting phase.

Engineers have rolled back a change to the website and at this time the website should now be responding as expected.

Reverting the identified change and purging known bad cache values has resolved the error rate within the API service. API and Web should be responding as expected at this time.

We believe that as of a few minutes ago the last customer impact from this issue has been resolved and all queues have cleared. To help you correlate this incident with any issues you may have seen: as far as we can tell the earliest customer impact from this incident started at about 11:00 UTC today.

The fix has been rolled out for all models.

We've identified the source of the errors -- a global load balancing service appears to have been misbehaving -- and made changes to how we serve api.replicate.com to mitigate the problem. As of a few minutes ago, we are no longer serving 520 error responses to customers.

We're back! We pushed a bad change and have rolled it back. Sorry for the inconvenience.

All prediction requests are now responding normally. We're still investigating the underlying cause.

The API is fully recovered. Unfortunately we are still at least partially in the dark about what triggered these problems. We're continuing to investigate.

Services have recovered. We'll be following up with our provider to understand how the scope of the planned maintenance expanded to affect customer workloads.

We've identified a service that was starved of compute resources and addressed that problem. Service has been restored.

We've restored service to the queueing system and predictions are flowing again.

Predictions are flowing as expected once again. We'll continue to monitor the situation.

The rollback fixed things and we're back to normal.

We've identified what's causing this issue and have rolled back the change. Affected predictions will have completed successfully and you can re-request their status through the API.

Autoscaling issues have been resolved for all models and everything should be operating normally.