Previous incidents

We have now fully resolved the issues and API and replicate.com website are fully operational.

The API and website are now working as expected for predictions and trainings.

We have identified a problematic ingress pod and have caused it to reschedule. The API and website are now working as expected for predictions and trainings.

We have rolled back the problematic change. Website functionality has been restored and API error rate has returned to normal.

We have worked through the pending predictions and trainings and now see normal start times.

We have recovered our caching service and see predictions and training succeeding.

We resolved the capacity issues.

Both the API and web are now back to normal. Predictions, trainings are functioning as expected.

We are continuing to monitor things.

API is now behaving normally.

Everything is resolved and back to normal.

During the downtime predictions were completing normally in API, but are not persisted.

The issue with the upstream provider has been resolved. Predictions and Trainings are expected to be starting within normal timeframes.

Prediction processing and prediction are working as expected now.

Everything should be working normally at this time.

Thank you for your patience. At this time all hung workloads targeted for the T4 hardware should no longer be stuck in starting phase.

Engineers have rolled back a change to the website and at this time the website should now be responding as expected.

Reverting the identified change and purging known bad cache values has resolved the error rate within the API service. API and Web should be responding as expected at this time.

We believe that as of a few minutes ago the last customer impact from this issue has been resolved and all queues have cleared. To help you correlate this incident with any issues you may have seen: as far as we can tell the earliest customer impact from this incident started at about 11:00 UTC today.

The fix has been rolled out for all models.

We've identified the source of the errors -- a global load balancing service appears to have been misbehaving -- and made changes to how we serve api.replicate.com to mitigate the problem. As of a few minutes ago, we are no longer serving 520 error responses to customers.

We're back! We pushed a bad change and have rolled it back. Sorry for the inconvenience.

All prediction requests are now responding normally. We're still investigating the underlying cause.

The API is fully recovered. Unfortunately we are still at least partially in the dark about what triggered these problems. We're continuing to investigate.

Services have recovered. We'll be following up with our provider to understand how the scope of the planned maintenance expanded to affect customer workloads.

We've identified a service that was starved of compute resources and addressed that problem. Service has been restored.

We've restored service to the queueing system and predictions are flowing again.