Previous incidents

We have confirmed and corrected any model versions erroneously disabled during this issue.

Use of A40s for predictions and trainings is now working as expected.

It looks to us like one of our providers had a brief outage and things are now coming back. We're continuing to monitor the situation.

(Technical details: it looks like an upstream provider had a brief DNSSEC zone signing outage.)

The slow model startup has resolved. We will continue to work internally and with our provider to remediate the root cause.

The prediction and webhook delivery issues are resolved now. There might be still a delay in webhook delivery of older predictions.

The issue has been resolved and predictions are now functioning normally.

Rollback of the problematic change has completed and Replicate website is now functioning normally again.

We have rolled out a fix and confirmed finetunes are working as expected.

Predictions and trainings are back to normal.

We identified a problem affecting a small portion of customers -- slow responses to webhooks caused a backlog in processing outbound webhooks -- and have deployed a change to increase available webhook processing capacity. Webhook delivery is back to normal as of a few minutes ago.

We've fixed the issue and you should be able to push new versions again.

Database load is back to normal, performance should be back to usual levels.

We have now fully resolved the issues and API and replicate.com website are fully operational.

The API and website are now working as expected for predictions and trainings.

We have identified a problematic ingress pod and have caused it to reschedule. The API and website are now working as expected for predictions and trainings.

We have rolled back the problematic change. Website functionality has been restored and API error rate has returned to normal.

We have worked through the pending predictions and trainings and now see normal start times.

We have recovered our caching service and see predictions and training succeeding.

We resolved the capacity issues.

Both the API and web are now back to normal. Predictions, trainings are functioning as expected.

We are continuing to monitor things.

API is now behaving normally.

Everything is resolved and back to normal.

During the downtime predictions were completing normally in API, but are not persisted.

The issue with the upstream provider has been resolved. Predictions and Trainings are expected to be starting within normal timeframes.

Prediction processing and prediction are working as expected now.

Everything should be working normally at this time.