Previous incidents

The error rate seen has subsided and models are seeing previous startup and runtime behavior. We are working with our providers mitigate impact of future incidents like this.

The fix has been deployed and all model starts should be back to normal.

All services are working as expected and all workarounds have been restored to normal behavior. Additionally we have made improvements to ensure we can more quickly respond by adding mitigations to any future incidents of this manner.

Code has been rolled back and models are no longer failing setup due to this issue.

All queues have been processed and service should be back to normally. Sorry for the interruption folks.

We have cleared up the backlog of models seeing a slow starts.

We have identified a few nodes within one of our regions that exhibit issues with NVIDIA drivers not being installed. We have isolated these nodes from further workload scheduling (both inference and training) and will recycle the problematic nodes.

Thank you for your patience. We have cleared up the remaining backlog of pending workloads. Inference and Trainings are now running as expected for all hardware types.

The maintenance event has passed. We believe impact to Replicate customers was minimal.

We have confirmed and corrected any model versions erroneously disabled during this issue.

Use of A40s for predictions and trainings is now working as expected.

It looks to us like one of our providers had a brief outage and things are now coming back. We're continuing to monitor the situation.

(Technical details: it looks like an upstream provider had a brief DNSSEC zone signing outage.)

The slow model startup has resolved. We will continue to work internally and with our provider to remediate the root cause.

The prediction and webhook delivery issues are resolved now. There might be still a delay in webhook delivery of older predictions.

The issue has been resolved and predictions are now functioning normally.

Rollback of the problematic change has completed and Replicate website is now functioning normally again.

We have rolled out a fix and confirmed finetunes are working as expected.

Predictions and trainings are back to normal.

All done! Thanks for your patience.

We identified a problem affecting a small portion of customers -- slow responses to webhooks caused a backlog in processing outbound webhooks -- and have deployed a change to increase available webhook processing capacity. Webhook delivery is back to normal as of a few minutes ago.

We've fixed the issue and you should be able to push new versions again.

Database load is back to normal, performance should be back to usual levels.