Previous incidents

We noticed elevated error rates (500 class responses) on our API. Investigation of the errors resulted in discovering one of the APIs in the primary loadbalancer was having issues making requests to one of our serving regions.

Our engineers have temporarily removed this api endpoint from production traffic while we investigate.

The elevated error rate has returned to normal.

We are back under capacity for L40S hardware. Thanks for waiting!

We are back below capacity limits for the L40S hardware type. Thanks for your patience!

Most models running on L40S hardware should not be experiencing delays. We are still seeing a handful of models unable to setup due to download rate limiting from a few external providers, but we're going to continue working on that as a separate problem. Thanks for waiting!

All backlog is being worked through and at this point all services have been restored to full functionality.

Again thank you for your patience.

Replicate was altered to a brief issue with prediction creation, update, and completion. There was a window for about 5 minutes starting at 2025-02-24 15:22:30 UTC.

A database update caused a brief disruption causing delays in persisting data. At this time the Replicate platform has resumed normal operations.

Things have been stable for 15 minutes now. We believe this to be resolved.

Webhooks are now being delivered in a timely fashion. Thanks for your patience!

We are back at full capacity. Thanks for your patience!

The rollback of the suspected misconfiguration is complete and all queues have recovered. Thanks for your patience!

We are now caught up and running below capacity. Thanks for your patience!

This issue appears to have been a result of another bandwidth spike partly as a result of our incident earlier today. The issue has now been resolved. We are going to be working to prevent incidents of this kind from recurring.

This incident is now resolved.

The cache used by the API for predictions was misconfigured for a period of ~20 minutes beginning at 20:34 UTC until a rollback completed at 20:56 UTC. Models using the L40S and H100 hardware types were affected. During the period of misconfiguration, prediction creation was severely limited, resulting in many API responses with status 503.

The networking issue with our provider was resolved at 0940 UTC, and all requests have been running normally since then.

The background jobs are running again, and we've caught up to present as of about 1609 UTC (20 minutes ago).

The redirect has been reverted and the dashboard should be accessible again.