Previous incidents

Upon further investigation we were unable to work through the backlog of predictions. Backlogged predictions have been cancelled. It will take some time before the prediction IDs report failed in the replicate web console.

Users may resubmit any of these cancelled predictions.

All predictions that have been submitted since the last update at Sep 26 2024 at 11:52pm UTC are unaffected by this cancellation of predictions.

Queues for predictions remain fairly high for black-forest-labs/flux-schnell and meta/meta-llama-3.1-405b-instruct. All other models should be behaving normally.

Things have been running normally for at least the last 10 minutes. This incident was -- ironically -- triggered by work we're doing to improve the overall performance and reliability of our primary database. We apologise for the disruption.

We were alerted to a potential issue with prediction serving. Upon investigation, one of our providers used to monitor is seeing an outage impacting some automated monitoring. We've taken steps to isolate the problematic monitors while our provider works to resolve the issue.

A40 workloads are running again. We're continuing to monitor and investigate the underlying cause.

We believe these problems have now been resolved. Please contact us if you are still seeing issues with streaming from Europe.

A40 behavior has been stable for some time now. All systems are green.

This has been resolved and predictions should be handled normally.

We've fixed the issue and predictions on trained versions are running again.

Things are running normally as of about 15 minutes ago.

Service has been restored. Thanks for your patience!

The model is processing predictions properly again, and the queue is empty.