Back to overview
Degraded

Prediction serving degraded

Sep 26 at 11:39pm UTC
Affected services
Prediction serving

Status Report Update State Resolved
Sep 27 at 12:25am UTC

Upon further investigation we were unable to work through the backlog of predictions. Backlogged predictions have been cancelled. It will take some time before the prediction IDs report failed in the replicate web console.

Users may resubmit any of these cancelled predictions.

All predictions that have been submitted since the last update at Sep 26 2024 at 11:52pm UTC are unaffected by this cancellation of predictions.

Status Report Update State Updated
Sep 26 at 11:52pm UTC

We have shifted new flux fine-tune predictions to another region while we look into the root of the slow prediction processing.

At this time previous predictions submitted for Flux fine tunes can be cancelled and resubmitted. Predictions that are not cancelled will eventually process as we work through the backlog.

We will provide further information as details become available.

Status Report Update State Created
Sep 26 at 11:39pm UTC

Seeing large queues for Flux fine tunes resulting in longer than anticipated times for inference. Currently investigating source and will provide updates.

Only affect Flux fine tunes.