Prediction serving degraded
Resolved
Sep 27 at 12:25am UTC
Upon further investigation we were unable to work through the backlog of predictions. Backlogged predictions have been cancelled. It will take some time before the prediction IDs report failed in the replicate web console.
Users may resubmit any of these cancelled predictions.
All predictions that have been submitted since the last update at Sep 26 2024 at 11:52pm UTC are unaffected by this cancellation of predictions.
Affected services
Prediction serving
Updated
Sep 26 at 11:52pm UTC
We have shifted new flux fine-tune predictions to another region while we look into the root of the slow prediction processing.
At this time previous predictions submitted for Flux fine tunes can be cancelled and resubmitted. Predictions that are not cancelled will eventually process as we work through the backlog.
We will provide further information as details become available.
Affected services
Prediction serving
Created
Sep 26 at 11:39pm UTC
Seeing large queues for Flux fine tunes resulting in longer than anticipated times for inference. Currently investigating source and will provide updates.
Only affect Flux fine tunes.
Affected services
Prediction serving