Previous incidents

We're seeing healthy behavior since our upstream provider applied further fixes in the last hour. We will be sharing further details of how this happened once they are available.

The back log of predictions has been worked through and we are seeing normal prediction times return. Thank you for your patience.

A of 0223 UTC Nov 17th the backlog has been processed and Flux Dev is handling requests as expected.

We have identified a hardware failure and have isolated the affected node(s). We are seeing a return to normal service for H100-targeted predictions and trainings.

We identified an internal component that caused errors with Flux Pro, Recraft, and Ideogram models. The errors occurred between approximately 1540 UTC and 1709 UTC on November 11, 2024.

As of the time this status update is published, the internal component has been rolled back and we are seeing normal prediction handling for impacted models.

Between the hours of 16:30 - 19:00 UTC, predictions sent to flux-1.1-pro and meta-llama-3-70b-instruct were delayed by up to 1 hour. This was the result of a rollout of an internal component that broke a small number of models which we then rolled back. Due to the high volume of predictions handled by these two models, the backlog grew fairly quickly, but we have now caught up with ...

After evaluation the queues impacted (predictions submitted prior to migration to the alternate region) are being truncated.

You will not be billed for predictions that are dropped in this manner, however, the predictions may appear as "in process" or "queued" for a period of time until the platform automation identifies them as dropped. It is safe to cancel and/or resubmit predictions impacted in this manner.

This truncation impacts a few thousand total predictions across all models target...

Our fix has rolled out and file output streaming is now working again.

A40 scaling has returned to normal time frames.

We have moved traffic for the models impacted (H100 target hardware) back to the H100 class GPUs. Predictions and trainings targeting H100 class GPUs have returned to normal.

We're processing work without problems once again (as of about 06:30UTC) but are continuing to investigate the source of instability.

[This is a retrospective status update published at 08:00 UTC]

We identified the problem -- we had rolled out a version of cog to our serving cluster that reintroduced a bug we'd previously fixed -- and have now rolled back that change.

The underlying platform disruption has been resolved. Scaling for A100 Hardware has returned to normal.

We have deployed a permanent fix for the 503s for the files-api

We have rolled out a change that resolved the failure cases.

Our infrastructure provider is continuing to work on a network problem, but mitigations appear to be allowing workloads to flow normally.

We have removed an upstream from our API load balancer that seems to be accounting for all of the timeout errors and we are seeing an immediate improvement.

We truncated the queues, so in-flight fine-tunes will not complete and you will not be charged for them.

Fine-tunes are now working normally.

At this time all but a handful instances have recovered and prediction serving should be normal for the A40 hardware type.

We expect the remaining (low single digit) number of instances to be running within the next few minutes.

The queue is now drained, performance is back to normal levels.

We've deployed a fix. Things look to be stabilizing but we are continuing to monitor.