Previous incidents

The background jobs are running again, and we've caught up to present as of about 1609 UTC (20 minutes ago).

The redirect has been reverted and the dashboard should be accessible again.

At 22.15 UTC Jan 13 an issue forced us to shift some GPU workloads, which caused stock outs leading to increased wait times to spin up new model instances using L40s.

The work has completed and GPUs are now available as normal as of 00.15 UTC Jan 15th.

We've caught up with prediction deletion, and our system is once again deleting predictions on time.

T4 predictions were unavailable approximately between the hours of 1800 and 2027 UTC. We found an issue with the nvidia driver installation on our T4 hardware targets.

This only affected predictions running against the T4 hardware.

We have deployed a fix and are backfilling the outstanding predictions.

We're seeing healthy behavior since our upstream provider applied further fixes in the last hour. We will be sharing further details of how this happened once they are available.

The back log of predictions has been worked through and we are seeing normal prediction times return. Thank you for your patience.

A of 0223 UTC Nov 17th the backlog has been processed and Flux Dev is handling requests as expected.

We have identified a hardware failure and have isolated the affected node(s). We are seeing a return to normal service for H100-targeted predictions and trainings.

We identified an internal component that caused errors with Flux Pro, Recraft, and Ideogram models. The errors occurred between approximately 1540 UTC and 1709 UTC on November 11, 2024.

As of the time this status update is published, the internal component has been rolled back and we are seeing normal prediction handling for impacted models.

Between the hours of 16:30 - 19:00 UTC, predictions sent to flux-1.1-pro and meta-llama-3-70b-instruct were delayed by up to 1 hour. This was the result of a rollout of an internal component that broke a small number of models which we then rolled back. Due to the high volume of predictions handled by these two models, the backlog grew fairly quickly, but we have now caught up with ...

After evaluation the queues impacted (predictions submitted prior to migration to the alternate region) are being truncated.

You will not be billed for predictions that are dropped in this manner, however, the predictions may appear as "in process" or "queued" for a period of time until the platform automation identifies them as dropped. It is safe to cancel and/or resubmit predictions impacted in this manner.

This truncation impacts a few thousand total predictions across all models target...

Our fix has rolled out and file output streaming is now working again.