After evaluation the queues impacted (predictions submitted prior to migration to the alternate region) are being truncated.

You will not be billed for predictions that are dropped in this manner, however, the predictions may appear as "in process" or "queued" for a period of time until the platform automation identifies them as dropped. It is safe to cancel and/or resubmit predictions impacted in this manner.

This truncation impacts a few thousand total predictions across all models targeting H100 hardware type.

Additionally Flux Fine Tune predictions are not being truncated in this manner and will continue to process the backlog.

All new predictions for H100-class hardware will now be routed to the alternate region.

Past predictions that are impacted by this outage may see significant delays for processing. We are working to address the large queue buildup prior to moving to the new region.

Approximately 50% of all h100 traffic has redirected to our alternate region. We are working to shift the rest of the prediction workloads as quickly as possible.

The impact of this incident encompasses all workloads targeting h100 hardware classes:

Flux Fine Tunes
Flux Dev (migrated; new predictions not impacted)
Flux Schnell (migrated; new predictions not impacted)
Stable Diffusion 3.5 (all variants)
bytedance/hyper-flux-16step

(list above is not all inclusive)

We are actively migrating all workloads to additional capacity to alleviate the problems. Updates will be provided as each model is migrated.

We have moved new traffic for flux schnell to additional capacity in another region. New predictions for flux schnell will be processed within the expected timeframes. The backlog of predictions will continue to be processed.

Flux Dev traffic will be migrated soon.

We are seeing a rapid buildup of queued predictions to flux-dev and flux-schnell models.