Back to overview
Degraded

H100 Hardware Queueing

Nov 06 at 06:48pm UTC
Affected services
Prediction serving

Resolved
Nov 06 at 08:34pm UTC

After evaluation the queues impacted (predictions submitted prior to migration to the alternate region) are being truncated.

You will not be billed for predictions that are dropped in this manner, however, the predictions may appear as "in process" or "queued" for a period of time until the platform automation identifies them as dropped. It is safe to cancel and/or resubmit predictions impacted in this manner.

This truncation impacts a few thousand total predictions across all models targeting H100 hardware type.

Additionally Flux Fine Tune predictions are not being truncated in this manner and will continue to process the backlog.

Updated
Nov 06 at 08:12pm UTC

All new predictions for H100-class hardware will now be routed to the alternate region.

Past predictions that are impacted by this outage may see significant delays for processing. We are working to address the large queue buildup prior to moving to the new region.

Updated
Nov 06 at 07:48pm UTC

Approximately 50% of all h100 traffic has redirected to our alternate region. We are working to shift the rest of the prediction workloads as quickly as possible.

Updated
Nov 06 at 07:29pm UTC

The impact of this incident encompasses all workloads targeting h100 hardware classes:

Flux Fine Tunes
Flux Dev (migrated; new predictions not impacted)
Flux Schnell (migrated; new predictions not impacted)
Stable Diffusion 3.5 (all variants)
bytedance/hyper-flux-16step

(list above is not all inclusive)

We are actively migrating all workloads to additional capacity to alleviate the problems. Updates will be provided as each model is migrated.

Updated
Nov 06 at 07:11pm UTC

We have moved new traffic for flux schnell to additional capacity in another region. New predictions for flux schnell will be processed within the expected timeframes. The backlog of predictions will continue to be processed.

Flux Dev traffic will be migrated soon.

Created
Nov 06 at 06:48pm UTC

We are seeing a rapid buildup of queued predictions to flux-dev and flux-schnell models.