We have moved traffic for the models impacted (H100 target hardware) back to the H100 class GPUs. Predictions and trainings targeting H100 class GPUs have returned to normal.

Predictions on flux-dev are now also running in a different cluster.

Predictions on flux-schell and flux fine tunes are successfully running in another cluster. Predictions on flux-dev are still not working.

We're moved flux models and fine tunes to run in a different cluster until we can get this cluster back online.

One of our clusters is currently down. We know the immediate cause, and are working on fixing it. This is the cluster that runs our H100s, so all H100 models are currently down, including flux and flux fine tunes.