Back to overview
Degraded
Flux Dev Prediction Delays
Nov 17 at 02:05am UTC
Affected services
Prediction serving
Resolved
Nov 17 at 02:35am UTC
A of 0223 UTC Nov 17th the backlog has been processed and Flux Dev is handling requests as expected.
Affected services
Prediction serving
Created
Nov 17 at 02:05am UTC
Starting at about 19:42 UTC on November 16 2024 the Flux Dev model started to have significant delays in processing predictions. Over the course of ~5 hours the predictions showed a longer delay being picked up for processing. At 0159 UTC on November 17 2024, the instances were administratively restarted to provide relief for the building queue.
At this time we have added ~20 instances and the queue backlog is rapidly being processed.
Only the Flux Dev model is impacted by this issue.
Affected services
Prediction serving