Degraded autoscaling performance
Status Report Update State Resolved
May 22 at 02:05pm UTC
Backlogs have been cleared and all models are now running smoothly.
Affected services
Prediction serving
Status Report Update State Updated
May 22 at 12:07pm UTC
The original issue has been resolved, but we will have elevated contention for a while as workloads that built up during the outage are processed.
Affected services
Prediction serving
Status Report Update State Updated
May 22 at 11:05am UTC
Models that run on A40 or A100 hardware are currently unable to boot up or scale out. Furthermore existing instances are suffering significant degradation and not all predictions are completing successfully in a timely manner.
We are actively monitoring the system and working with upstream providers to resolve the issue.
Affected services
Prediction serving
Status Report Update State Created
May 22 at 10:40am UTC
Models that run on A40 or A100 hardware are currently unable to boot up or scale out. Instances that are already running will continue to process predictions as normal.
We are actively monitoring the system and working with upstream providers to resolve the issue.
Affected services
Prediction serving