Back to overview
Degraded

Delays for models on L40S hardware type

Mar 11 at 09:49pm UTC
Affected services
Prediction serving

Resolved
Mar 11 at 11:31pm UTC

We are back below capacity limits for the L40S hardware type. Thanks for your patience!

Created
Mar 11 at 09:49pm UTC

We are hitting capacity limits for the L40S hardware type, although we are also seeing that many of the running workloads are failing to setup within 10 minutes seemingly due to download throttling. This is effectively the same problem we saw yesterday, but the mitigations we saw succeeding yesterday don't appear to be working today.