Maintenance within one region
Resolved
Feb 23 at 05:01am UTC
This maintenance has concluded.
At this time all of our accelerated content delivery has been restored to full working order. There may continue to be some minor slowness loading weights as the systems complete their operational warmup. We do not expect any significant issues.
Affected services
API
Prediction serving
Updated
Feb 23 at 04:38am UTC
The first phase of the maintenance has completed. As expected, we are seeing some fallback-to-origin requests when downloading via the accelerated content delivery.
These fallbacks are being served normally with a slight increased delay.
Affected services
API
Prediction serving
Updated
Feb 23 at 04:00am UTC
Scheduled Maintenance has begun.
Affected services
API
Prediction serving
Created
Feb 23 at 04:00am UTC
We are working with one of our providers to implement mitigations based upon a series of recent outages. These mitigations will ensure that our accelerated data delivery and other associated services will be more resilient.
During this maintenance we expect some slower than normal model boot times impacting the following Hardware Types:
A40
A100-80g
Additionally a subset of A100-40G hardware types will be affected.
The slower-than normal boot times should not be wide spread. We will be making adjustments to the internal Content Delivery Systems to ensure retries (via a fallback-to-origin mechanism) are served normally.
This maintenance notification will be updated if the scheduled time or other details change.
Affected services
API
Prediction serving