This incident is now resolved.

Most systems are now operating normally again. We are continuing to monitor the situation.

As some of you may have noticed, things got worse before they got better. When the upstream storage provider restored service, models pending setup resulted in a large bandwidth surge. We're currently managing the effects of that surge, which has affected the speed of predictions and prediction webhook delivery.

We've identified the underlying problem -- a storage outage at an upstream provider -- and are investigating paths to mitigate the impact of the upstream outage.

We're investigating an issue that's preventing some models running on L40S hardware from successfully completing setup. We'll update when we have more information.