We have identified a problematic ingress pod and have caused it to reschedule. The API and website are now working as expected for predictions and trainings.

Our API (and subsequently web-based predictions and trainings) is seeing higher than expected errors causing delays in starting predictions and trainings for some hardware types (A40 and some A100s).

We are investigating the issue.