Back to overview

Issues starting predictions

Sep 01 at 04:46pm UTC
Affected services

Sep 01 at 08:45pm UTC

Everything should be working normally at this time.

Sep 01 at 08:16pm UTC

Backlog of predictions should be cleared at this time. Engineers are rolling out a restart of an internal service to ensure all things are working as expected.

Predictions and Trainings are expected to be working normally at this time. Incident will be closed as soon as we confirm the restart has completed.

Sep 01 at 07:11pm UTC

We are still working through the backlog of problematic workloads and predictions. Errors will continue to occur in some circumstances both from web and via API.

Sep 01 at 05:00pm UTC

We have remediated the issue and now newly created predictions are working. Engineers are working to clear out the backlog of predictions that failed to start.

Sep 01 at 04:46pm UTC

The team is aware of an issue that is causing failures starting predictions. Engineers are working on correcting the root cause.

New predictions will fail to start in some cases both through web and API. General web usage is not impacted. Currently running predictions will continue to work.