Everything should be working normally at this time.

Backlog of predictions should be cleared at this time. Engineers are rolling out a restart of an internal service to ensure all things are working as expected.

Predictions and Trainings are expected to be working normally at this time. Incident will be closed as soon as we confirm the restart has completed.

We are still working through the backlog of problematic workloads and predictions. Errors will continue to occur in some circumstances both from web and via API.

We have remediated the issue and now newly created predictions are working. Engineers are working to clear out the backlog of predictions that failed to start.

The team is aware of an issue that is causing failures starting predictions. Engineers are working on correcting the root cause.

New predictions will fail to start in some cases both through web and API. General web usage is not impacted. Currently running predictions will continue to work.