[This is a retrospective status update published at 08:00 UTC]

We identified the problem -- we had rolled out a version of cog to our serving cluster that reintroduced a bug we'd previously fixed -- and have now rolled back that change.

[This is a retrospective status update published at 08:00 UTC]

Between about 00:00 UTC and 07:00 UTC on 24 Oct 2024, a small number of models will have stopped working. Predictions on these models may have errored with "Prediction timed out" or other generic errors.

We rolled out a version of cog to our serving cluster that reintroduced a bug we'd previously fixed. This change has been rolled back.

We'd like to acknowledge that this is not the first time a cog update has broken some subset of models running on Replicate. We know this isn't acceptable and we will be working to change how these rollouts work. Thank you for your patience and understanding.