Previous incidents

We identified the root cause of the issue, all jobs submitted after 11 am will work. The issue was due to a massive burst of jobs which backlogged a service that uploads files during job completion.

We've resolved this by allocating more resources to the affected services to allow them to scale much higher than before.

We will reimburse all affected jobs, including some old jobs which are continuing to hang.

This is resolved for all new jobs. Older broken jobs from the past few hours will unfortunately still be broken. We will reimburse all broken jobs during the next billing period.

We have pushed a fix. This issue is resolved.

The issue has been resolved.