We identified the root cause of the issue, all jobs submitted after 11 am will work. The issue was due to a massive burst of jobs which backlogged a service that uploads files during job completion.

We've resolved this by allocating more resources to the affected services to allow them to scale much higher than before.

We will reimburse all affected jobs, including some old jobs which are continuing to hang.

We are experiencing an issue with the high load this morning where sieve File objects are hanging in the backend. We will update with a resolution as soon as possible, and will reimburse all affected jobs