Previous incidents

The issue is resolved. It was due to an error caused by our DB provider when performing a standard DB operation. We are contacting them to ensure this never happens again, and apologize for any inconvenience caused.

This has been resolved, will post a more detailed RCA soon.

Issue was resolved

We identified the root cause of the issue, all jobs submitted after 11 am will work. The issue was due to a massive burst of jobs which backlogged a service that uploads files during job completion.

We've resolved this by allocating more resources to the affected services to allow them to scale much higher than before.

We will reimburse all affected jobs, including some old jobs which are continuing to hang.