Previous incidents

December 2024
Dec 05, 2024
1 incident

API is unavailable

Downtime

Resolved Dec 05 at 04:09pm PST

The issue is resolved. It was due to an error caused by our DB provider when performing a standard DB operation. We are contacting them to ensure this never happens again, and apologize for any inconvenience caused.

1 previous update

November 2024
Nov 22, 2024
1 incident

A few jobs stuck on queued and processing.

Degraded

Resolved Nov 22 at 10:33am PST

This has been resolved, will post a more detailed RCA soon.

1 previous update

Nov 10, 2024
1 incident

Jobs stuck on queued

Downtime

Resolved Nov 10 at 07:30pm PST

Issue was resolved

1 previous update

October 2024
Oct 20, 2024
1 incident

Jobs involving sieve.File objects are hanging and occasionally returning no o...

Downtime

Resolved Oct 20 at 11:00am PDT

We identified the root cause of the issue, all jobs submitted after 11 am will work. The issue was due to a massive burst of jobs which backlogged a service that uploads files during job completion.

We've resolved this by allocating more resources to the affected services to allow them to scale much higher than before.

We will reimburse all affected jobs, including some old jobs which are continuing to hang.

1 previous update