Previous incidents

November 2024
Nov 22, 2024
1 incident

A few jobs stuck on queued and processing.

Degraded

Resolved Nov 22 at 10:33am PST

This has been resolved, will post a more detailed RCA soon.

1 previous update
Nov 10, 2024
1 incident

Jobs stuck on queued

Downtime

Resolved Nov 10 at 07:30pm PST

Issue was resolved

1 previous update
October 2024
Oct 20, 2024
1 incident

Jobs involving sieve.File objects are hanging and occasionally returning no o...

Downtime

Resolved Oct 20 at 11:00am PDT

We identified the root cause of the issue, all jobs submitted after 11 am will work. The issue was due to a massive burst of jobs which backlogged a service that uploads files during job completion.

We've resolved this by allocating more resources to the affected services to allow them to scale much higher than before.

We will reimburse all affected jobs, including some old jobs which are continuing to hang.

1 previous update
September 2024
Sep 23, 2024
1 incident

Dashboard shows "No outputs found" for certain outputs

Downtime

Resolved Sep 23 at 04:22pm PDT

This is resolved for all new jobs. Older broken jobs from the past few hours will unfortunately still be broken. We will reimburse all broken jobs during the next billing period.

1 previous update