Previous incidents

November 2024
Nov 02, 2024
1 incident

1/1000 jobs failing due to an internal issue

Degraded

Resolved Nov 02 at 03:06pm PDT

This issue has been resolved. All affected jobs will be reimbursed. We apologize for any inconvenience caused due to this issue, and have made the appropriate checks to make sure this doesn't happen again.

1 previous update

October 2024
Oct 20, 2024
1 incident

Jobs involving sieve.File objects are hanging and occasionally returning no o...

Downtime

Resolved Oct 20 at 11:00am PDT

We identified the root cause of the issue, all jobs submitted after 11 am will work. The issue was due to a massive burst of jobs which backlogged a service that uploads files during job completion.

We've resolved this by allocating more resources to the affected services to allow them to scale much higher than before.

We will reimburse all affected jobs, including some old jobs which are continuing to hang.

1 previous update

September 2024
Sep 29, 2024
1 incident

Jobs stuck on processing, unable to process, or returning phantom statuses.

Downtime

Resolved Sep 29 at 08:26am PDT

We've pushed the fix for this, monitoring to make sure our systems are back online

1 previous update

Sep 23, 2024
1 incident

Dashboard shows "No outputs found" for certain outputs

Downtime

Resolved Sep 23 at 04:22pm PDT

This is resolved for all new jobs. Older broken jobs from the past few hours will unfortunately still be broken. We will reimburse all broken jobs during the next billing period.

1 previous update