Incidents | Sieve

API is down. Google Cloud has an outage.

Thu, 12 Jun 2025 21:00:00 -0000

Google and Cloudflare have resolved their issues, and we have restarted our services and are back online. Apologies for any inconvenience caused.

API is down. Google Cloud has an outage.

Thu, 12 Jun 2025 21:00:00 -0000

Google and Cloudflare have resolved their issues, and we have restarted our services and are back online. Apologies for any inconvenience caused.

L4 recovered

Thu, 12 Jun 2025 20:43:39 +0000

L4 recovered

CPU recovered

Thu, 12 Jun 2025 20:42:59 +0000

CPU recovered

A100 recovered

Thu, 12 Jun 2025 20:42:50 +0000

A100 recovered

API is down. Google Cloud has an outage.

Thu, 12 Jun 2025 18:12:00 -0000

Update here, both Cloudflare and Google are down. We are in touch with the teams here and will update as soon as we can.

API is down. Google Cloud has an outage.

Thu, 12 Jun 2025 18:12:00 -0000

Update here, both Cloudflare and Google are down. We are in touch with the teams here and will update as soon as we can.

API is down. Google Cloud has an outage.

Thu, 12 Jun 2025 17:53:00 -0000

Our API is currently down, we are investigating right now

API is down. Google Cloud has an outage.

Thu, 12 Jun 2025 17:53:00 -0000

Our API is currently down, we are investigating right now

L4 went down

Thu, 12 Jun 2025 17:41:58 +0000

L4 went down

CPU went down

Thu, 12 Jun 2025 17:41:47 +0000

CPU went down

A100 went down

Thu, 12 Jun 2025 17:41:38 +0000

A100 went down

Sieve Job Processing is Degraded

Fri, 30 May 2025 02:41:00 -0000

Jobs were backed up due to an issue with our cloud provider. The immediate problem is resolved and we are now working through the backlog of jobs Issue has been resolved

Several Sieve APIs stuck in processing

Tue, 06 May 2025 01:14:00 -0000

Job processing issue has been resolved. All jobs should complete normally

Several Sieve APIs stuck in processing

Mon, 05 May 2025 23:15:00 -0000

Several Sieve APIs, including Dubbing, Lipsync, and Audio-Enhance, are currently stuck processing due to an issue with our internal file handling. We are working on a fix.

File servers overwhelmed causing dubbing, autocrop slowdowns

Tue, 08 Apr 2025 22:46:00 -0000

L4 recovered

Tue, 08 Apr 2025 19:01:36 +0000

L4 recovered

L4 went down

Tue, 08 Apr 2025 18:54:36 +0000

L4 went down

API requests have slow roundtrip time

Tue, 08 Apr 2025 09:30:00 -0000

This issue is resolved. A full RCA can be viewed here: https://docs.google.com/document/d/1h79gU9BnvsVNplL2N7n60Uwq5SLMibGhrfqbMxN66cc/edit?tab=t.0

API requests have slow roundtrip time

Tue, 08 Apr 2025 06:00:00 -0000

L4 recovered

Tue, 08 Apr 2025 01:18:47 +0000

L4 recovered

L4 went down

Tue, 08 Apr 2025 01:11:47 +0000

L4 went down

1 in 20 API requests are hanging or failing with 500s

Mon, 07 Apr 2025 22:00:00 -0000

This issue is resolved. Full RCA can be viewed here: https://docs.google.com/document/d/1KLIZIAYAuy5XgmIy8lInfxQcgT32DyIN4O58gfM3Dsg/edit?tab=t.0

L4 recovered

Sun, 06 Apr 2025 22:42:13 +0000

L4 recovered

L4 went down

Sun, 06 Apr 2025 22:26:12 +0000

L4 went down

L4 recovered

Sun, 06 Apr 2025 22:14:16 +0000

L4 recovered

L4 went down

Sun, 06 Apr 2025 21:53:13 +0000

L4 went down

L4 recovered

Sun, 06 Apr 2025 21:46:11 +0000

L4 recovered

L4 went down

Sun, 06 Apr 2025 21:28:14 +0000

L4 went down

L4 recovered

Sun, 06 Apr 2025 21:15:16 +0000

L4 recovered

L4 went down

Sun, 06 Apr 2025 19:28:17 +0000

L4 went down

1 in 20 API requests are hanging or failing with 500s

Sun, 06 Apr 2025 19:00:00 -0000

File servers overwhelmed causing dubbing, autocrop slowdowns

Sat, 05 Apr 2025 05:00:00 -0000

We've resolved the issue. Full RCA can be viewed here: https://docs.google.com/document/d/18Ch92tWJbYcSqJa88bKlvPaUCGN-KrckiHtf7TgKn_U/edit?tab=t.0

API recovered

Sat, 05 Apr 2025 01:38:00 +0000

API recovered

API went down

Sat, 05 Apr 2025 01:32:55 +0000

API went down

L4 recovered

Tue, 01 Apr 2025 06:20:22 +0000

L4 recovered

L4 went down

Tue, 01 Apr 2025 06:14:25 +0000

L4 went down

L4 recovered

Sun, 30 Mar 2025 20:24:28 +0000

L4 recovered

L4 went down

Sun, 30 Mar 2025 20:17:28 +0000

L4 went down

L4 recovered

Sat, 29 Mar 2025 20:03:29 +0000

L4 recovered

L4 went down

Sat, 29 Mar 2025 19:55:30 +0000

L4 went down

A100 recovered

Fri, 28 Mar 2025 21:30:26 +0000

A100 recovered

A100 went down

Fri, 28 Mar 2025 21:22:25 +0000

A100 went down

L4 recovered

Thu, 27 Mar 2025 16:08:23 +0000

L4 recovered

L4 went down

Thu, 27 Mar 2025 16:02:22 +0000

L4 went down

L4 recovered

Thu, 27 Mar 2025 14:41:21 +0000

L4 recovered

L4 went down

Thu, 27 Mar 2025 14:05:36 +0000

L4 went down

L4 recovered

Wed, 26 Mar 2025 15:49:22 +0000

L4 recovered

L4 went down

Wed, 26 Mar 2025 15:43:25 +0000

L4 went down

L4 recovered

Wed, 26 Mar 2025 15:30:25 +0000

L4 recovered

L4 went down

Wed, 26 Mar 2025 15:21:33 +0000

L4 went down

sieve/youtube-downloader suffering long queue times, several other functions are slow to process.

Thu, 13 Mar 2025 14:10:00 -0000

We've root caused and fixed the issue, the full RCA is written below: -------- Root Cause Analysis (RCA) - March 13, 2025 Incident Summary: On March 13, 2025, between 2:30 AM - 5:30 AM PST, a large push of jobs by our internal team to the sieve/youtube-downloader function caused a queue buildup, stalling customer requests to this function. This overwhelmed the servers responsible for video file CRUD operations, affecting multiple functions that generate and output video files. The issue was mitigated by 7:00 AM PST after manual intervention. Timeline: 2:30 AM PST - Internal team initiated a large push of jobs to sieve/youtube-downloader. 3:00 AM PST - Queue buildup began, impacting file handling servers. 4:00 AM PST - Degradation in video file processing observed. 5:30 AM PST - Alerts triggered, issue identified as an internal push causing excessive load. 6:00 AM PST - Large internal push was removed from the queue to allow customer jobs to process. 7:00 AM PST - Additional manual scaling applied to video file storage services; system returned to normal operation. Root Cause: A sudden and unexpected influx of jobs from an internal team overloaded the queue, leading to excessive demand on video file CRUD operations. The scaling mechanism for video file storage servers was not aggressive enough to handle the spike in demand. Alerts were delayed as the load originated from an internal push rather than customer requests. Resolution & Mitigation: Queue Separation: Segregating internal and customer job queues to prevent internal pushes from affecting customer processing. This has already been implemented. Improved Scaling: Implementing more aggressive auto-scaling policies for video file storage servers to handle sudden spikes. Will be implemented by March 25th. Enhanced Monitoring & Alerting: Refining alerting mechanisms to detect large internal job influxes earlier. This is done. Setting up specific monitoring for queue buildup to ensure proactive mitigation. This is done. Internal Process Changes: Implementing guidelines for internal teams to coordinate with infrastructure teams before large job pushes. This is done. Conclusion: This incident on March 13, 2025, was caused by an internal job push overwhelming video file handling services. While the issue was resolved through manual intervention, long-term mitigations including queue separation, improved scaling, and better monitoring are being implemented to prevent recurrence.

sieve/youtube-downloader suffering long queue times, several other functions are slow to process.

Thu, 13 Mar 2025 12:34:00 -0000

We're looking into why this is happening, will keep you posted. We believe this started at around 2:30 - 3 am PDT.

Jobs Unable to Process

Fri, 07 Feb 2025 12:19:00 -0000

Root Cause Analysis (RCA) Incident Summary: On [date/time of the incident], our services experienced a disruption due to a database service undergoing automatic maintenance. This maintenance reset the DNS configuration, causing our systems to lose connection to the database. The issue persisted because the affected services did not reestablish the connection properly. Additionally, certain users experienced degraded performance for approximately 3 hours following the incident. These users had concurrency limits in place and continued pushing jobs at their normal throughput levels. Due to the disruption, a backlog of jobs built up, and our systems did not automatically increase their concurrency limits to help process the queue more efficiently. Resolution: Our team quickly identified the root cause of the database connection issue, restored service functionality, and worked to process the backlog of jobs. Preventative Actions: To prevent similar incidents in the future: We are transitioning automated maintenance windows for critical database services to manual scheduling to minimize unexpected changes. We are enhancing our internal service mechanisms to better handle connection reestablishment in the event of similar disruptions. We are implementing improvements to automatically adjust concurrency limits during recovery periods to ensure backlogs are cleared more quickly. We sincerely apologize for the inconvenience caused and appreciate your understanding as we take these steps to ensure greater reliability and performance. If you have further questions or concerns, please don't hesitate to reach out to our support team. Thank you for your continued trust.

Jobs Unable to Process

Fri, 07 Feb 2025 11:15:00 -0000

Our job processing and monitoring infrastructure is down, we are working on resolving it urgently.

API is unavailable

Fri, 06 Dec 2024 00:09:00 -0000

The issue is resolved. It was due to an error caused by our DB provider when performing a standard DB operation. We are contacting them to ensure this never happens again, and apologize for any inconvenience caused.

API is unavailable

Fri, 06 Dec 2024 00:09:00 -0000

API is unavailable

Fri, 06 Dec 2024 00:09:00 -0000

API is unavailable

Thu, 05 Dec 2024 23:55:00 -0000

Our API was down due to a failure with our DB provider. We are working urgently to fix this and will provide an update shortly

API is unavailable

Thu, 05 Dec 2024 23:55:00 -0000

Our API was down due to a failure with our DB provider. We are working urgently to fix this and will provide an update shortly

API is unavailable

Thu, 05 Dec 2024 23:55:00 -0000

Our API was down due to a failure with our DB provider. We are working urgently to fix this and will provide an update shortly

A few jobs stuck on queued and processing.

Fri, 22 Nov 2024 18:33:00 -0000

This has been resolved, will post a more detailed RCA soon.

A few jobs stuck on queued and processing.

Fri, 22 Nov 2024 18:30:00 -0000

We've noticed <30 jobs total that were stuck on either queued or processing. This was due to an internal service maintenance that lasted a few seconds. Users should not have been billed.

Jobs stuck on queued

Mon, 11 Nov 2024 03:30:00 -0000

Issue was resolved

Jobs stuck on queued

Mon, 11 Nov 2024 02:30:00 -0000

We are actively investigating an issue where jobs were not processing between 6:30 and 7:30 PM PDT Sunday Nov 10th. This issue has been resolved and jobs appear to be running.

Jobs involving sieve.File objects are hanging and occasionally returning no outputs

Sun, 20 Oct 2024 18:00:00 -0000

We identified the root cause of the issue, all jobs submitted after 11 am will work. The issue was due to a massive burst of jobs which backlogged a service that uploads files during job completion. We've resolved this by allocating more resources to the affected services to allow them to scale much higher than before. We will reimburse all affected jobs, including some old jobs which are continuing to hang.

Jobs involving sieve.File objects are hanging and occasionally returning no outputs

Sun, 20 Oct 2024 17:40:00 -0000

We are experiencing an issue with the high load this morning where sieve File objects are hanging in the backend. We will update with a resolution as soon as possible, and will reimburse all affected jobs

Dashboard shows "No outputs found" for certain outputs

Mon, 23 Sep 2024 23:22:00 -0000

This is resolved for all new jobs. Older broken jobs from the past few hours will unfortunately still be broken. We will reimburse all broken jobs during the next billing period.

Dashboard shows "No outputs found" for certain outputs

Mon, 23 Sep 2024 23:22:00 -0000

This is resolved for all new jobs. Older broken jobs from the past few hours will unfortunately still be broken. We will reimburse all broken jobs during the next billing period.

Dashboard shows "No outputs found" for certain outputs

Mon, 23 Sep 2024 23:22:00 -0000

This is resolved for all new jobs. Older broken jobs from the past few hours will unfortunately still be broken. We will reimburse all broken jobs during the next billing period.

Dashboard shows "No outputs found" for certain outputs

Mon, 23 Sep 2024 23:04:00 -0000

A recent push today caused certain jobs to show "No Outputs Found" in the dashboard upon completion. We are working on the fix now and it will be out in a few minuts

Dashboard shows "No outputs found" for certain outputs

Mon, 23 Sep 2024 23:04:00 -0000

A recent push today caused certain jobs to show "No Outputs Found" in the dashboard upon completion. We are working on the fix now and it will be out in a few minuts

Dashboard shows "No outputs found" for certain outputs

Mon, 23 Sep 2024 23:04:00 -0000

A recent push today caused certain jobs to show "No Outputs Found" in the dashboard upon completion. We are working on the fix now and it will be out in a few minuts

Jobs stuck on processing

Fri, 23 Aug 2024 06:32:00 -0000

We have pushed a fix. This issue is resolved.

Jobs stuck on processing

Wed, 21 Aug 2024 23:31:00 -0000

We noticed an issue were certain jobs were preempted but not resumed. We are pushing a fix for the issue. Users will not be charged for these jobs.

Jobs with child jobs may sporadically hang

Wed, 22 May 2024 04:50:00 -0000

We have identified and pushed a fix at 9:58pm PST so future jobs will not be affected, in addition with remedying many of the existing jobs. Customers will not be billed for any extra compute used during this time period.

Jobs with child jobs may sporadically hang

Tue, 21 May 2024 19:43:00 -0000

For jobs that called several child jobs, we noticed an issue starting at 12:43pm PST that would happen due to preemption-based retries in children not being properly caught in the parent.