# CircleCI: Delays in pipelines, workflows, UI data, and notifications - Provider: [CircleCI](https://outages.cc/circleci.md) - Type: Incident - State: Resolved - Worst impact: Partial outage - Started: 2026-10-06 16:00 UTC - Resolved: 2026-10-06 21:18 UTC (lasted 5 h 18 min) - Components: Pipelines & Workflows, Notifications & Status Updates - Official update: https://stspg.io/s7tpg47srxdd ## Updates from CircleCI ### postmortem, 2026-10-07 22:08 UTC ## Summary On October 6, 2026, from 15:30 to 20:30 UTC, CircleCI customers experienced delays creating pipelines and starting workflows and jobs. From 15:48 to 16:38 UTC, no workflows or jobs started, and pipelines triggered from 16:02 UTC failed. Shorter, intermittent delays began earlier, at 13:30 UTC. Data in the CircleCI UI, notifications, and commit status updates to version control providers were also delayed. The incident started in the database behind our workflow orchestration service, which coordinates every pipeline, workflow and job on CircleCI. Routine maintenance on the database's largest tables wrote transaction logs faster than the database could archive them, and the logs filled the storage volume set aside for them. To keep the database from becoming read-only, our engineers moved the logs to the database's main storage volume. That move took the database offline for 50 minutes. When the database came back, it processed work at about half its normal rate until our eng ### resolved, 2026-10-06 21:18 UTC Between approximately 15:30 UTC and 20:30 UTC on October 6, 2026, customers experienced delays and failures affecting pipelines, workflows, and jobs. From approximately 16:02 UTC to 16:39 UTC, pipelines were failing and no new workflows or jobs could start. After service resumed, workflows and jobs continued to start with delays until approximately 20:00 UTC. Start times averaged up to approximately 40 minutes at their peak, and a very small number of workflows waited more than 1 hour. Outbound notifications and webhooks were also delayed until approximately 19:00 UTC, typically by about a minute. As the backlog of delayed work cleared, the resulting increase in demand placed pressure on our execution capacity, causing further delays for Docker and Linux jobs. From approximately 19:45 UTC to 20:30 UTC, these jobs waited in the queue longer than usual before starting. At the peak, around 20:05 UTC, waits reached up to approximately 8 minutes for 2xlarge gen2, 6 minutes for xlarge gen2, ### monitoring, 2026-10-06 20:31 UTC Workflows and jobs are starting normally, and wait times for Docker and Linux jobs have returned to normal. Our engineers are continuing to monitor this issue. If your job failed to start, please rerun the job. We thank you for your patience while our engineers work on this. We will share another update within the next 30 minutes, or sooner if more information becomes available. ### identified, 2026-10-06 20:13 UTC The backlog of delayed work has been cleared, and workflows are starting again with no delays in notifications. However, customers using Docker and Linux jobs across multiple resource classes, including medium, large, and 2xlarge gen2, may now see jobs waiting in the queue for few minutes before they start. Our engineers are continuing to work on mitigating this issue. If your job failed to start, please rerun the job. We thank you for your patience while our engineers work on this. We will share another update within the next 30 minutes, or sooner if more information becomes available. ### identified, 2026-10-06 19:51 UTC Delays for workflows and jobs to start are continuing to decrease. Most are now starting within approximately 10 minutes, improved from approximately 40 minutes earlier in the incident. A very small number of workflows may have waited more than 1 hour. If your job failed to start, please rerun the job. Delays in outbound notifications have improved. The remaining delay is due to a backlog of work that built up during the incident, and that backlog has reduced by more than half. We expect the backlog to be cleared in approximately 15 minutes. We will share another update once it has cleared, or within the next 30 minutes at the latest. ### identified, 2026-10-06 19:16 UTC Customers continue to experience delays. Most are now starting within approximately 20 minutes, improved from approximately 40 minutes earlier in the incident. A very small number of workflows may have waited more than 1 hour. Our engineers have made changes that have improved service stability, and the remaining delay is due to a backlog of work that built up during the incident. The backlog is decreasing, and we are focused on clearing it faster. We will share more information as progress continues. We thank you for your patience while our engineers work through the backlog. We will share another update within the next 30 minutes, or sooner if more information becomes available. ### identified, 2026-10-06 18:55 UTC Customers continue to experience longer than usual delays before workflows and jobs start, in some cases up to approximately 30 minutes. This is improved from earlier in the incident, but delays remain. Delays in outbound notifications have improved. We thank you for your patience while our engineers continue to work toward restoring normal job start times. We will share another update within the next 30 minutes, or sooner if more information becomes available. ### identified, 2026-10-06 18:04 UTC Customers continue to experience longer than usual delays between jobs in a workflow, as well as delays in outbound notifications. We thank you for your patience while our engineers continue to investigate and work on mitigating the issue. We will share another update within the next 30 minutes, or sooner if more information becomes available. ### identified, 2026-10-06 17:21 UTC While Pipelines, workflows, and jobs are running again, customers may continue to see longer than usual delays between jobs in a workflow, and outbound notifications are delayed by approximately 1 minute. Notifications and status updates to your version control system are beginning to go out again. Our engineers are continuing to work toward a full recovery. ### identified, 2026-10-06 16:42 UTC As of 16:39 UTC, we are starting to see recovery for new pipelines and workflows being submitted, and jobs are beginning to run again. Our engineers are continuing to work on the recovery and will share more information as soon as we have it. ### identified, 2026-10-06 16:31 UTC As of 16:02 UTC, pipelines are failing entirely, so no new workflows or jobs are running at this point. This is in addition to the delays and failed jobs reported earlier. Our engineers are continuing to work on mitigating the issue, and we will share more information as soon as we have it. ### investigating, 2026-10-06 16:10 UTC Since 15:47 UTC, customers may also see failed jobs, in addition to the delays previously reported. Our engineers are continuing to investigate and mitigate the issue. ### investigating, 2026-10-06 16:00 UTC What's impacted Customers using CircleCI pipelines and workflows, including UI data display, outbound notifications, and outbound webhooks. What can you expect Delays in pipelines being created and in workflows and jobs starting. Job, workflow, and pipeline data may take up to 15 minutes to appear in the UI. Outbound notifications and webhooks are delayed by up to 1 minute. Next update We will provide another update within 30 minutes or as we have more information to share.