# Buildkite: Buildkite service disruption

- Provider: [Buildkite](https://outages.cc/buildkite.md)
- Type: Incident
- State: Resolved
- Worst impact: Partial outage
- Started: 2026-09-09 19:07 UTC
- Resolved: 2026-09-10 01:54 UTC (lasted 6 h 47 min)
- Components: REST API, Web, Agent API, Job Queue
- Official update: https://stspg.io/s27r4b25q77q

## Updates from Buildkite

### postmortem, 2026-09-11 06:03 UTC

## Service Impact _All times UTC unless stated otherwise._ Customers experienced elevated errors and latency across the Buildkite web interface, REST API, Agent API, and job queue during two periods: 18:43:00–19:21:00 and 20:02:00–20:28:00 on September 9, 2026. During these periods, customers encountered failed API requests, delayed job dispatch, and errors in agent operations including authentication, job acceptance, and artifact uploads. Some jobs impacted by the incident did not recover automatically. These jobs had to be manually cancelled and retried to complete successfully. Between 18:43:00–19:21:00: Agent API had an error rate up to 11.2% with no latency impact. REST API had a 9.7% error rate with no latency impact. Web request had a 2.6% error rate with no latency impact. Job queues were delayed by up to 3 minutes. Between 20:02:00–20:28:00: Agent API had an error rate up to 24.7% with no latency impact. REST API had a 4.9% error rate with no latency impact. Web requests had a

### resolved, 2026-09-10 01:54 UTC

We have seen full recovery for customers since 20:28 UTC. We experienced an autoscaling feedback loop which increased the number of connections to our redis cluster above its ability to respond. This had widespread impact for all of our customers with Web UI, Agent API, REST API and job queue impact between 18:43-19:21 UTC, and again between 20:02-20:28 UTC. A full post incident review will be available later this week.

### monitoring, 2026-09-09 20:53 UTC

We are seeing improvements across the affected services, and are seeing services return to normal functionality. We are continuing to monitor and are determining the root cause.

### investigating, 2026-09-09 20:28 UTC

We are continuing to investigate elevated error rates across multiple services, and are working to determine the cause.

### investigating, 2026-09-09 19:41 UTC

We are continuing to investigate this issue. We are seeing impact on the Agent API which will affect job scheduling, artifact uploads, and an increase in 5xx responses from the Agent API endpoints.

### investigating, 2026-09-09 19:07 UTC

We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.