Buildkite service disruption
## Service Impact _All times UTC unless stated otherwise._ Customers experienced elevated errors and latency across the Buildkite web interface, REST API, Agent API, and job queue during two periods: 18:43:00–19:21:00 and 20:02:00–20:28:00 on September 9, 2026. During these periods, customers encountered failed API requests, delayed job dispatch, and errors in agent operations including authentication, job acceptance, and artifact uploads. Some jobs impacted by the incident did not recover automatically. These jobs had to be manually cancelled and retried to complete successfully. Between 18:43:00–19:21:00: Agent API had an error rate up to 11.2% with no latency impact. REST API had a 9.7% error rate with no latency impact. Web request had a 2.6% error rate with no latency impact. Job queues were delayed by up to 3 minutes. Between 20:02:00–20:28:00: Agent API had an error rate up to 24.7% with no latency impact. REST API had a 4.9% error rate with no latency impact. Web requests had a
The latest update from Buildkite.
State
Resolved
Started
53 d ago
2026-09-09 19:07 UTC
Lasted
6 h 47 min
until 2026-09-10 01:54 UTC
Worst impact
Partial outage
Timeline
6 updatesInvestigating
1 h 46 min 26%
Monitoring
5 h 1 min 74%
Updates
Times in UTCAs posted by Buildkite
## Service Impact _All times UTC unless stated otherwise._ Customers experienced elevated errors and latency across the Buildkite web interface, REST API, Agent API, and job queue during two periods: 18:43:00–19:21:00 and 20:02:00–20:28:00 on September 9, 2026. During these periods, customers encountered failed API requests, delayed job dispatch, and errors in agent operations including authentication, job acceptance, and artifact uploads. Some jobs impacted by the incident did not recover automatically. These jobs had to be manually cancelled and retried to complete successfully. Between 18:43:00–19:21:00: Agent API had an error rate up to 11.2% with no latency impact. REST API had a 9.7% error rate with no latency impact. Web request had a 2.6% error rate with no latency impact. Job queues were delayed by up to 3 minutes. Between 20:02:00–20:28:00: Agent API had an error rate up to 24.7% with no latency impact. REST API had a 4.9% error rate with no latency impact. Web requests had a
We have seen full recovery for customers since 20:28 UTC. We experienced an autoscaling feedback loop which increased the number of connections to our redis cluster above its ability to respond. This had widespread impact for all of our customers with Web UI, Agent API, REST API and job queue impact between 18:43-19:21 UTC, and again between 20:02-20:28 UTC. A full post incident review will be available later this week.
We are seeing improvements across the affected services, and are seeing services return to normal functionality. We are continuing to monitor and are determining the root cause.
We are continuing to investigate elevated error rates across multiple services, and are working to determine the cause.
We are continuing to investigate this issue. We are seeing impact on the Agent API which will affect job scheduling, artifact uploads, and an increase in 5xx responses from the Agent API endpoints.
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Affected components
4Buildkite in the last 90 days
Full historyIncidents in 90 days
26
Days with an incident
26
Median duration
1 h 9 min
This incident
5.9× the median
Other Buildkite incidents
All incidentsBuildkite·Elevated job dispatch times
We are seeing the job dispatch times are back to normal.
Buildkite·Increased latency and error rates
We identified that the majority of the impact was for a single customer and was caused by a database failover. We are investigating the cause and we will work with the impacted customer.
Buildkite·Elevated error rates
Between 23:16 and 23:27 we saw elevated error rates across our platform due to a scheduled database upgrade. That upgrade was not intended to cause any impact, but due to an unforeseen dependency on this database customers experienced error rates averages 2.5% during this period.
Buildkite·Delays with Hosted Agents jobs
Processing of Hosted Agents jobs are no longer delayed.
Buildkite·Buildkite response to the eMudhra certificate compromise
All services that power Buildkite have been updated to distrust the affected certificates, and updated releases are available for the components we distribute that run in customer infrastructure: Buildkite agent: v4.7.0 and v3.138.2 Buildkite CLI: v3.64.2 bktec: v3.3.1 elastic-ci-stack-for-aws: v7.8.0 agent-stack-k8s: v0.52.0 We recommend updating to these versions to prevent exposure to the affected certificate providers.
Follow Buildkite
Your coding agent knows when Buildkite is down, and tells you when a failure is theirs, not yours. You can also get Buildkite incidents in a feed reader or in Slack.
Install the outages.live MCP server, so that when something breaks you can tell if the cause is a provider we depend on and not my code. It reads the official status pages of cloud, AI, payment and developer providers and gives you their incidents as MCP tools.
Add it as a remote MCP server named "outages" at https://outages.cc/mcp (Streamable HTTP, no authentication), with your CLI command or your MCP config file.
Then:
1. Make sure it connects and shows the tools get_provider, list_feed and send_feedback.
2. Call get_provider with "buildkite" and tell me if Buildkite has a problem right now.
3. From now on, when a call to a third-party service fails or becomes slow, call get_provider for that service before you change my code.Paste this into any coding agent. It adds the server with its own MCP setup.
https://outages.cc/feed.xml?providers=buildkite/feed subscribe https://outages.cc/feed.xml?providers=buildkiteFAQ
Go from tracking outages to fixing production issues with Polylane.
Try Polylane