Increased latency and error rates
Buildkite reports increased latency and error rates that affect the Job Queue for a subset of customers. Buildkite patched the webhook-processing deployment and processing lag has dropped. Buildkite is monitoring recovery as the remaining backlog clears.
State
Monitoring
Started
Ongoing for
1 h 4 min
Worst impact
Partial outage
Timeline
2 updatesInvestigating
22 min 34%
Monitoring
42 min 66%
Updates
Times in UTCAs posted by Buildkite
Delays in processing incoming webhooks and creating builds are improving for affected customers. We patched the webhook-processing deployment to increase capacity and processing lag has dropped. We are monitoring recovery as the remaining backlog clears.
We're still seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.
Affected components
1Buildkite in the last 90 days
Full historyIncidents in 90 days
30
Days with an incident
27
Median duration
1 h 10 min
This incident
About the median
Other Buildkite incidents
All incidentsBuildkite·Partially Degraded Service
Buildkite reported partially degraded service for its GitHub component. Buildkite later reported that the issue is resolved. The incident lasted 99 minutes.
Buildkite·Elevated API error rates
Buildkite reported elevated API error rates. The REST API and the Agent API had a partial outage. Buildkite reports that the incident is resolved. The outage lasted about 33 minutes.
Buildkite·Increased latency and error rates
Buildkite reported increased latency and error rates for a small subset of customers. Buildkite said the incident is resolved and latency returned to normal. The incident lasted about 2 hours and 8 minutes.
Buildkite·Elevated job dispatch times
We are seeing the job dispatch times are back to normal.
Buildkite·Increased latency and error rates
We identified that the majority of the impact was for a single customer and was caused by a database failover. We are investigating the cause and we will work with the impacted customer.
Follow Buildkite
Your coding agent knows when Buildkite is down, and tells you when a failure is theirs, not yours. You can also get Buildkite incidents in a feed reader or in Slack.
Install the outages.live MCP server, so that when something breaks you can tell if the cause is a provider we depend on and not my code. It reads the official status pages of cloud, AI, payment and developer providers and gives you their incidents as MCP tools.
Add it as a remote MCP server named "outages" at https://outages.cc/mcp (Streamable HTTP, no authentication), with your CLI command or your MCP config file.
Then:
1. Make sure it connects and shows the tools get_provider, list_feed and send_feedback.
2. Call get_provider with "buildkite" and tell me if Buildkite has a problem right now.
3. From now on, when a call to a third-party service fails or becomes slow, call get_provider for that service before you change my code.Paste this into any coding agent. It adds the server with its own MCP setup.
https://outages.cc/feed.xml?providers=buildkite/feed subscribe https://outages.cc/feed.xml?providers=buildkiteFAQ
Go from tracking outages to fixing production issues with Polylane.
Try Polylane