# Buildkite: Buildkite service disruption

- Provider: [Buildkite](https://outages.cc/buildkite.md)
- Type: Incident
- State: Resolved
- Worst impact: Partial outage
- Started: 2026-08-25 22:55 UTC
- Resolved: 2026-08-26 00:18 UTC (lasted 1 h 23 min)
- Components: REST API, Web, GitHub Commit Status Notifications, Email Notifications, Agent API, Remote MCP Server, Slack Notifications, Job Queue, Webhook Notifications
- Official update: https://stspg.io/8zqwyvss22tp

## Updates from Buildkite

### postmortem, 2026-08-27 23:53 UTC

## Service Impact Customers experienced a site-wide disruption affecting the Buildkite web interface, REST API, Agent API, job queue, and notifications. The disruption began at 22:47 UTC and continued until 23:16 UTC. During this period, all customers were unable to create new builds via the UI or REST API, and job dispatch was delayed. We continued to receive webhooks from source control providers, but processing of those webhooks was delayed. New build processing resumed at 23:10 UTC and the accumulated backlog of work was processed by 23:16 UTC. Some builds that were already in progress at the beginning of the impact period remained temporarily stuck until services recovered. Some customers also experienced errors in the web UI, and on receiving webhooks from SCM providers. This impacted <1.5% of all requests during the impact period. ## Incident Summary **Background** Buildkite services running in one of our production Kubernetes clusters depend on an internal DNS service CoreDNS t

### resolved, 2026-08-26 00:18 UTC

We're considering this incident resolved.

### monitoring, 2026-08-25 23:38 UTC

Systems have recovered and we are seeing new builds created. For a period of about 20min new builds could not be created due to a failure of our internal DNS services. No build requests were lost during this time and we are continuing to process the backlog

### investigating, 2026-08-25 23:23 UTC

We have identified a fix and applied mitigations, and we are seeing signs of recovery.

### investigating, 2026-08-25 23:05 UTC

We are currently investigating a sitewide outage.

### investigating, 2026-08-25 22:55 UTC

We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.