# Buildkite: Issues with webhook ingestion, GraphQL queries

- Provider: [Buildkite](https://outages.cc/buildkite.md)
- Type: Incident
- State: Resolved
- Worst impact: Partial outage
- Started: 2026-09-22 20:07 UTC
- Resolved: 2026-09-22 20:50 UTC (lasted 42 min)
- Official update: https://stspg.io/b9fymqh5z2nh

## Updates from Buildkite

### postmortem, 2026-09-28 04:12 UTC

## Service Impact On September 22, 2026, between 19:09 and 19:32 UTC, customers experienced failures ingesting webhooks and executing GraphQL queries. Some web application requests also failed or timed out. During this period, 2.0% of requests to our webhook ingestion endpoints and 20.3% of requests to our external GraphQL endpoint returned HTTP 500 errors. Webhook ingestion errors ceased by 19:30 UTC, and external GraphQL errors returned to background levels by 19:32 UTC. ## Incident Summary While investigating a separate API performance issue, we made a manual configuration change intended to rollback a routing change for one database proxy. The command instead changed the routing configuration for another database proxy, disrupting access to a database shard used internally by Buildkite. Some customer-facing operations search across all database shards to locate their shard, where routing information isn't available in the request. Those requests round robin through available shards

### resolved, 2026-09-22 20:50 UTC

Services are fully recovered. Webhook ingestion was affected between 19:09 UTC - 19:32 UTC.

### identified, 2026-09-22 20:26 UTC

Our engineers have addressed the issue and we're monitoring recovery.

### investigating, 2026-09-22 20:07 UTC

We're seeing an issue affecting webhook ingestion and execution of GraphQL queries and are investigating the cause.