LiveKit

This incident has been resolved. Between 07:10 and 07:43 UTC on October 31, a software bug caused media servers in several regions to restart repeatedly. During this window: - Participants connected to affected servers were disconnected. Impact was heaviest in US East, US Central, US West, Canada and Mexico, where about 18% of participants were disconnected at least once. Many were disconnected more than once, as automatic reconnects landed on servers that then restarted as well. A small share of participants in Europe Central, Europe South, UAE, Saudi Arabia and Israel were also disconnected. Most disconnected participants reconnected automatically, about half within 5 seconds, but each reconnect started a new session rather than resuming the existing one, so participants may have seen repeated audio and video interruptions. Some participants did not reconnect. Participants joining during this window may also have taken longer to connect or failed to join. - Most agents reconnected anstatus.livekit.io

## Summary On October 28, 2026, a network disruption at our cloud provider affected our US East data center. Starting at 23:28 UTC, services in US East lost the ability to reach our database and certain external services, such as model providers. Connections to end users and to our other regions kept working, so realtime media continued to flow. This caused widespread failures across SIP calls, Room Service, Egress and Ingress APIs, and Cloud Agents in US East. A related issue affected our Mexico region for longer. Connectivity in US East came back at around 23:58 UTC. Mexico recovered at 01:01 UTC after we moved its traffic to nearby regions. Our automatic failover should have moved traffic away from US East, and it didn't. That's the main thing we're fixing. We sincerely apologize to customers whose calls and agents were disrupted. ## Impact **US East \(23:28 to 23:58 UTC\)** * **SIP inbound:** a large share of inbound calls failed, rejected with 500 or 503 responses. Answered inbounstatus.livekit.io

Between 21:30 and 23:17 UTC, roughly 10% of all WebRTC connections in our US East and US Central regions took 5 seconds or longer to establish. We applied a mitigation at 23:15 UTC, and join latencies have returned to normal by 23:17 UTC. We will follow up with more details on the root cause and preventive actions in a postmortem. We apologize for the impact and appreciate your patience.status.livekit.io

## Summary On October 19, 2026, between 16:34 and 17:03 UTC and again between 17:47 and 17:54 UTC, some room management, agent dispatch and SIP API requests failed for rooms hosted in our US Central \(Chicago\) region. Requests from other regions that targeted those rooms also failed. During these windows some rooms could not be created or joined, and some agents were not dispatched. The cause was CPU exhaustion on the node running one of our message bus instances in US Central. A recently deployed overlay network used more CPU than it requested, and because our monitoring tracked CPU per pod rather than per node, we didn't see the node run out. That message bus instance began dropping messages, and a room-coordination service that depended on it stopped completing requests. We have moved the message bus in US Central onto larger nodes and resized other regions at similar risk. Node-level CPU alerting and automatic recovery for the coordination service are in progress. ## Impact Betweestatus.livekit.io

**Root Cause** On 2026-10-16, a subset of SIP calls in our US West region failed. Two of our SIP servers advertised IPv6 addresses for signaling and media, which many carrier providers don't support. Carriers rejected outbound calls routed through these servers, and inbound calls were answered but dropped almost immediately without audio. We had recently started running SIP in a US West cluster that uses both IPv4 and IPv6 networking and our SIP servers didn't enforce IPv4 addresses. **Technical Details** When a SIP server starts, it determines its public address by querying an external address-lookup service. On the two affected servers, our primary lookup service didn't respond in time so they fell back to a backup service that returned their IPv6 address instead. Nothing in the SIP server checked that the address it advertised was IPv4. **Timeline \(UTC\)** 2026-10-16 14:49 - First affected SIP server started with an IPv6 address; call failures began 2026-10-16 15:41 - Second affectstatus.livekit.io

This incident is resolved. Connections to the Frankfurt region have returned to normal.status.livekit.io

**Root Cause** During a security rotation of access credentials, we revoked a storage credential that we believed was no longer in use. It was still being used by our observability platform, which stores the data behind the dashboard and the Analytics API, the Agent Insights data that agents upload at the end of each session, and by the service that serves that data back for downloads and Agent Simulations. Once the credential was revoked, uploads failed in all regions. Restoring service took longer than a credential swap normally would for three reasons: * New credentials had to be set up, and each affected service had to be updated to use them. * The new credentials took longer than expected to become available in all regions \(about two hours in our US East region\). * Once the observability platform was working again, we found additional services that also needed updating. The incident was resolved once they were updated. **Timeline \(UTC\)** * 2026-10-13 08:13 - A storage credentistatus.livekit.io

A fix was applied and all builds are back to normal.status.livekit.io

This incident has been resolved, and we have confirmed full recovery.status.livekit.io

Please see a full postmortem [here](https://status.livekit.io/incidents/dbqb4jhxcg9h).status.livekit.io

### Summary On 15 September, LiveKit Cloud experienced three periods of degraded performance between 13:34–13:41, 16:06–16:11 and 21:10–21:22 UTC. Ingress creation was the most affected: more than half of `CreateIngress` requests failed at the worst point. A number of in-progress recordings ended prematurely, and a small percentage of inbound SIP calls failed during a one-minute period \(during each impact window\). Realtime connections were not affected and media already flowing continued normally. The cause was a dropped connection to our primary metadata database that neither the client nor server side detected, which left a transaction open and holding row locks for up to 17 minutes. Requests queued behind those locks, and the sudden release of that backlog — not the wait itself — is what briefly degraded the wider platform. ### Root Cause Our services share a distributed global metadata database that stores ingress and egress state. At the time of the incident, a connection betweestatus.livekit.io

This incident has been resolved. Between 12:08 and 14:23 UTC, customers were unable to access the Cloud Dashboard and the Analytics Services APIs. The cause was a control plane failure in our last remaining legacy Kubernetes cluster, which took out a dependency of those services. Our user-facing services (RTC, SIP, and others) were unaffected, as they run redundantly across our modern clusters and have no dependency on the legacy cluster. We have no indication of data loss. Recovery took longer than we would like, as mitigation required working within the legacy cluster. Work was already underway to make the affected dependency redundant across our modern clusters, and that cluster is scheduled for decommissioning, which removes this class of failure. We will follow up with a postmortem.status.livekit.io

**Root Cause** A configuration change introduced a formatting error in the startup settings for the server pools that run hosted agents in our eu-central region. Newly provisioned servers failed to start, so the region could not add capacity. Agents that were already running were unaffected, but new agent deployments, version rollouts, and automatic scale-ups were delayed or stuck pending until the change was reverted. **Timeline \(UTC\)** `2026-09-04 23:56` - Configuration change deployed. The error only affected newly provisioned servers, so there was no immediate impact. `2026-09-05 18:07` - New servers began failing to start. Impact begins, intermittent at first. `2026-09-05 22:23` - Our monitoring alerted us to agent deployments stuck pending and we began investigating. `2026-09-06 00:50` - We identified the malformed startup configuration and reverted the change. `2026-09-06 01:17` - New capacity provisioned successfully, all pending deployments recovered, and we validated the fistatus.livekit.io

This incident has been resolved. We will follow up with a detailed postmortem as soon as possible.status.livekit.io

This incident has been resolved and dashboard load times returned to normal by 10:55 UTC. As a note, our beta agent simulations feature was also impacted from approximately 06:07 - 07:52 UTC.status.livekit.io

Between 08:47 and 08:54 UTC, the service that routes LLM, STT and TTS requests through LiveKit Inference in our US East region was unavailable, causing inference requests from agents in that region to fail or time out. Sessions depending on those responses may have degraded or ended prematurely. Replacement capacity came online at 08:54 UTC and request volumes returned fully to normal levels by 09:02 UTC. We have confirmed no further impact since then. The cause was a loss of capacity during automated infrastructure maintenance. We have identified fixes to prevent a recurrence.status.livekit.io