Investigating reports of failed SIP Transfer calls in US East
## Summary On 17 August, between 17:24 and 18:05 UTC, a small percentage of SIP call transfers failed in our US East region during a routine configuration rollout. An issue in the automation that manages our SIP signaling servers prevented outgoing servers from being taken out of service safely, and those servers shut down while calls were still active on them. Transfers have completed normally since 18:05 UTC. Calls themselves stayed connected, and no other region was affected. ## Root Cause The configuration change restarts the servers that handle SIP signaling one at a time. Before a server is shut down, it is removed from service so that no new calls reach it, and it is then given some time to finish the calls it is already handling. In this case that removal did not complete expectedly and the outgoing servers kept receiving new calls for the entire drain duration, then shut down on schedule with calls still active on them. ## Timeline \(UTC\) * 16:52 - A routine configuration rol
The latest update from LiveKit.
State
Resolved
Started
76 d ago
2026-08-17 18:10 UTC
Lasted
1 h 12 min
until 2026-08-17 19:22 UTC
Worst impact
Degraded
Timeline
4 updatesInvestigating
1 h 12 min 100%
Updates
Times in UTCAs posted by LiveKit
## Summary On 17 August, between 17:24 and 18:05 UTC, a small percentage of SIP call transfers failed in our US East region during a routine configuration rollout. An issue in the automation that manages our SIP signaling servers prevented outgoing servers from being taken out of service safely, and those servers shut down while calls were still active on them. Transfers have completed normally since 18:05 UTC. Calls themselves stayed connected, and no other region was affected. ## Root Cause The configuration change restarts the servers that handle SIP signaling one at a time. Before a server is shut down, it is removed from service so that no new calls reach it, and it is then given some time to finish the calls it is already handling. In this case that removal did not complete expectedly and the outgoing servers kept receiving new calls for the entire drain duration, then shut down on schedule with calls still active on them. ## Timeline \(UTC\) * 16:52 - A routine configuration rol
We have identified the root cause of both spikes and have ensured safeguards to prevent a recurrence. We've been monitoring since 18:05 UTC and have seen no further transfer failures, and have confirmed that SIP transfers are operating normally. We will follow up with a detailed post-mortem.
No transfer failures has been observed since 18:05 UTC and SIP transfers are currently completing normally. Impact was limited to a small percentage of calls between 15:48-16:08 UTC and 17:24-18:05 UTC. We're actively monitoring while we investigate the cause and put safeguards in place to prevent a recurrence.
We're investigating an elevated rate of failures when transferring active SIP calls in the US East region. SIP calls themselves remain connected, and inbound and outbound calling are otherwise operating normally.
Affected components
1LiveKit in the last 90 days
Full historyIncidents in 90 days
27
Days with an incident
25
Median duration
1 h 26 min
This incident
16% shorter than the median
Other LiveKit incidents
All incidentsLiveKit·Disconnections and failed connections in multiple regions
This incident has been resolved. Between 07:10 and 07:43 UTC on October 31, a software bug caused media servers in several regions to restart repeatedly. During this window: - Participants connected to affected servers were disconnected. Impact was heaviest in US East, US Central, US West, Canada and Mexico, where about 18% of participants were disconnected at least once. Many were disconnected more than once, as automatic reconnects landed on servers that then restarted as well. A small share of participants in Europe Central, Europe South, UAE, Saudi Arabia and Israel were also disconnected. Most disconnected participants reconnected automatically, about half within 5 seconds, but each reconnect started a new session rather than resuming the existing one, so participants may have seen repeated audio and video interruptions. Some participants did not reconnect. Participants joining during this window may also have taken longer to connect or failed to join. - Most agents reconnected an
LiveKit·Investigating failing calls and RoomService API failures in US East and dashboard issues
## Summary On October 28, 2026, a network disruption at our cloud provider affected our US East data center. Starting at 23:28 UTC, services in US East lost the ability to reach our database and certain external services, such as model providers. Connections to end users and to our other regions kept working, so realtime media continued to flow. This caused widespread failures across SIP calls, Room Service, Egress and Ingress APIs, and Cloud Agents in US East. A related issue affected our Mexico region for longer. Connectivity in US East came back at around 23:58 UTC. Mexico recovered at 01:01 UTC after we moved its traffic to nearby regions. Our automatic failover should have moved traffic away from US East, and it didn't. That's the main thing we're fixing. We sincerely apologize to customers whose calls and agents were disrupted. ## Impact **US East \(23:28 to 23:58 UTC\)** * **SIP inbound:** a large share of inbound calls failed, rejected with 500 or 503 responses. Answered inboun
LiveKit·Increased join latency in US East and US Central
Between 21:30 and 23:17 UTC, roughly 10% of all WebRTC connections in our US East and US Central regions took 5 seconds or longer to establish. We applied a mitigation at 23:15 UTC, and join latencies have returned to normal by 23:17 UTC. We will follow up with more details on the root cause and preventive actions in a postmortem. We apologize for the impact and appreciate your patience.
LiveKit·Investigating reports of elevated API failures in Chicago region
## Summary On October 19, 2026, between 16:34 and 17:03 UTC and again between 17:47 and 17:54 UTC, some room management, agent dispatch and SIP API requests failed for rooms hosted in our US Central \(Chicago\) region. Requests from other regions that targeted those rooms also failed. During these windows some rooms could not be created or joined, and some agents were not dispatched. The cause was CPU exhaustion on the node running one of our message bus instances in US Central. A recently deployed overlay network used more CPU than it requested, and because our monitoring tracked CPU per pod rather than per node, we didn't see the node run out. That message bus instance began dropping messages, and a room-coordination service that depended on it stopped completing requests. We have moved the message bus in US Central onto larger nodes and resized other regions at similar risk. Node-level CPU alerting and automatic recovery for the coordination service are in progress. ## Impact Betwee
LiveKit·Degraded SIP Performance in US West
**Root Cause** On 2026-10-16, a subset of SIP calls in our US West region failed. Two of our SIP servers advertised IPv6 addresses for signaling and media, which many carrier providers don't support. Carriers rejected outbound calls routed through these servers, and inbound calls were answered but dropped almost immediately without audio. We had recently started running SIP in a US West cluster that uses both IPv4 and IPv6 networking and our SIP servers didn't enforce IPv4 addresses. **Technical Details** When a SIP server starts, it determines its public address by querying an external address-lookup service. On the two affected servers, our primary lookup service didn't respond in time so they fell back to a backup service that returned their IPv6 address instead. Nothing in the SIP server checked that the address it advertised was IPv4. **Timeline \(UTC\)** 2026-10-16 14:49 - First affected SIP server started with an IPv6 address; call failures began 2026-10-16 15:41 - Second affect
Follow LiveKit
Your coding agent knows when LiveKit is down, and tells you when a failure is theirs, not yours. You can also get LiveKit incidents in a feed reader or in Slack.
Install the outages.live MCP server, so that when something breaks you can tell if the cause is a provider we depend on and not my code. It reads the official status pages of cloud, AI, payment and developer providers and gives you their incidents as MCP tools.
Add it as a remote MCP server named "outages" at https://outages.cc/mcp (Streamable HTTP, no authentication), with your CLI command or your MCP config file.
Then:
1. Make sure it connects and shows the tools get_provider, list_feed and send_feedback.
2. Call get_provider with "livekit" and tell me if LiveKit has a problem right now.
3. From now on, when a call to a third-party service fails or becomes slow, call get_provider for that service before you change my code.Paste this into any coding agent. It adds the server with its own MCP setup.
https://outages.cc/feed.xml?providers=livekit/feed subscribe https://outages.cc/feed.xml?providers=livekitFAQ
Go from tracking outages to fixing production issues with Polylane.
Try Polylane