Investigating failing calls and RoomService API failures in US East and dashboard issues
## Summary On October 28, 2026, a network disruption at our cloud provider affected our US East data center. Starting at 23:28 UTC, services in US East lost the ability to reach our database and certain external services, such as model providers. Connections to end users and to our other regions kept working, so realtime media continued to flow. This caused widespread failures across SIP calls, Room Service, Egress and Ingress APIs, and Cloud Agents in US East. A related issue affected our Mexico region for longer. Connectivity in US East came back at around 23:58 UTC. Mexico recovered at 01:01 UTC after we moved its traffic to nearby regions. Our automatic failover should have moved traffic away from US East, and it didn't. That's the main thing we're fixing. We sincerely apologize to customers whose calls and agents were disrupted. ## Impact **US East \(23:28 to 23:58 UTC\)** * **SIP inbound:** a large share of inbound calls failed, rejected with 500 or 503 responses. Answered inboun
The latest update from LiveKit.
State
Resolved
Started
4 d ago
2026-10-28 23:47 UTC
Lasted
1 h 32 min
until 2026-10-29 01:19 UTC
Worst impact
Degraded
Timeline
10 updatesInvestigating
29 min 32%
Monitoring
58 min 64%
Identified
4 min 5%
Updates
Times in UTCAs posted by LiveKit
## Summary On October 28, 2026, a network disruption at our cloud provider affected our US East data center. Starting at 23:28 UTC, services in US East lost the ability to reach our database and certain external services, such as model providers. Connections to end users and to our other regions kept working, so realtime media continued to flow. This caused widespread failures across SIP calls, Room Service, Egress and Ingress APIs, and Cloud Agents in US East. A related issue affected our Mexico region for longer. Connectivity in US East came back at around 23:58 UTC. Mexico recovered at 01:01 UTC after we moved its traffic to nearby regions. Our automatic failover should have moved traffic away from US East, and it didn't. That's the main thing we're fixing. We sincerely apologize to customers whose calls and agents were disrupted. ## Impact **US East \(23:28 to 23:58 UTC\)** * **SIP inbound:** a large share of inbound calls failed, rejected with 500 or 503 responses. Answered inboun
This incident has been resolved. Between 23:30 and 23:58 UTC, a network disruption at an upstream infrastructure provider affected our US East region. During this window: Inbound SIP calls in US East failed at elevated rates, with affected calls rejected with 500 or 503 responses. Answered inbound calls were about 65% below normal across the window, and up to 90% below normal between 23:45 and 23:55 UTC. About 10% of outbound SIP call requests (CreateSIPParticipant) in US East failed, some returning 408 deadline_exceeded. Between 23:28 and 23:58 UTC, there were elevated rates of Room Service, Egress, and Ingress API request server errors in US East. Some participants saw failed room joins or increased packet loss. The LiveKit Cloud dashboard was inaccessible for part of this window. Network connectivity recovered at approximately 23:58 UTC, and error rates and call volumes returned to normal by 00:20 UTC. A related network issue also affected our much smaller Mexico region from 23:29 U
We've applied a mitigation for our Mexico region and services have returned to baseline. We are continuing to monitor and will update again within 15 minutes.
While the US regions still appear to be stable, we are tracking degraded services including SIP, RTC, and APIs in our Mexico region. We will update again as soon as possible.
We believe the mitigation was successful and are seeing services return to normal. The dashboard is accessible and error rates have gone down across services. We are continuing to monitor to make sure that the situation is resolved. We will update again as soon as possible.
We are still determining the full scope of impact, but we believe that the Room Service has also been impacted. We will update again within 15 minutes.
We are currently attempting a mitigation. We will follow up with more information in the next 15 minutes.
We believe there is also impact to inbound calls in the affected regions. We are working on identifying the cause and will update again within 15 minutes.
The LiveKit cloud dashboard is currently inaccessible. We are investigating this and will provide another update within 15 minutes.
We are currently investigating an increased rate of failing outbound calls in US East. We will post another update with more information as soon as possible.
Affected components
5LiveKit in the last 90 days
Full historyIncidents in 90 days
27
Days with an incident
25
Median duration
1 h 26 min
This incident
About the median
Other LiveKit incidents
All incidentsLiveKit·Disconnections and failed connections in multiple regions
This incident has been resolved. Between 07:10 and 07:43 UTC on October 31, a software bug caused media servers in several regions to restart repeatedly. During this window: - Participants connected to affected servers were disconnected. Impact was heaviest in US East, US Central, US West, Canada and Mexico, where about 18% of participants were disconnected at least once. Many were disconnected more than once, as automatic reconnects landed on servers that then restarted as well. A small share of participants in Europe Central, Europe South, UAE, Saudi Arabia and Israel were also disconnected. Most disconnected participants reconnected automatically, about half within 5 seconds, but each reconnect started a new session rather than resuming the existing one, so participants may have seen repeated audio and video interruptions. Some participants did not reconnect. Participants joining during this window may also have taken longer to connect or failed to join. - Most agents reconnected an
LiveKit·Increased join latency in US East and US Central
Between 21:30 and 23:17 UTC, roughly 10% of all WebRTC connections in our US East and US Central regions took 5 seconds or longer to establish. We applied a mitigation at 23:15 UTC, and join latencies have returned to normal by 23:17 UTC. We will follow up with more details on the root cause and preventive actions in a postmortem. We apologize for the impact and appreciate your patience.
LiveKit·Investigating reports of elevated API failures in Chicago region
## Summary On October 19, 2026, between 16:34 and 17:03 UTC and again between 17:47 and 17:54 UTC, some room management, agent dispatch and SIP API requests failed for rooms hosted in our US Central \(Chicago\) region. Requests from other regions that targeted those rooms also failed. During these windows some rooms could not be created or joined, and some agents were not dispatched. The cause was CPU exhaustion on the node running one of our message bus instances in US Central. A recently deployed overlay network used more CPU than it requested, and because our monitoring tracked CPU per pod rather than per node, we didn't see the node run out. That message bus instance began dropping messages, and a room-coordination service that depended on it stopped completing requests. We have moved the message bus in US Central onto larger nodes and resized other regions at similar risk. Node-level CPU alerting and automatic recovery for the coordination service are in progress. ## Impact Betwee
LiveKit·Degraded SIP Performance in US West
**Root Cause** On 2026-10-16, a subset of SIP calls in our US West region failed. Two of our SIP servers advertised IPv6 addresses for signaling and media, which many carrier providers don't support. Carriers rejected outbound calls routed through these servers, and inbound calls were answered but dropped almost immediately without audio. We had recently started running SIP in a US West cluster that uses both IPv4 and IPv6 networking and our SIP servers didn't enforce IPv4 addresses. **Technical Details** When a SIP server starts, it determines its public address by querying an external address-lookup service. On the two affected servers, our primary lookup service didn't respond in time so they fell back to a backup service that returned their IPv6 address instead. Nothing in the SIP server checked that the address it advertised was IPv4. **Timeline \(UTC\)** 2026-10-16 14:49 - First affected SIP server started with an IPv6 address; call failures began 2026-10-16 15:41 - Second affect
LiveKit·Intermittent connection timeouts in the Frankfurt region
This incident is resolved. Connections to the Frankfurt region have returned to normal.
Follow LiveKit
Your coding agent knows when LiveKit is down, and tells you when a failure is theirs, not yours. You can also get LiveKit incidents in a feed reader or in Slack.
Install the outages.live MCP server, so that when something breaks you can tell if the cause is a provider we depend on and not my code. It reads the official status pages of cloud, AI, payment and developer providers and gives you their incidents as MCP tools.
Add it as a remote MCP server named "outages" at https://outages.cc/mcp (Streamable HTTP, no authentication), with your CLI command or your MCP config file.
Then:
1. Make sure it connects and shows the tools get_provider, list_feed and send_feedback.
2. Call get_provider with "livekit" and tell me if LiveKit has a problem right now.
3. From now on, when a call to a third-party service fails or becomes slow, call get_provider for that service before you change my code.Paste this into any coding agent. It adds the server with its own MCP setup.
https://outages.cc/feed.xml?providers=livekit/feed subscribe https://outages.cc/feed.xml?providers=livekitFAQ
Go from tracking outages to fixing production issues with Polylane.
Try Polylane