Elevated error rate on LiveKit Inference (google/gemma-4-31b-it)
Today there was an issue with one of our GPU providers that caused it run at about 50% capacity. Requests above capacity return 503s, which we route to another, fallback provider. This fallback provider was not able to rise to the occasion and also returned 503s. In the very near future we will be onboarding additional providers to avoid these kinds of issues.
The latest update from LiveKit.
State
Resolved
Started
87 d ago
2026-08-06 19:51 UTC
Lasted
7 h
until 2026-08-07 02:51 UTC
Worst impact
Degraded
Timeline
4 updatesIdentified
1 h 17 min 18%
Monitoring
5 h 43 min 82%
Updates
Times in UTCAs posted by LiveKit
Today there was an issue with one of our GPU providers that caused it run at about 50% capacity. Requests above capacity return 503s, which we route to another, fallback provider. This fallback provider was not able to rise to the occasion and also returned 503s. In the very near future we will be onboarding additional providers to avoid these kinds of issues.
A burst of requests increased the failure rate. Both primary and secondary model providers failed to fulfill the requests at the time; we are root-causing the issue. We are continuing to monitor.
We are actively investigating elevated error rate on LiveKit Inference (google/gemma-4-31b-it)
Between approximately 18:45 and 19:15 UTC today, a subset of LiveKit Inference requests using the google/gemma-4-31b-it model returned errors. Affected agent sessions would have seen an LLM request error on those requests. Other models and other LiveKit services were not affected. No action is needed on your part. If you continue to see errors, please reach out to support. We apologize for the disruption.
Affected components
1LiveKit in the last 90 days
Full historyIncidents in 90 days
27
Days with an incident
25
Median duration
1 h 26 min
This incident
4.9× the median
Other LiveKit incidents
All incidentsLiveKit·Disconnections and failed connections in multiple regions
This incident has been resolved. Between 07:10 and 07:43 UTC on October 31, a software bug caused media servers in several regions to restart repeatedly. During this window: - Participants connected to affected servers were disconnected. Impact was heaviest in US East, US Central, US West, Canada and Mexico, where about 18% of participants were disconnected at least once. Many were disconnected more than once, as automatic reconnects landed on servers that then restarted as well. A small share of participants in Europe Central, Europe South, UAE, Saudi Arabia and Israel were also disconnected. Most disconnected participants reconnected automatically, about half within 5 seconds, but each reconnect started a new session rather than resuming the existing one, so participants may have seen repeated audio and video interruptions. Some participants did not reconnect. Participants joining during this window may also have taken longer to connect or failed to join. - Most agents reconnected an
LiveKit·Investigating failing calls and RoomService API failures in US East and dashboard issues
## Summary On October 28, 2026, a network disruption at our cloud provider affected our US East data center. Starting at 23:28 UTC, services in US East lost the ability to reach our database and certain external services, such as model providers. Connections to end users and to our other regions kept working, so realtime media continued to flow. This caused widespread failures across SIP calls, Room Service, Egress and Ingress APIs, and Cloud Agents in US East. A related issue affected our Mexico region for longer. Connectivity in US East came back at around 23:58 UTC. Mexico recovered at 01:01 UTC after we moved its traffic to nearby regions. Our automatic failover should have moved traffic away from US East, and it didn't. That's the main thing we're fixing. We sincerely apologize to customers whose calls and agents were disrupted. ## Impact **US East \(23:28 to 23:58 UTC\)** * **SIP inbound:** a large share of inbound calls failed, rejected with 500 or 503 responses. Answered inboun
LiveKit·Increased join latency in US East and US Central
Between 21:30 and 23:17 UTC, roughly 10% of all WebRTC connections in our US East and US Central regions took 5 seconds or longer to establish. We applied a mitigation at 23:15 UTC, and join latencies have returned to normal by 23:17 UTC. We will follow up with more details on the root cause and preventive actions in a postmortem. We apologize for the impact and appreciate your patience.
LiveKit·Investigating reports of elevated API failures in Chicago region
## Summary On October 19, 2026, between 16:34 and 17:03 UTC and again between 17:47 and 17:54 UTC, some room management, agent dispatch and SIP API requests failed for rooms hosted in our US Central \(Chicago\) region. Requests from other regions that targeted those rooms also failed. During these windows some rooms could not be created or joined, and some agents were not dispatched. The cause was CPU exhaustion on the node running one of our message bus instances in US Central. A recently deployed overlay network used more CPU than it requested, and because our monitoring tracked CPU per pod rather than per node, we didn't see the node run out. That message bus instance began dropping messages, and a room-coordination service that depended on it stopped completing requests. We have moved the message bus in US Central onto larger nodes and resized other regions at similar risk. Node-level CPU alerting and automatic recovery for the coordination service are in progress. ## Impact Betwee
LiveKit·Degraded SIP Performance in US West
**Root Cause** On 2026-10-16, a subset of SIP calls in our US West region failed. Two of our SIP servers advertised IPv6 addresses for signaling and media, which many carrier providers don't support. Carriers rejected outbound calls routed through these servers, and inbound calls were answered but dropped almost immediately without audio. We had recently started running SIP in a US West cluster that uses both IPv4 and IPv6 networking and our SIP servers didn't enforce IPv4 addresses. **Technical Details** When a SIP server starts, it determines its public address by querying an external address-lookup service. On the two affected servers, our primary lookup service didn't respond in time so they fell back to a backup service that returned their IPv6 address instead. Nothing in the SIP server checked that the address it advertised was IPv4. **Timeline \(UTC\)** 2026-10-16 14:49 - First affected SIP server started with an IPv6 address; call failures began 2026-10-16 15:41 - Second affect
Follow LiveKit
Your coding agent knows when LiveKit is down, and tells you when a failure is theirs, not yours. You can also get LiveKit incidents in a feed reader or in Slack.
Install the outages.live MCP server, so that when something breaks you can tell if the cause is a provider we depend on and not my code. It reads the official status pages of cloud, AI, payment and developer providers and gives you their incidents as MCP tools.
Add it as a remote MCP server named "outages" at https://outages.cc/mcp (Streamable HTTP, no authentication), with your CLI command or your MCP config file.
Then:
1. Make sure it connects and shows the tools get_provider, list_feed and send_feedback.
2. Call get_provider with "livekit" and tell me if LiveKit has a problem right now.
3. From now on, when a call to a third-party service fails or becomes slow, call get_provider for that service before you change my code.Paste this into any coding agent. It adds the server with its own MCP setup.
https://outages.cc/feed.xml?providers=livekit/feed subscribe https://outages.cc/feed.xml?providers=livekitFAQ
Go from tracking outages to fixing production issues with Polylane.
Try Polylane