Railway Platform Outage

This incident has been resolved. Builds, deployments, the Railway API, and the dashboard are operating normally. New signups remain temporarily paused while we focus on existing customers. station.railway.com remains unavailable, and we are tracking it in a separate notice here https://status.railway.com/incident/ZEO9X68O Our preliminary report on the October 21 security incident is available at https://blog.railway.com/p/preliminary-report-october-security-incident. If you have not already done so, please rotate sensitive credentials stored in Railway environment variables, revoke the old credentials, and redeploy dependent services. We apologize for the disruption. For help, contact support@railway.com.

Builds are operational, but they may take longer than usual to start or finish while we work through redeploys across the platform. The Railway API and dashboard are fully functional. Our support and community site, station.railway.com, remains unavailable while we work on restoring our remaining non-critical services. For help in the meantime, contact support@railway.com. New signups remain temporarily paused while we focus on existing customers. We will continue to monitor and post another update once build times are back to normal.

We’ve published our preliminary report on the October 21 security incident. This is an update on the existing incident, not a new incident. As of October 23 at 18:57:54 UTC, we have completed notifications to users whose workspaces met our criteria for increased risk. No further higher-risk notifications are pending based on our findings as of that time. If new findings change our assessment of your workspace, we will contact you directly. Workspaces outside those higher-risk groups are not confirmed safe from compromise. If you have not already done so, please rotate sensitive credentials stored in Railway environment variables during the incident, revoke the old credentials, update your variables, and redeploy dependent services. Continue reviewing provider activity for suspicious usage. If you have already completed these rotations since the incident was contained, this update alone does not require you to repeat them. For help, contact support@railway.com.

The preliminary report is now available: https://blog.railway.com/p/preliminary-report-october-security-incident Our investigation is still ongoing and we are still reaching out to customers we believe are at risk. Our team will remain engaged on the matter until we have notified everyone we believe is at risk.

We are in the process of publishing the preliminary report. A clarifying addendum on outreach emails, the Railway engineering team has performed preliminary forensics on the intent of the attacker, as such, we have bucketed customers based on risk of exposure based on logs and data we have collected over the last 48 hours. We have already sent the emails to highest risk targeted workspaces. We are now actively sending emails to the remaining buckets of users where they are considered at risk. If you have known compromised use of a token, please reach out to support@railway.com which will help us with our investigation.

Reminder: if you are facing issues with deployments, you may need to disconnect and reconnect your GitHub integration. Redeploying code from before the incident would not work and result in an error. Builds and deployments for workspaces continue to run normally, and there is no other change since our last update.

Builds and deployments for Enterprise, Pro, Hobby, Trial and Free workspaces are proceeding normally. New signups remain temporarily paused while we focus on existing customers. We have notified customers with a very high likelihood of impact by email. We are still conducting forensic analysis to find more cohorts of likelihood of impact. The team still expects the preliminary report by noon EST.

Builds may stay in a queued state while the Railway team deploys an upgrade to the build system. This update is not related to the security disclosure. New signups remain temporarily paused while we focus on existing customers.

We are continuing to send notification emails to customers with high likelihood of impact. At the current pace, we expect this to take at least several more hours, but we cannot give a reliable estimate for when all emails will be sent. We will post an update once sending is complete, or sooner if there are any material changes. We apologize that we cannot provide a more definite timeline. We still plan to publish our Preliminary Incident Report around midday Eastern Time on October 23.

We are continuing to notify customers with a high likelihood of impact by email. We will update this status page as soon as all notification emails have been sent. We will post an update not later than 03:00 UTC on October 23.

We had been targeting publication of our Preliminary Incident Report tonight, but we need more time before it is ready to publish. We are notifying customers with high likelihood of impact by email now. Builds and deployments for Enterprise, Pro, Hobby, Trial and Free workspaces remain available and are proceeding normally. New signups remain temporarily paused while we focus on existing customers. We will share further updates as new information becomes available and link to the report here when it is published.

There are no meaningful changes since our last update. Builds and deployments for Enterprise, Pro, Hobby, Trial and Free workspaces are proceeding normally. We are continuing to work on completing our Preliminary Incident Report and will share it here when it is published. New signups remain temporarily paused while we focus on existing customers. We will provide another update by 02:00 UTC on October 23 at the latest.

Builds and deployments across Enterprise, Pro, Hobby, Trial and Free workspaces continue to run at approximately normal speed. We remain on track to publish our Preliminary Incident Report soon and will share it here when it is available. New signups remain temporarily paused while we focus on existing customers. We will provide another update by 01:00 UTC on October 23 at the latest.

Builds and deployments for Enterprise, Pro, Hobby, Trial and Free workspaces continue to run at approximately normal speed. We are working to publish our Preliminary Incident Report soon. New signups remain temporarily paused while we focus on existing customers. We will provide another update by 00:15 UTC on October 23 at the latest.

Builds are now available for Free and Trial workspaces, alongside Hobby, Pro and Enterprise. Builds and deployments across all five plans are running at approximately normal speed. New signups remain temporarily paused while we focus on existing customers. We will provide another update by 23:30 UTC at the latest.

Builds and deployments for Hobby, Pro and Enterprise workspaces are running at approximately normal speed. We are continuing to work on enabling builds for Free and Trial workspaces and will post here when they become available. New signups remain temporarily paused while we focus on existing customers. We will provide another update by 23:30 UTC at the latest.

Builds are enabled for Hobby, Pro and Enterprise workspaces. Builds and deployments are generally completing successfully but may still take longer than usual to complete. We will provide an update when we enable builds for Free and Trial workspaces. New signups remain temporarily paused while we focus on existing customers. We will provide another update by 22:45 UTC at the latest.

Builds are now available for Hobby workspaces. We’ve published Security Guides to help you take the next steps relevant to your setup. You can find them on our engineering blog: https://blog.railway.com/engineering Deployments are likely to take longer than usual for Hobby users while we work through demand. Once Hobby users are stable, we will work to enable builds for Free and Trial workspaces. New signups remain temporarily paused while we focus on existing customers. Builds and deployments for Pro and Enterprise workspaces continue to start normally but may take longer to complete. We will provide another update by 22:00 UTC at the latest.

Builds and deployments for Pro and Enterprise workspaces continue to run normally, and there is no other change since our last update. We are still working on enabling builds for Hobby, with Free and Trial to follow, and will post here as each is enabled. New signups remain temporarily paused. We will provide another update by 21:15 UTC at the latest.

There is no change since our last update. New signups remain temporarily paused while we focus on existing customers. Builds and deployments for Pro and Enterprise workspaces are running normally, and enabling builds for Hobby remains our current focus. Free and Trial will follow after Hobby. We will provide another update by 20:30 UTC at the latest.

We have temporarily paused new signups so we can focus on making sure everything is working well for existing customers. We will re-enable signups once that is done, and will post here when they are available again. Builds and deployments for Pro and Enterprise workspaces continue to run normally. We are still working on enabling builds for Hobby, followed by Free and Trial, and will post here as each is enabled. We will provide another update by 19:45 UTC at the latest.

Deployments in US West are working normally again. If a deployment in US West failed after its build completed, please redeploy the service. Builds and deployments for Pro and Enterprise workspaces continue to run normally, and we are still working on enabling builds for Hobby. We will post here as soon as they are available. We will provide another update by 19:15 UTC at the latest.

We are aware that some deployments in US West are failing, even after their build completes successfully. We are investigating and working on a fix. Deployments in other regions are not affected by this issue. We will provide another update by 19:15 UTC at the latest.

Builds and deployments for Pro and Enterprise workspaces are running normally, and there is no change to report since our last update. We are still working on enabling builds for Hobby, and will post here as soon as they are available. Free and Trial will follow after Hobby. If a build fails shortly after it starts, please retry it. We will provide another update by 19:15 UTC at the latest.

There is no change since our last update. Builds and deployments for Pro and Enterprise workspaces continue to run at normal speed. If your build fails shortly after it starts, please retry it once. We are continuing to work on enabling builds for Hobby, which will be next, followed by Free and Trial. We will post here as each is enabled. We will provide another update by 18:30 UTC at the latest.

The API is now fully restored, including features that were temporarily unavailable. API tokens and Project Tokens created before the security incident remain invalidated, so if you use the Public API or the cli with a token, please create a new token. Deployments for both Pro and Enterprise workspaces have been available since 06:30 UTC and continue to run at normal speed. The build failure errors we were seeing before should now be resolved. If your build does still fail shortly after start, please retry it once. We are working to enable builds for Hobby, then Free and Trial, later today. We will post here as each is enabled, and will update this page if that changes. Once this incident is resolved, we will publish a full incident report on our blog and link to it from this page. We will provide another update by 17:45 UTC at the latest.

Deployments for Pro and Enterprise workspaces are now running at normal speed. We are working to get builds enabled for Hobby next, and will post here when they are enabled. We are aware that some builds are failing shortly after they start, and we are working on a fix. If your build fails this way, please retry it once. We will provide another update by 17:00 UTC at the latest.

Deployments for Pro and Enterprise workspaces continue to run a little slower than usual, and some may take longer than expected to complete. We are working to bring them back to normal speed. We will post here when builds for Hobby, Free and Trial are re-enabled. We will provide another update by 16:15 UTC at the latest.

Deployments for Pro and Enterprise workspaces are running a little slower than usual, and some may take longer than expected to complete. We are working to bring them back to normal speed. We are continuing to work on enabling builds for Hobby, then Free and Trial, and will post here as each is enabled. We will provide another update by 15:30 UTC at the latest.

We have worked through the queue of delayed deployments, and deployments for Pro and Enterprise workspaces are now starting at normal speed. No action is needed for deployments that were queued. We are continuing to work on enabling builds for Hobby, then Free and Trial, and will post here as each is enabled. We will provide another update by 14:45 UTC at the latest.

We have almost worked through the queue of delayed deployments, and wait times are close to normal. A small number of deployments may still take longer than usual to start. Builds remain available for Pro and Enterprise workspaces. We are continuing to work on enabling builds for Hobby, then Free and Trial. We will provide another update by 14:00 UTC at the latest.

Deployment wait times are improving significantly as we work through the queue, and most deployments are now starting much sooner. Some may still take longer than usual to start. We are continuing to work on bringing deployment times fully back to normal. Builds remain available for Pro and Enterprise workspaces. We are continuing to work on enabling builds for Hobby, then Free and Trial. We will provide another update by 13:15 UTC at the latest.

Deployment wait times are coming down as we work through the queue, though some deployments may still take longer than usual to start. Builds remain available for Pro and Enterprise workspaces. We are continuing to work on enabling builds for Hobby, then Free and Trial. We will provide another update by 12:30 UTC at the latest.

The number of queued deployments has stopped growing, and queued deployments are now starting, though more slowly than usual. To avoid adding to the wait, please do not trigger the same deployment more than once. We are also continuing to work on build error rates. Builds remain available for Pro and Enterprise workspaces. We are continuing to work on enabling builds for Hobby, then Free and Trial. We will provide another update by 11:45 UTC at the latest.

There is no change since our last update. Some builds are still failing more often than usual, and many deployments remain queued with longer than usual wait times. We are continuing to work on both. Builds remain available for Pro and Enterprise workspaces. We are continuing to work on enabling builds for Hobby, then Free and Trial. We will provide another update by 11:00 UTC at the latest.

Some builds are failing more often than usual, and we are working to bring error rates back down. Many deployments are also still queued, and wait times remain longer than usual. To avoid adding to the wait, please do not trigger the same deployment more than once. Builds remain available for Pro and Enterprise workspaces. We are continuing to work on enabling builds for Hobby, then Free and Trial. We will provide another update by 10:15 UTC at the latest.

There is no change since our last update. Many deployments are still queued, and wait times may be longer than earlier. To avoid adding to the wait, please do not trigger the same deployment more than once. We are continuing to work on bringing deployment times back to normal. Builds remain available for Pro and Enterprise workspaces. We are continuing to work on enabling builds for Hobby, then Free and Trial. We will provide another update by 09:30 UTC at the latest.

Deployments are going live, but many are taking longer than usual to start and may stay queued for some time. To avoid adding to the wait, please do not trigger the same deployment more than once. We are working to bring deployment times back to normal. Builds remain available for Pro and Enterprise workspaces. We are continuing to work on enabling builds for Hobby, then Free and Trial. We will provide another update by 08:45 UTC at the latest.

Deployments are currently taking longer than usual to start, and some may stay queued for longer than expected. We are investigating and working to bring them back to normal speed. Builds remain available for Pro and Enterprise workspaces. We are continuing to work on enabling builds for Hobby, then Free and Trial. We will provide another update by 08:00 UTC at the latest.

Builds are now available for Pro and Enterprise workspaces. Pro customers can now complete the steps in our published guides that require a redeploy. We will provide another update by 07:10 UTC at the latest.

Enterprise deployments are now moving through each stage at normal speed, including the "Running pre-deploy command" stage. Deployments that were waiting at one stage are continuing on their own, and no action is needed. We will provide another update by 06:45 UTC, or sooner if there is news before then.

We are continuing to work on deployments that stay at one stage, such as "Running pre-deploy command", for longer than expected. There is no other change to report since our last update. Builds remain available for Enterprise workspaces. We are continuing to work on enabling builds for Pro, Hobby, then Free and Trial. We will provide another update by 06:45 UTC, or sooner if there is news before then.

Deployments are taking longer than usual to progress through each stage. Users may see a deployment stay at one status, such as "Running pre-deploy command", for longer than expected. We are investigating and working on a fix. We will provide another update by 06:00 UTC at the latest.

The Redeploy feature has been fixed. Enterprise customers can now use the Redeploy feature. We are continuing to work on enabling builds for all other tiers. We will provide another update by 05:30 UTC at the latest.

Services with pre-deploy commands are stuck at the “Running pre-deploy command” stage. We are currently investigating and working on a fix. We will provide another update by 05:15 UTC at the latest.

In accordance with our security disclosure, we have invalidated all Railway API tokens. If you are using an API token to integrate with Railway’s Public API (https://docs.railway.com/integrations/api), it will no longer work. Please create new API tokens on Railway to continue using Railway’s Public API. We will provide another update by 05:00 UTC at the latest.

Builds for Enterprises continue to be available. Please note that the Redeploy feature is currently unavailable. To initiate a redeploy, please browse to your service -> Ctrl/CMD+K -> Deploy Latest Commit. We are working to re-enable the Redeploy feature and will provide an update as soon as available. We will provide another update by 04:45 UTC at the latest.

Builds for Enterprise workspaces are now available. Enterprise customers can now complete the steps in our published guides that require a redeploy, including applying any new credentials you have added in Railway. Pro, Hobby, then Free and Trial will follow in stages. We will post here as each is enabled, and the guides remain the recommended next steps regardless of plan. We will provide another update by 04:15 UTC or sooner.

Builds remain paused. We are continuing to work to bring them back online and expect to have an update for Enterprise users here soon. The guides published earlier cover the steps you can begin taking now. We will provide another update by 03:45 UTC, or sooner if there is news before then.

We are still working to bring builds back online. There is no change to report since our last update. The guides published in our previous update cover the recommended next steps regarding the security notice. Some of those steps require a redeploy and will have to wait until builds are available, but others you can begin now. We will provide another update by 03:00 UTC, or sooner if builds are back before then.

We have published guides to help users through recommended next steps regarding the security notice. These guides may require redeployment of services which you will not be able to complete until Builds are available again. However, there are steps you may begin to take. We will update as soon as Builds are available. This page remains the fastest place for updates. Published Guides: - Check for Unexpected OpenRouter Usage: https://blog.railway.com/p/security-disclosure-recommendation-check-for-unexpected-openrouter-usage - Rotate PostgreSQL Database Credentials: https://blog.railway.com/p/security-disclosure-recommendation-rotate-postgresql-database-credentials - Check for Unexpected Grok Usage: https://blog.railway.com/p/security-disclosure-recommendation-check-for-unexpected-grok-usage - Redeploy Buckets and Reconnect GitHub After the Credential Reset: https://blog.railway.com/p/security-disclosure-recommendation-redeploy-buckets-and-reconnect-github-after-the-credential-reset - Che

We are continuing to work to make builds available again. We will be publishing guides shortly to help users through recommended next steps regarding the security notice. This page remains the fastest place for updates.

Please note that customers may see revocation of storage bucket credentials performed by Railway employee "Faraz Patankar" in their audit logs or activity panel. This is expected, as we previously indicated, we are revoking all existing storage bucket credentials out of an abundance of caution.

We are continuing to rotate bucket credentials and are working to make builds available again as soon as possible. We do not have further details to share right now, and this page remains the fastest place for updates as we learn more. If you need to reach us, you can email support@railway.com. Please note that we may not be able to share more detail than is published here. Builds remain paused. We will post here as soon as they resume.

We are rotating all Railway bucket credentials in response to the security incident. New credentials will appear in your dashboard automatically. Because deployments are paused, you will not be able to redeploy your services to pick up the new credentials until builds are restored. We will post here when builds resume with any additional details.

We have published a security notice covering what we know so far and the actions we recommend you take: https://blog.railway.com/p/security-disclosure-wednesday-october-21st We strongly recommend that you revoke all environment variables stored in Railway, including API keys and private registry credentials, and review your service logs for unusual activity. Deployments remain paused while we investigate, including for Enterprise customers. We understand you are unable to update your applications with new credentials at this time and sincerely apologize for the inconvenience. We will post here when builds are restored. As we confirm which accounts were affected, we will contact those customers directly by email.

We do not yet have additional information to provide about the availability of builds. We apologize for the inconvenience and we will provide an update as soon as we can.

We are validating builds for Enterprise workspaces and should be re-enabling for Enterprises broadly very soon. Pro, Hobby, then Free and Trial will follow in stages, and we will post here as each is enabled. Some users will be signed out and will need to log in again. We will continue posting updates at least every 30 minutes, and sooner if we have more information. The next update will by 21:00 UTC.

We are continuing to work towards enabling builds for Enterprise workspaces. We will continue posting updates at least every 30 minutes, and sooner if we have more information. The next update will be at 21:00 UTC latest.

We are continuing to work on a fix for build issues. Due to impact from adjacent cache systems, we are invalidating OAuth sessions as part of our mitigation. Affected users will be signed out and will need to log in again. We continue to expect builds for Enterprise workspaces to be available by 20:30 UTC. It is possible that this could be delayed by up to 30 minutes, but we will provide an update at 20:30 UTC regardless. Pro, Hobby, then Free and Trial will follow in stages, and we will post here as each is enabled. Our API, dashboard and station.railway.com remain affected. We will continue posting updates at least every 30 minutes, and sooner if we have more information. The next update will be at 20:30 UTC.

We have mitigated some issues impacting builds and are continuing to work on bringing builds back online. We continue to expect builds for Enterprise workspaces to be available by 20:30 UTC. Pro, Hobby, then Free and Trial will follow in stages, and we will post here as each is enabled. Our API, dashboard and station.railway.com remain affected. We will continue posting updates at least every 30 minutes, and sooner if we have more information. The next update will be at 20:25 UTC

Earlier today an issue affecting multiple systems led us to pause builds across the platform. We are now working to bring builds back. We expect builds for Enterprise workspaces to be available by 20:30 UTC. Pro, Hobby, then Free and Trial will follow in stages, and we will post here as each is enabled. Our API, dashboard and station.railway.com remain affected. We will continue posting updates at least every 30 minutes, and sooner if anything changes. The next update will be at 20:00 UTC.

Builds and deployments remain paused, and our API, dashboard and station.railway.com are still affected. When we re-enable builds, we will do it in stages to keep the platform stable as capacity comes back: Enterprise workspaces first, followed by Pro, then Hobby, then Free and Trial. We will post here as each stage is enabled. We will continue posting updates at least every 30 minutes, and sooner if anything changes. The next update will be at 19:45 UTC.

Work on the issues affecting our API, dashboard, and the build and deployment pipeline is ongoing. Builds and deployments remain paused, and station.railway.com is still unavailable. We do not have new information to share at this time. We will continue posting updates at least every 30 minutes, and sooner if anything changes. The next update will be at 19:15 UTC.

Builds and deployments remain paused, and our API, dashboard and station.railway.com are still affected. We do not have new information to share at this time. We will continue posting updates at least every 30 minutes, and sooner if anything changes. The next update will be at 18:45 UTC.

There is no change to report. Our API, dashboard, and the build and deployment pipeline remain affected, builds and deployments are still paused, and station.railway.com remains unavailable. We will continue posting updates at least every 30 minutes, and sooner if anything changes. The next update will be at 18:15 UTC.

We are still working to resolve the issues affecting our API, dashboard, and the build and deployment pipeline. Builds and deployments remain paused, and our support site (station.railway.com) is unavailable. We do not have further detail to share yet. We will post updates at least every 30 minutes, and sooner if we have more information. The next update will be at 17:45 UTC.

We are still working to resolve the issues affecting our API, dashboard, and the build and deployment pipeline. Builds and deployments remain paused, and our support site (station.railway.com) is also unavailable, so new support requests cannot be submitted there at this time.

There is no change since our last update. Builds remain paused and some API features are still unavailable. Our support platform is still down as well. We are continuing to work on a fix and will post updates here.

Builds remain paused. Our API is back up, but some features remain unavailable. We are working on restoring access to our support platform and will continue providing updates here.

We have identified the issue and are working on a fix. While we do, the Railway dashboard, API and CLI are unavailable, and new builds remain paused in all regions. Our support site is also affected, so replies to support requests will be delayed.

We are seeing issues on Railways backend as well. We are still investigating this issue.

We are currently investigating this issue