Railway crashed!!!
Even heathchecks crashed.
https://docs.railway.app/reference/healthchecks
Solution:Jump to solution
This has been resolved, though if you had already switched off the Legacy proxy, please keep it off as the edge proxy provides many improvements!
62 Replies
Project ID:
26e99e23-574d-4cb3-95e8-a01e5129f031For me too
damn
504
@Vin
https://api.postiz.com/user/self
Project ID: 26e99e23-574d-4cb3-95e8-a01e5129f031
Project ID: 028f59e8-895a-40ed-bd82-05a25a81146c
Likewise. Services down
I'm notifying the team
@Nevo David @RenderCoder @preetpatel in your service settings, if you untick "Use legacy proxy", it should redeploy, and resolve the issue.
Happening to me too
My server is down
Should I try that @Vin ?
Yes, you can try that as well
Should I untick and trigger a redeploy?
Unticking should do it, but if not, go ahead and try a redeploy
ok unticking didn't do it, so I did it manually. Will report back in a couple mins when it's done building thank you for the help
@devenbhooshan please see my message above
@Brilew please see my message above
confirmed issue with the legacy proxy, we are looking into it but we strongly recommend switching it off
ok removed it
ty
shouldn't change anything right /
No, switching off the legacy proxy will not change how your app operates
same here, trying the legacy switch
alright I'm back online that did work. Thank you @Vin
didnt work. triggered a redeploy, waiting now...
Since switching off the legacy proxy is a DNS update, it could take a moment for it to take effect
makes sense, thanks @Vin
back online
thanks @Vin
all back online here too
tyty
The custom domains are not working
legacy proxy was always off in our case
I unticked, redeployed, nada
please see vins answer to why it could take a while to switch it off
need to wait for dns to update
you could be behind an ISP DNS server that caches results for a long time and doesn't respect TTL
any idea what to do in our case?
Could you try toggle it on and off and then see if that works? Possible it might not have been migrated properly
Links to services where y'all are seeing issues would be helpful.
@kmskrishna @Nevo David
that's my project id: 355a123a-8f0d-473a-aa69-318226cfbe58
our proj id : a04a8788-1d7c-412d-8904-24fdc6b2c50b
our legacy proxy was always off from what I know
if you could provide the domain we can also manually check if our dns side of things has you on the new edge proxy
Dm'd you
btw, for me it's random, sometimes works sometimes not
Try incognito? I can access your API from my side.
Might be cache.
So it depends, same request sometimes work sometimes not
so I don't think it's cache
My applications are down...
still down after switching legacy proxy off
Probably it's jumping between edge proxies?
Switching off the legacy proxy is a DNS change, it can take some time, especially if your ISP doesn't respect the TTL of the dns update
This operation does not work for one of my applications, but it works for another one.
@odin88 it is not just you, please reference this message https://discord.com/channels/713503345364697088/1282559011274887210/1282562070067413032
Can you share a link to the service that is not working still, so I can pass it to the team?
It worked for a little bit, now its down again
and now its back again
...?
whats going on
Your DNS provider is likely returning stale values, they probably have multiple boxes to serve you DNS results and they aren't kept in sync
down again
Hey @Vin I'm back here. It was fixed for 30 min now I'm seeing intermittent failures.
If I refresh sometimes I get the Railway Application didn't respond error, other times it seves my app successfully
@rickitan
Your DNS provider is likely returning stale values, they probably have multiple boxes to serve you DNS results and they aren't kept in sync
Thank you Vin, do DNS providers usually offer a way to flush the cache? Or should I just wait? (researching into mine)
Any recommendations?
Unfortunately, when it comes to DNS, it's all a matter of waiting for them to update
Normal DNS propagation can take 24-48 hours, but can take up to 72 in some cases, but more and more, it is much faster than that. But depending on your ISP if they aren't keeping things in sync, it could take a bit
you can also temp switch to cloudflare's 1.1.1.1 dns
wdym?
It's really frustrating, as I refresh sometimes it works sometimes it doesn't and I get the railway application failed to respond page.
(this is after i turned off the legacy router an hour ago)
Are we 100% confident this is a DNS issue and not railway?
im looking at the uptime for the new edge proxy and its all green
Solution
This has been resolved, though if you had already switched off the Legacy proxy, please keep it off as the edge proxy provides many improvements!
jesus christ.. the edge proxy is hella fast
thanks team ❤️
yep thats the main point, you also get faster routing cutovers when a new deploy goes live!
holy moly... this is state of the art
going to mark this as solved now!