Railway <> Hetzner servers cant communicate
We had a deployment running succesfully for the last week
today, the railway instances cant communicate with the servers in Hetzner datacenter, and the otherway around is true too
is there some way we can diagnose this?
feels like railway has blocked the whole ip range
Solution:Jump to solution
New reply sent from Help Station thread:
Google blocks a littany of IP addresses due to Hertzner propensity to host...very sketchy shitDDOS: https://managingwp.io/live-blog/google-cloud-and-hetzner-connectivity-issues-gridpane-specifics/Violating Iranian Sanctions: https://blog.cloud66.com/hetzner-connectivity-issues-due-to-sanction-busting-activitiesMore: https://github.com/kubernetes/kops/issues/16466You're seeing this because this thread has been automatically linked to the Help Station thread....
58 Replies
Project ID:
1291fcb5-2ee9-496e-8868-28df0eb6ae6a1291fcb5-2ee9-496e-8868-28df0eb6ae6a
https://discord.com/channels/713503345364697088/727689277219012669/1229812857898799104
same message, with extra info
Railway does not block anything of this sort unless there was an attack originating from Hetzner
what errors are you getting?
connection timeouts
deployments on railway cant access items in hetzner (services)
and hetzner services cant access a DB in railway
let keep this chat in here please, chit chat would not be the place
I just pointed it as it had some extra info
Will chat here
@Brody
connections time out on both sides:
happy to give the server IP if you could check internally for blocks
please note I don't work for Railway.
can you connect to the database locally?
ah sorry
saw a fancy color and assumed
yup
i can connect to the db from my local machine
connect to both sets of servers without issue
but they just wont communicate
are you sure you are using all the correct credentials?
we haven't changed it, it worked fine until yesterday
we also reverted all commits, so nothing in code too
can you ssh into the VPS and do some debugging, pinging, etc
yup
can even connect to it from other VPSs elsewhere
it feel like a firewall block
let's not jump to such conclusions just yet
are you able to ping that host from your vps (the same VPS that is having connection issues)
yes, the vps can ping the ip of the railway service its trying to connect to
can you use the mysql cli from within the VPS to connect to the database
lemme install and test
connection times out
can connect with the exact same parameters from my local machine
how have you been able to confirm that this issue is not with hetzner's network?
the vps can connect to another db hosted elsewhere without any issues
unfortunately that doesn't confirm that this isn't an issue with hetzner's network
do you have other ideas to test?
try with another vps
does this VPS have a static IP?
yes
I tried connecting from that VPS to another db in another railway environment
same issue
are you able to renew that static IP?
possibly
but would rather leave it as a last resort since theres lots tied to it
do you have a separate VPS on hetzner that you can try mysql from?
ill spin one up, give me a sec
yes, it connects fine
its also in the same DC as the failing server
but is using a different IP range
so it seems that Hetzner is not blocking railway
i had the same issue on one of my projects
for some reason railway blocks calls towards Hetzner
getting another IP helped in my case
did it happen more than once ?
yes, there is some kind of black list that google (if im correct) has, which sometimes includes hetzner IP adresses
if yes, we need to move off railway
we have critical infrastructure that we cant swap out like this in production
but if you get another IP, it wont happen
at this time we don't have definitive proof of who's blocking who
may I ask why not move the services on the VPS to railway?
the services on vps needs multiple tcp ports, which railways fails to provide
you would need more than 1 TCP port per app?
this was hetzners response at that time
ah so really it's neither gcp or hetzner's fault
you could try to add an additional IP address and see if it works
(so you dont have to get a new VPS)
that IP is reported as location germany on all the lists
not the list gcp uses apparently
will test from another vps in the same IP range
works fine from another IP in the same range
49.13.X.X
I would go with bart's proposed solution
its a possibility
but i'd like to find a more permanent solution since we have an identical setup for production
thanks for the help diagnosing though 💪
if gcp is blocking hetzner, there really nothing railway can do about that, they are at gcp's whim here
railway isn't blocking your singular static IP, so we can rule railway out of the equation
unless you tried to open well over 10k concurrent requests? this isn't a block from the railway side of things
yup, seems like its something upstream
so does a singular app of yours need to open more than one TCP port?
there are 2 apps which needs us to run vps, one needs a tpc port with a custom subdomain on our own and another with access to like 4 ports
we also need fixed IPs
its a mail system
signup btw... could prob get brody@ https://unn.sh
may I ask if you both are pro on railway? you seen to be missing the badges if so
yes we have a pro team
it didn't give use badges for some reason
now you have them
🔥
Solution
New reply sent from Help Station thread:
Google blocks a littany of IP addresses due to Hertzner propensity to host...very sketchy shitDDOS: https://managingwp.io/live-blog/google-cloud-and-hetzner-connectivity-issues-gridpane-specifics/Violating Iranian Sanctions: https://blog.cloud66.com/hetzner-connectivity-issues-due-to-sanction-busting-activitiesMore: https://github.com/kubernetes/kops/issues/16466You're seeing this because this thread has been automatically linked to the Help Station thread.