RTT to nearby datacenter has gone way up in the past few days
Regions have been amazing and are super helpful for our use case. We host our database in AWS us-east-1, which is in the Washington DC area. We've moved our Railway servers to GCP us-east-4 as that is in the same local area. I was experimenting with this last week and was getting single digit to low double digit millisecond RTT/latency from Railway to AWS. This weekend I redeployed the same service and found that the latency has gone back up to >70 ms round-trip time.
I'm scratching my head here, because when I can double check the IP of my Railway service and AWS database and see they are both in the Washington DC area, yet this latency is really bad for that proximity.
My only thought is perhaps in your development work you all have added some configuration that adds some sort of an additional layer to this networking that adds additional latency.
What could be going on here?
11 Replies
Project ID:
95c304a2-fe2b-4ad1-bc3f-5296fd26f36c95c304a2-fe2b-4ad1-bc3f-5296fd26f36c
These datacenters should have very low latency between them: https://medium.com/@sachinkagarwal/public-cloud-inter-region-network-latency-as-heat-maps-134e22a5ff19
Medium
Public Cloud Inter-region Network Latency as Heat-maps
Optimize your public cloud region choices.
For context, my local instance has RTT of ~20ms, and I'm in NY. Theres definitely some funky networking going on here, given gcp-us-east4 (railway) <-> aws-us-east-1 is over 3x slower than NY <-> aws-is-east-1, and given that last week it was very low latency
actually, resolving the IP address for my services shows that they're still in us-west, despite having RAILWAY_REGION=us-east4
oh, it seems like RAILWAY_REGION does not work utilizing a shared variable!
This is resolved
for future reference, the region in use is printed in the first line in the deployment logs
yes it is! good to know
do you know how volumes work with regions? what happens if you change the region of a service that has a volume?
https://docs.railway.app/deploy/deployments#caveats
while your question is a little different than the wording used in the docs, I think they are similar enough to not risk it without first testing it in a test project
ah
I'd also be interested to know the results of that test!
We’re discussing this next week 🙂
Right now they don’t work. If you set a region on a svc with volume attached, it’ll be stuck at the deploy phase
Primarily because your volume can be on a different box than what it was on when switching regions