Private networking services double-route requests
Hi, we have a potentially show-stopping, but certainly very confusing, bug with the private networking rollout.
My expectation with private networking is that network addresses are per-environment -- for us, this is staging and production, as well as PR environments. My expectation then is that each environment has an isolated VPC and that resources cannot communicate via VPC across those environments. This tracks with the 'per-environment' and 'isolation' per environment language in the private networking and envirnoment docs.
In our case, we have two services, call them alice and bob. Alice is a frontend that talks to Bob, the backend. Bob has a private address,
bob.railway.internal. This address is the same across each environment by default - whereas the default HTTP domain is bob-staging.up.railway.app or bob-repo-pr-38.up.railway.app for example.
I can see that when Alice sends requests across to bob.railway.internal the request is routed to EACH instance of Bob that has private networking enabled! Each request is then operated twice. This does not fit my understanding, that each environment should be isolated.
Are PR envs not considered isolated envs for these purposes? Can Railway automatically prefix the private URL with the env name or something (though, it does break env vars, I guess?) Do we need to turn off PR environments or manually change the envs (!?) for each?16 Replies
Project ID:
a52f72a9-aa58-47ed-a28d-64a3661edb93a52f72a9-aa58-47ed-a28d-64a3661edb93
you are right, this should not be happening, will bring in the appropriate people now
a big chunk of the team is offline, but I have shown them this thread
Tagging in @borck6170 to take a look!
I am going to have a look now. This is indeed not supposed to happen at all.
I see that your production environment is indeed configured to reach the backend using the private network endpoint.
However, both your staging and PR environments are configured to use the public endpoint of the staging environment, which would explain why your staging backend receives requests from both the staging and PR frontends.
Could it be that you forgot to configure the backend URL variable to use the private endpoint for the staging and PR environments?
That is my assumption so far. If correct, that would mean this has nothing to do with private networking.
hm, I willl look more today but that is a bit confusing to me too -- the pr env's frontend has a
nextjs-repo-pr-95.up.railway.app url. we have a react native app which shuold not be hitting that particular URL, so the PR frontends are actually not supposed to be receiving any trafficSorry, I'm not sure I follow. You have 3 services in each of your 3 environments. Could you clarify which services you want to reach from which services, privately and/or publicly?
In your production environment, you have several public domains for your nextjs app, one for the second service, and none for the third one.
In your staging environment, you have one public endpoint for nextjs, one for the second service and none for the third one
Same for your PR environment.
And then, for all environments, you have private endpoints for all 3 services.
the react native app does not live on railway (ios on device). it talks to alice via public internet on a fixed url. alice talks to bob via a fixed url, but because alice is public internet, and we are targeting the staging alice in the app (not the pr alice), each request should only be processed once.
i'm going to try and replicate again and can try to set up a minimal reproduction
I can see that when Alice sends requests across to bob.railway.internal the request is routed to EACH instance of Bob that has private networking enabled! Each request is then operated twice.Is it always exactly twice? What happens if you add a third or fourth environment?
@d.aemon Apologies, but we are not clear at all what issue you are facing. It would help if you could either provide us with a minimal reproduction or at least some more details
ack --- let me create the reproduction, I agree that it's confusing me as well π
maybe a small drawing would help?
this is my understanding. I'm trying to repro it with a smaller setup to rule out bugs in my code.
So far so good, the drawing is clear. Now, which part of it is not working as you intend it to?
ok - I can't reproduce with a minimal example. What I saw earlier was that requests were being actioned twice and hypothesized that it was because multiple versions of the PR envs were alive. I remember seeing logs from the PR env instances of Bob noting that they were doing work. However those PR envs have been closed now and I don't have those logs anymore.
cdeb5710-474b-4115-ac27-9fed132f4ffd is a project which should be a minimal reproduction but so far is behaving properly. so now, maybe it's a problem in my code. need to continue to investage
my probelmatic repo was created before private networking rollout while the test repo is made after. there's no way that should affect it right?Nope, highly unlikely