Redis Networking/Egress despite using internal networking
Hi there
We are seeing Network activity on our Redis metrics despite connecting to it via internal networking. Is this expected? (Ignore the dip! We were down for maintenance 😆 )
Project ID
4c3b4b0e-006a-407e-90c7-9c3031cd622f
Solution:
Jump to solution
Ok - I figured it out. While waiting for DNS changes to propagate I left our Heroku Dyno live. I had it connected to our Railway Redis service via public networking. I figured if I disable public networking it would kill any active connections but it seems like thats not the case? The Heroku Dyno was still connected to the Redis Service.
Anyway, shutting down the dyno reduced network usage as expected...
52 Replies
Project ID:
4c3b4b0e-006a-407e-90c7-9c3031cd622fwell one way to make sure nothing is connecting to it publicly would be to remove its tcp proxy, then before long something will crash
😆 true!
i can do this for our stg and dev servers
lets see how it goes
oh wait, i think i know why
i am using the internal ip but using the public port
ie
**.rlwy.net:56258 -> :6379
Private Networking
redis.railway.internal
and I am connecting to it via redis.railway.internal:6379 lol
the public port is 56258 in this case, 6379 being the private port
oh, right right
then thats not it
what you actually described
redis.railway.internal:56258 would also just outright not workyeah i didn't think it would, i was kinda suprised it was working at all with what i described!
but yeah seems like I have that the right way around
disabling the public TCP proxy now for dev/stg and observing/testing
seems fine so far... strange.. there is so much traffic on prd
prod is somehow connecting to it via the public network?
hm... i dont think so
but we will see
i disbled it for prd too and just observing again
but both stg and dev still show some traffic, minor, but still
will see what happens with prd
sounds good
Hmm, nope
its like the Redis traffic counts as egress..?
been about 10 mins
the traffic on the private network shouldn't be measured, check the cost breakdown to see if your redis service is incurring any egress
good idea
yeah it is :blobconfused:
and 'double the cost' i guess since the service talking to it is also taking the egress hit
and you have completely removed its tcp proxy?
yep!
is redis somehow sending data elsewhere? there's no tcp proxy but that doesn't disconnect it from the internet
I am not sure how i would check that. I just spun up redis using right click > database > redis, nothing bespoke
what are you using redis for?
various things - caching, temp store, used by my redis io adapter
each has their own, and they are the only ones communicating with their own redis db
lol
okay well now I have to go off on a tangent here
why service groups for your different environments instead of project environments?
hahaha i was debating whether i should put this up because i figured you would mention that haha
is there a cross-project egress cost? because my API Gateway would have to be in its own project
then the game servers in a nother, with an environment for each
but it means all comms will happen from API Gateway <- across projects -> Game Server
well you would have a single project, with multiple environments, each environment would have a gateway, game server and it's redis database
yeah but the gateway directs traffic to one of the 3 environments
so which is the 'true true' gateway, and are the other 2 just dummies that dont do anything, i guess? haha
if you where to go with the architecture I mentioned, your gateway would only route traffic to a production (or set of production) game server services in the production environment, and the gateway in the development environment would route traffic to the development game server in the same environment
it makes most sense to me if i could nest this setup inside one another.. something like
yeah but I need the single gateway to be able to route traffic to the Dev or Stg or Prd version of the server
no matter how you do it, you absolutely wouldn't want to be using multiple projects, and honestly with the group UI you have done it quite cleanly, was just curious is all
basically I'm stalling while trying to think of potential reasons for seeing such high outbound usage
it would be cool to be able to nest a project inside another, and then from the parent project I could have an API gateway to proxy the traffic to different environments of the sub project (dev/stg/prd)
and then I could spin up a dev version fo this WHOLE setup and test the entire stack
because right now I don't really have a 'stg' or 'dev' version of the my API gateway
For that, I could use the environments feature. Have a Dev/Stg version of my current project that, inside it, has the Dev/Stg/Prd game services 'baked in'
hah! right? its a lot of traffic
we do run a multiplayer game so there is a lot of traffic going in and out, so I think the volume maybe makes sense
does the game server even have comparable traffic?
yeah, much more
i am grappling with this now, as our egress costs have made hosting on railway much more expensive than heroku hah
the gateway doesn't touch redis at all right?
yeah unfortunately there's not much railway can do about that right now since GCP has very high egress costs
so i am trying to make sure i am doing everything right and only incuring egress from what is absolutely neccessary, to get a better cost estimate. and thats how i encountered this odity with Redis
+1 yep
railway is moving to bare metal, I suspect the egress costs will drop and they will hopefully pass that down to the users too
but I am genuinely stumped here
fingers crossed, yeah it is a bit too pricing, it stings. I was excited about railway for future projects - for hosting game servers and such, but it might just cost too much in terms of egress for realtime applications
here is our prd game server
it might sting less if redis didn't have traffic for some reason :mildpanic:
exactly! hah 🙂
but good news, you're pro so I can escalate this to the team, hopefully they can dig into some of the internal network logs for these services, any last ideas before i do that though?
i just double checked my redis urls and such
but i guess with public networking disabled it doesnt mean much
so no, not really. I am also stumped! thanks 🙂
the team when available will communicate via that help station thread
I've heard they love when I escalate long debugging threads 🙂
😄 awesome thanks so much, curious to see what the outcome will be
me too!
Solution
Ok - I figured it out. While waiting for DNS changes to propagate I left our Heroku Dyno live. I had it connected to our Railway Redis service via public networking. I figured if I disable public networking it would kill any active connections but it seems like thats not the case? The Heroku Dyno was still connected to the Redis Service.
Anyway, shutting down the dyno reduced network usage as expected
how was the dyno generating so much network usage lol
Hmm, yeah, I am not sure why there is so much inbound traffic. I sort of expect the outbound. We're using the Redis Adapator with Socket IO to stream realtime multiplayer gameplay from Railways to the Heroku Dyno
anyway, I'll have to make sure I didn't write something too stupid on my end.
but atleast the egress issue is solved
tldr user error, though it is curious that the dynos connection to the Redis DB was not 'severed' when I disabled public networking
i will ask the person who wrote the tcp proxy if that would even be possible
but glad you where able to solve!
all the more reason to shut down all your dynos 😉