Issue with Redis Outbound
Hi, I have a doubt about Redis, Im using a Redis service for my API, both deployed here in Railway. Cache works fine right now for what im doing but theres an issue, its showing some serious (100 MB+ of outbound network) on a request with the cache, is that a bug?
138 Replies
Project ID:
N/AN/A
N/A because im not sure which id to show, the backend id, or redis id?
And i just saw this: , does that mean im doing it wrong? Sorry im newbie with this stuff
you should check the variables on your app's service, not the Redis service
oh, okey, im going to check that now
mm I dont see any:
not sure if im doing it right
im confused right now I read that but im kinda same, im a noob at this
look at the picture
the picture i sent?
the picture in the docs page i sent
ah okey
^
im looking at it but i cant figure out, i mean what i understand is that some variables that are used in 2 places must be placed here, right
right
no, you have no need for shared variables
but this worked before for me at some point which is even more strange
ah right thats why it worked, cause they not needed then
I got these envs right now on my 2 services
wait when you say app service you mean like, my service-backend-project, right?
yes
send a link to the project please
here in my code i reference this
yes
Railway
Railway
Railway is an infrastructure platform where you can provision infrastructure, develop with that infrastructure locally, and then deploy to the cloud.
youre good, its using the private url
and REDIS URL seems a string I dont even remember where I got the value from, maybe it should be reference variable?
aah okey so im good there
it already is a reference variable
okey ā
so thats ok
but yeah outbound is still like, big, it wasnt at some point god knows what happened not me xd
i dont really see any outbound on redis
thats not the redis service
cause im just basically getting the data from redis on cache in theory
ah
wait can you see where it comes from? i got a login and user-settings but no way thats 130 mb
no i cant
maybe my cache is just not working now? but it works locally, how could it not work same on railway? strange issue
i think maybe i messed up railway redis
because locally i always do redis flush all
but i cant flsh all on redis railway service, is there a way to flush all?
maybe old data is caussing some issue idk
again, you are connecting to redis via the private network, so there would be no measurable network
right, so all that data is 100% not redis then... mmm what could it even be idk
its any outbound data leaving the private network, could be mongo, could be sending data to your frontend
side question, where is the frontend service?
yeah its here, also having an issue with it where it crashes on a certain page when it tries to fetch some data: https://railway.app/project/e1a3daa8-0161-42a8-9945-2ddf4a33a66d
Railway
Railway
Railway is an infrastructure platform where you can provision infrastructure, develop with that infrastructure locally, and then deploy to the cloud.
why is it in a different project?
wait what? i just see theres no repo connected to it, wtf
cause i had to create new project when i was setting it up
or maybe its mistake and should been a service?
but how is it no repo connected
ah wait right
because i uploaded with the weird react method
OOH
via cli
then its not updating
like backend
f0ck
^
funk
right
hold on hold on
how can I make it so that it does like backend it auto updates on git commit? is possible? that must be the issue 99%
cause i did commit pushed and thought already frontend updated like backend but no
thats gotta be the issue
1st question: can i fix it where it is now then move it to the same project?
delete it and deploy it into the correct project this time
ah okey, lets do that
if i rename project, does it affect anything in terms of functionality or its just name change?
just so instead of backend project ah i change it to project ah and put all the 3 services together: backend, frontend, and redis
just a name change
okey i changed that name now
so for the react thingy is there a way to make it like, automatically build from github repo everytime or i gotta do the cli setup every time?
lets get the service into the correct project first
yeah ima delete the frontend project 1 sec
done
right, so first of all i create an empty service
then i use this, right: https://docs.railway.app/guides/react
oh theres a one click template option, but i never used template before
no, you already did that.
empty service -> name the service -> set variables -> deploy -> create domain -> attach github repo -> deploy
ah, okey
theres no variables in the frontend except the process.env.REDIS_URL I think being only exception. So for that I gotta use reference variable I believe
the frontend does not need redis
ah right
you are right
the frontend should need a backend api domain variable though
for now im just hardcoding the values but will change in future dont worry
okey so i cant see the domaint hingy
yeah i cant see it in settings
fixed
ah deploy
okey
shows me this
i click there to make a deployment yes?
idk
id go with the big deploy button
oke
it just sends me here: https://docs.railway.app/quick-start
no it does not
wait
ah no
the purple button you meant right
the deploy one, wasnt looking at it
it shows me this now
it seemed pretty noticeable to me lol
sorry im not used to the interface yet
i miss on obvious stuff
continue with the steps i gave you
i been user of railway for few years but actively went back just like few days ago
okey
okey i created domain, next is attach github repo
and next deploy, okey
okey its doing stuff its loading
YEEES
GOOD JOB!
it was this thing man, i can finally see my data again my stuff š„¹ nice!
lets see if it also fixes the outbound stuff, right? because now in theory it same project
okey so 1st request is this: 130mb, completely normal, data is not cached (i need to re-cache every hour btw its not a permanent cache)
so now i wait like 2 mins and reload the page again to re-fetch, in theory from cache and hopefully no more big outbound data like this 130 mb
no it wont, to avoid outbound traffic you need to use the private network and you cant use the private network for client side rendered websites, or 3rd party services such as mongo
wait...
š¢
if i recall correctly ive said that a few times
you probably did but i did not get it, im too slow with all these stuff im new to it and my mind if like close to explode plus im coding so much these days learning stuff too im like exhausted mentally tbh
so theres no absolute way to cache the 130 mb response for the frontend theres no trick or workaround? my app literally wont be profitable if I dont find a way to do that idk what to do š¦
and i spent months on it omg idk
dont send 130mb worth of data to the frontend then
right...right, i just cant, it defeated my entire plan but theres no work around
though im just going off that you are saying, id first figure out what this outbound traffic is
must think of something
ah
yeah its the 120k items, but its weird because the size is about 65 MB
55 to 65 MB, no idea why it shows 130 MB, its like it fetching the data twice, but is not.
figure out where this outbound traffic is going
ima try
is your frontend actually downloading 130mb of data?
only 65
cant be a coincidence than 130 mb is twice 65
we are not double counting outbound
else {
console.log("through the else");
const userDesiredItemsFromCache = await retrieveArrayFromRedis();
res.json({
items: userDesiredItemsFromCache,
update: {
nextMinute: timer.minute,
nextSecond: timer.second,
},
});
}
thats my cache code
maybe retrieveArray is doing it twice idk need to see that
doesnt seem like
are you console logging 65mb worth of data?
yeah
lets not do that
that could be the issue maybe?
no
ah
yeah i removed the console.logs
rest of code on the cached version are these:
theres an if below, but im going through the else on all cached (verified im going through it)
find a way to not send 65 mb worth of data to the frontend
yeah thats something i need to do, but lets ignore this for a moment now and try to see why it takes 130 mb
im not doing a double operation there and im just fetching a timer from mongoDB thats an object with 3 fields
connectedRealms is not even a fetch but its an object with like 100 fields, neither can be that
dont really know what to tell you, i have no way of looking at your network traffic
and getHighestTime returns about 300 KB
ah right
well no worries I will have to change all of this somehow and hopefully that fixes that issue as well, who knows
but yeah
brb dinner
Hi Brody, I got it way way lower now frontend response is about 2 MB, for some reason it will still shows 4 MB but i think thats fine.
So I want to try and calculate the cost of a user network usage based on this, you let me know if what i say makes sense
scroll to the bottom - https://railway.app/pricing
So a user makes request of 4 MB to my app, he will lets say, do it on average around 4 times per hour, so 16 MB per hour. I expect the user to (at most, lets put higher case) use this app for 8 hours a day, 25 days a month. so thats 3200 MB, 3,2GB
so that should be 0.32$ network usage
does this make sense?
yeah I know pricing its 0.1$ per 1 GB of network egress
just want to make sure I understand the thing with this simplified math I did for an estimate
ooh
scroll to the bottom please
I can make example there, nice
okey I do example there thanks ā
switch to resource-based
yes
ā
well you absolutely will not be using 0 cpu and 0 bytes of memory
and that's only 1 users worth of networking
oh yeah yeah
i meant just to calculate the network of 1 user, yes
cause then I have some doubt, RAM for example, it takes around 400 MB of RAM per app load, but like they just load the app and then they can see all data they dont need to reload often just once every hour for user, for new data to come in because external API updates every 1 hour. so maybe they are using the app 1/3 of all the entire time of a day, so that would be 400 MB of RAM divided between 3, so 133 MB RAM average cost per user, does my example make sense?
nope
oh, how do i calculate it then
put in your average cpu and memory usage
right its what i mean, the average it is the cost for RAM and CPU
so put in the average and it will tell you the cost
so if user peaks 400 MB on app load doesnt matter because lets say he does 1 app load per month then average would be super low, like 10 MB or whatever
right
I highly doubt your app could idle down to 10mb
especially with replicas
you think even less, or more?
far more, it's node
ah no no, forget the app, it will have a base RAM cost
but im thinking the added RAM usage per user right now
idle is probably 300 MB, but per user maybe on average it adds 10 MB, is what I mean,based on my above example here
then put that into the calculator
average is good because nobody uses app 24 hours, at most they will use 12 hours and 99% of people will use less
yeah
another question: lets imagine im hitting the 32 GB RAM limit, can I add 1 replica and then it scales about 64 GB of RAM? (I know its not exactly like that in theory, but in practice, in a very simplified way of putting it, its basically double hardware power, right?)
and another replica for 96 GB of RAM, etc etc...
it's still 32gb of ram per instance, just now that you have two running the total combined can be 64gb
yeah its what i mean, the total combined
then yes, but that may be subject to change
so theorical limit is crazy high because if I have 1 service + the 49 replicas available is 1600 GB of RAM
but dont worry cause my bank account is little
and i will have little server
im not rich xd
exactly, and what's why I say it's subject to change, that's far too much resources for a single user
but its good to know the limit, i think i will never find limit in this platform which is possitive to hear š
yeah it really is lol