App is down
please help
114 Replies
Project ID:
N/Aplease provide more information
i upgraded
instance to pro
requests failing
due to high load
how many connections is "many connections"
100,000
or so
not clear
concurrently?
yes
what can i do
i see this in memory
can we jump on call @Brody
its urgent to fix
thank you
i can upgrade to any plan, already on pro
i don't work for Railway
oh
is there anyone on railway that c help
but I can tell you with absolute certainty that you can not do 100k concurrent requests
ok
railway is down
nevertheless
railway isn't down, they just aren't allowing you to do 100k concurrent requests
what can i do for them to allow
quick question, are you on the new proxy
dont know
please check
i didnt setup anything
how
in the service settings
where
that's your project settings, look in the service settings
ok enabled
is this 100k of legitimate traffic?
yes
its real users
100k is estimate
did something go viral?
just link to app was shared in group with around 80k users
so yes
i thought railway will handle it
in the mean time you can try spinning up more replicas
how
service settings
how to upgrade my instance
im confused
i press this
it wants me to enter name
Nikita?you are upgraded, you are just selected on the hobby workspace, you need to move your project to the pro workspace
how can i move
while keeping same
url for deploy
you would have been asked to move projects over upon upgrading, if you closed that you can move a project across workspaces from your project settings
but pro users do not get higher request rates
so this won't help?
not for your particular issue since you are not CPU memory bound
you need replicas for this
i have 4
is that enough?
when where they added
2 min ago
if I understood mig correctly each replica should be permitted to allow for 1k RPS with 5k burst
5k burst?
yes
i don't know what the window for that is though
let me know if more replicas make a difference
still failing requests
its unclear why
its on railway though 100%
as I am in close contact with db
and all writes go through there
how are requests failing, please be more specific
its infinite loading
so requests just stalling
for many users
project id please
how to get
from your url
I'm sorry but I didn't ask for an invite
Railway
Railway
Railway is an infrastructure platform where you can provision infrastructure, develop with that infrastructure locally, and then deploy to the cloud.
@Mig connections hanging with increased traffic
pls resolve
thank you ā¤ļø
I can add 20 replicas
not 4
i guess
but not sure what cost of many replicas is
and if thats the issue
connections just stalling
i need all requests to go through
there is a limit on the total amount of RPS from the host itself, that would be 50k with 500k burst
there's no base cost, you pay for the resources used by them
so what can i do
to make requests go through
im messing around with caching on client now
you can try adding more replicas, but ultimately we will want to hear back from mig
i don't get though why connections stall
graphs look fine
yet connections stall for many
connecting to railway
as I've previously mentioned, you are likely hitting your RPS limit
how to up rps
more replicas
ok 8 replicas
but you could also be hitting the RPS of the host, so we will need to hear back from mig for a definite answer here
requests are stalled again
š
fk
i hoped railway would be ok
to host my stuff on
if it was personal project i would be fine bu this is client project
its not even free plan
its crazy that it stalls even on 8 replicas
now its ok again
i have asked another team member since mig is offline
i added client side caching
at least for users that loaded before they will see things
thanks
though its monday so they are likely in meetings
Hi!
It looks like the drophunt URL is returning quickly here and you were able to bump replicas
@nikivi lmk if you still need something here
yes pls
i expected railway to support many requests
and many users just had stalled requests
so client probably won't work with me again but oh well
my issue with not understanding railway i guess
but i want to make sure this does not happen again
I was not on free plan and was in close contact with the db I used so all writes to it went through, all issues were on railway
and even now some requests still stall
with 8 replicas
for some reason I thought railway would not just drop requests on non free plan
I guess I should start launches with maybe 20 replicas just in case but that seems like too much
I wish I could just scale the load and so that railway won't be dropping a single request
would be wonderful
I assume railway infra should allow for this
that was 14 separate messages, please try to keep such things contained in as little amount of messages as possible
yea sorry
how can I make railway project where I don't have to think about replicas
and where railway won't be dropping or stalling requests 100%
I am coming to railway from cloudflare workers where thats the default
from ux perspective I assumed that would already be the case, and that things like
proxy would be on by defaultwhat you switched on was the beta proxy, its not like your app didnt come with a proxy already
so issue is that railway just drops requests if too many requests land to the replica instance, it does not queue them or anything
I will always turn on new
proxy (it seems to be better, not sure)
and ideally I don't want to provision replicas as I don't know load beforehandyes that is how limits work
can I make railway work with these requirements
I did not set any limits
the platform has RPS limits
ok will try move to self hosted solution with no rps limits
i do like railway ui though
š
If you're doing self hosted, you're gonna have to manage this
Plus a littany of other issues. Can you show me where the requests are being queued? You seem to have more than enough headroom
You're running at 700mb
somehow requests were still stalling though
i did not get observability into why
at least I did not see in railway how
it's also possible that your app just simply wasn't responding fast enough
Yup. You've got a lot of headroom so, from what I can tell, it's something at the application level
Maybe a downstream service, some errant code, etc
can i cut replicas to 2 at least
what is the approximate level of replicas i have to hold for how many concurrent requests
i wish i didnt have to think about this š¦
we both no longer think this is an issue with railway, after some thinking it's unlikely you are hitting any RPS limit considering 8 replicas did not solve this issue
so with 1 replica
if i have 10,000 concurrent requests
hypothetically
what will railway do
sorry for the confusion, but all signs are pointing to an issue with your application or something your application interacts with
trying this š
lets see
now traffic is lower
+ i have caching
so should be safer
please try to limit the amount of messages you send back to back as I am bombarded with notifications
@nikivi is this open source
GitHub
GitHub - nikitavoloboev/drophunt
Contribute to nikitavoloboev/drophunt development by creating an account on GitHub.
p.s. can I clear logs in dashboard
this is not the whole thing
also - what is ronin
my data store
its whole thing
Solution
yeah so that is your bottleneck
ronin are not bottleneck as they checked their writes coming through
i was on call with them
i would suggest instrumenting your app with something like Opentelemetry and see which call is slow
yea will do
i cant run this locally
yea its bit annoying, will resolve in a bit, ronin uses private packages
well - i can't really help much until then
yea understandable
thanks for offer though ā¤ļø
what's your reason for using ronin btw?
it seems to just be a database?
yea they provide a nice sdk to query https://github.com/ronin-co/client
GitHub
GitHub - ronin-co/client: Access RONIN via TypeScript.
Access RONIN via TypeScript. Contribute to ronin-co/client development by creating an account on GitHub.
and dashboard
im about 99% sure your issues lie in either the SDK they provide or the service itself
i would suggest using a normal database along with something like drizzle-orm
railway supports postgres and mysql