Service is down
hi guys my service is down:
https://railway.app/project/19be19f8-fe9a-438b-8425-b269d8c24d06/service/364380c7-b21a-4fde-9f86-ec63b19110e3?id=d3ecf291-3ff8-4c8c-99ff-8e90c8330bd2
it has been just hanging, i dont get any notification or indication whats up. any chance there is a diagnosis? this has been happening frequently on multiple apps
want to figure out of this is user error or an issue on your end
much thanks
Railway
404 - Page not found
Railway is an infrastructure platform where you can provision infrastructure, develop with that infrastructure locally, and then deploy to the cloud.
37 Replies
Project ID:
19be19f8-fe9a-438b-8425-b269d8c24d06,364380c7-b21a-4fde-9f86-ec63b19110e3,d3ecf291-3ff8-4c8c-99ff-8e90c8330bd2still shows successful in railway?
but the logs stopped a while ago
which is the issue -- silently stopped
now its gone?
getting a 404 on the project now 😯
for instance: http://chatwithwebsite.sdan.io/openapi.yaml
this link should work. if it didn't it should show up in the logs. but logs stopped and now project is also gone.
the project is gone???
yep
like the entire project just disappeared from your dashboard??
i can click on it
but it shows up 404 right after i messaged u here
Railway
Railway
Railway is an infrastructure platform where you can provision infrastructure, develop with that infrastructure locally, and then deploy to the cloud.
and refreshing just continually gives you a 404?
I'm getting 404 on my projects too
yep just refreshed a dozen times
6a6c4123-f4c9-4c41-ba5f-c3115ddc1753
@Angelo - something not good is happening
lmao now all my projects are pointing to 404
ill try logging in and out
same thing
#🚨｜incidents
calling incident
reverting, thanks for catching @sdan
thanks but my initial issue was one ive been having for the few weeks
can you refresh my memory here, you mean on the intermittent 404s?
or the deployment hangs
alright all back up! however if you are capable, pls look at this: https://railway.app/project/19be19f8-fe9a-438b-8425-b269d8c24d06/service/364380c7-b21a-4fde-9f86-ec63b19110e3
Railway
404 - Page not found
Railway is an infrastructure platform where you can provision infrastructure, develop with that infrastructure locally, and then deploy to the cloud.
the logs stopped at 00:51
i want to figure out if this is my issue or issue on your end. i have similar services that have the same logging and are still streaming logs. but this service has stopped logging entirely
how long ago was it deployed?
this website powers https://chatwithweb-production.up.railway.app for instance but it doesn't seem to be running
a day ago
logging issues are definitely on their end
my end?
railways end
alright i know i just brought this up a minute ago but any chance you guys can figure out if this is an actual issue? like logging or something else? i have to restart this service which runs for like an hour it seems and then stops.
hoping to make sure this is an issue on my end before continuing
notice the cpu usage for an hour and then drops? i am unsure if i am causing this... i am running a flask app
your app stopping would be on you, loops like a soft lock from an out of heap error or something along those lines, but that's a big guess
and if the logs stop when the service stops responding, that would no longer be railways fault either
Yep- the logs is an us issue. I am going to flag it to Greg as its related to our clickhouse migration.
heap error... is that a memory thing? i thought i get 16gb ram?
Re: app, I have so much consternation with the Python runtime, you can run into heap errors even if there aren't container caps.
One sec digging
yep or gunicorn just silently stopping a worker if it's using too much mem
8 GB of RAM fwiw
How many workers are you running?
just 1... could be the issue
can you bump to 4
cool will try, thanks angelo!
maybe also worth looking into running gevent type workers
seen lots of headaches from running sync workers