RAILWAY_DEPLOYMENT_DRAINING_SECONDS and debugging graceful shutdowns
Hi there, I saw that I could set
RAILWAY_DEPLOYMENT_DRAINING_SECONDS to give my change an app to shut down gracefully.
I have some logic in there that works fine on Heroku - I can see my dynos cleaning up and shutting down when I scale down my dynos. I see all my clean up logs and messages.
When I try the same on Railway, I don't see any logs. I am not sure if my app is gracefully shutting down. I am not sure how to debug this.49 Replies
Project ID:
N/Aare you catching sigterm?
yes, should be, i use
enableShutdownHooks in my nest app which by default catches...
do you see logs during regular app activity?
yeah I see my app working normally, and when I restart or redeploy my service it just 'cuts out' where it is and then the new one boots up
okay thanks for the info, ill try to get a proof of concept working
ok that would be fantastic, thanks. I thought maybe it is working in the background but the logs aren't being captured anymore? But i have no way of verifying it without seeing the logs
one idea popped into my head, what logger are you using?
just the default one what nest comes with
have you disabled log buffering?
just checking, sorry
i dont specify, and I can see that the default for bufferLogs is false
sounds good, i shall get back to my tests
:salute:
i will let you know if i discover anything, i am poking around too
i wonder if it is because i am using newrelic and starting my app that way, ie
node -r newrelic dist/apps/ssr-game-server/main
maybe the sigterm is not being propogated properlythat is very possible
i wouldn't know if newrelic forwards the signals along, are you doing the same on heroku?
no, so it very well might be the culprit
try without it
roger
it does work, i set it to 15, the default being 3
any luck taking newrelic out of the equation?
no luck here, hmm
nixpacks or dockerfile?
nixpacks
iirc nixpacks wraps the start command in a shell, that also might not be passing down the signals
hmm...
how hard would it be to write a dockerfile for your app? it would give you much better control and allow you to ensure the node process is started in exec mode
I have no experience in it, so honestly I have no idea! I am transitioning from Heroku to Railway, solo indie game dev wearing so many hats, haha. My experience is limited. Though I am not doing anything fancy. It's just a boring old nodejs/nest app
end of day now.. tomorrow problem...
thanks for all the help thus far
no problem, ill see if i can replicate this issue when starting my app wrapped in a shell like how nixpacks starts node apps
there is an issue on nixpacks about this https://github.com/railwayapp/nixpacks/issues/1032
but neither me or dush can reproduce, and i've tried wrapping my app in a bash and sh shell.
i think it might still be worth a try to write a dockerfile, i can try to help you with that if you'd like?
ok, back in the office...!
i'd love a hand with that, sounds like a good test considering the open issue
for me to have a good chance at writing a working dockerfile for you, please share your most recent build log, anything you have changed nixpacks or service settings wise, and your repo
Hey thanks for offering, i did some more digging and I belive i've found the root cause
oh yeah? do tell!
but having a hard time verifying as there seems to be some issues with logs right now..? I have a scattering of logs missing from obervability
hmmm I haven't seen any other reports of that, what kind of logs are you missing?
my deploy logs for a fresh deploy look like this:
- nothing
- business as usual
It's missing the typical 'Nest app started' etc. Its not even visible in the Observability panel when i narrow down to the specific service ...
incorrect log level? are we sure next doesn't buffer logs by default?
my typical logs look like for the same service
mmm pretty certain, and I haven't touched that
anyway, lets ignore that for now, I did observe a log that shows my shutdown signals are working
what was the fix?
though, i mentioned the log thing because this is just a few of my graceful shutdown logs haha (I have some logs missing within the same function that those 2 are from hah)
Solution
the fix was not to use
npm run ... and use node ... insteadhttps://help.heroku.com/ROG3H81R/why-does-sigterm-handling-not-work-correctly-in-nodejs-with-npm
so for me it was a matter of changing my custom start command from
npm run start-game-server:newrelic
to
node -r newrelic dist/apps/ssr-game-server/main
which is just what the start-game-server:newrelic command is, anywayI thought you where using
node -r already?no, i had a custom start command as i have a mono repo and had to specify which to run
ah I see
anyway thanks for helping me trouble shoot 🙂
happy to help where I can
if you are still having issues with logs, please open another thread! 🙂