Old builds keep running in the background
I'm running a nodejs application on railway with bull queue. After a new deploy, it seems the old instance of my application keeps running in the background instead of shutting down.
This causes jobs to get occasionally passed to the old instance that is running an outdated version of our application.
I would like railway to only have the latest version of my application running at any point in time.
21 Replies
Project ID:
8929a4ad-5433-4d16-a231-4d6a8d23a0128929a4ad-5433-4d16-a231-4d6a8d23a012
the
RAILWAY_DEPLOYMENT_OVERLAP_SECONDS variable may be applicable
https://docs.railway.app/develop/variables#user-provided-configuration-variables@Brody This is more than seconds. It's like days. The new deploy gets deployed alright and handles all new API requests. However, some other instance of the application still pick ups some of the jobs. It's as if the old deploy is still running(in the background) although it's no longer handling API requests.
As I speak atm, a version older than four days is still handling some of the jobs. Despite the latest version handling all API requests
ah okay then that would be what we call a phantom container
I will flag for team review and they will hunt down your phantom container when they have time
okay
will this be a temporary fix or a permanent one
Also, what usually causes this?
temporary, and I am not aware of the exact cause, I don't think the team is either tbh
Hmm okay
Because we would like to reliably deploy stuff
This currently happens in prod
Kindly let me know once it's resolved
I don't know when the team will have time, they are very busy lately
Any ETA? So we can decide whether to switch to a different cloud service for now
sorry I don't have an ETA for you unfortunately, I have already flagged the team for you though so they will help when they are free
Hey @Greal - to be clear, you are trying to get requests to go to the new deploy?
Also this isn't a phantom, rather this is a deployment overlap issue
Shouldn't be days here
May I ask whats the usecase?
No, requests already go to the new deploy.
We use bull queues in our application. So when we pass a job to a queue, it gets picked up by any available instance of our application that is least busy.
it seems like a phantom container to me? not entirely sure though
However, we noticed there are other instances running the old version of the app on railway that pick up the jobs as well. Even after new build has been deployed.
Looking into this as we speak
okay thanks
Hi @Angelo , any update on this?
Hi @Angelo, pls could we get an ETA for this?
just a friendly reminder that if you're looking for priority support you may want to look into the teams plan
Yh I just need an ETA, so we can decide if we can wait or move to a different provider in the meantime. You can take as long as you like to fix it. I just need to know the time frame @Angelo
This is a bug. Isn't it?
This is a bug- forgive me, just been running around
ETA EoD
Forgive me, old tasks took a bit too much time will knock this out in the morning
Okay π