Horizontal Scaling -- Scaling controls
We're currently using Railway to host a critical node microservice that processes long-running tasks. Our architecture utilizes BullMQ for task distribution from a coordinator across multiple node workers. I would like some guidance on the ability to control how my scaling is done:
When downscaling, we need to ensure that the long tasks are not abruptly terminated, causing a data loss. Is it possible to target and gracefully kill specific replicas? This would allow us to implement a controlled downscaling process where we can ensure all tasks on a replica are completed before it's terminated.
We're considering using
process.exit(0) call if it works. Would this approach work within your infra? If so, at what point in the scaling process should we trigger this - before or after making the API call to reduce the number of replicas?
Currently im planning on using the railway API to add and remove replicas but I don't know how to control how these actions are completed. Internally I can track the RAILWAY_REPLICA_ID without an issue but I have no way to target a replica in any of the actions.
Thanks!Solution:Jump to solution
ideally, you would run say 5 replicas, and each of those replicas would use a small amount of memory at idle and then each scale vertically when needed
18 Replies
Project ID:
5ffdeb0b-59bc-4b91-9046-48f61f35249e5ffdeb0b-59bc-4b91-9046-48f61f35249e
question, how much resources does an idle worker use?
prob like 10gb of memory because it has a couple hundred counccurrent ws connections
i guess i could kill those ws connections
thats a lot of memory for only a couple hundred connections...
its a lot of data lol
yeah but when its idle why would there be data flowing?
because accounts would be marked as inactive but still have running tasks
i could poll prisma and kill on the node
respectfully, this sounds like a architecture issue that you are trying to patch with replicas and a home grown autoscaler
Solution
ideally, you would run say 5 replicas, and each of those replicas would use a small amount of memory at idle and then each scale vertically when needed
imo architecting your app to run like this is going to be a far better use of your time
the issue is each nodes operations are statful but i guess keeping 10 - 20 replicas is a good idea
and will scale better!
i agree, thanks for the insight
also thanks for the quick support
i just have scaling to zero in my mind too much
:Laughing_pepe:
i work a lot with stateless infra so im used to doing it this way. but i agree with what your saying and i think thats the best approach
thanks again!
you are on the right track, but you want your workers to be able to scale to near zero.. vertically instead