Node Memory and CPU Usage is low
I have a node instance that I'm running with tsx / node commands. I notice the job is slower when run on Railway than when on my local machine. Yet, the Railway memory and CPU usage are very low. How can I have the job use more CPU and memory on Railway?
10 Replies
Project ID:
98f66160-ce86-4b15-a73b-cab91225620898f66160-ce86-4b15-a73b-cab912256208
how low are we talking? what is the job execution time difference between hosted and local? what does this job do?
please provide more information
low, like 4-6 gb memory, and 1-2 vCPU
on Railway the job never completes! I let it run for 3-4 straight days.
Locally it completes in 12 hours
The job makes a large amount of database calls, then parses location data. The database is not the blocker though, its the local compute that checks various @turf / geojson functions
If you have a job that is running for 12 hours on a local machine then it's probably the sort of thing you may want to batch so its not an all or nothing job where you will lose work if the server resets on you or is disrupted for any other reason.
If the batch chunks are small enough and you can monitor progress you can see exactly how much of a performance penalty you have for running in the cloud vs. on bare metal.
Thanks for the idea! Is there some way to run like X workers per batch on Railway on different instances or something? It seems one railway instance is not enough to run this job
any update @Brody
while its quite possible you have faster hardware locally, there is not enough evidence here to support the fact that this could outright be an issue with the platform.
i would take joseph's and my advice into account, batched jobs and verbose debug logging.
How can I do this with Railway? the batch will stills be run on the same railway hardware no? Can replica's handle this/
its something that you would need to implement in code, not really anything to do with the platform itself
if you need to split your job into replicas, consider using a queue mechanism like bullmq.
but tbh I would look more in detail on why its taking that long, 3 days difference is weird, is your database and worker on private networking?
If it was me, I would start by adding Axiom and ploting a graph on db calls, locations parse, everything that takes time and see the diff between local and railway