Limit CPU Utilization on cli imports?
Is there a way to limit CPU utilization on CLI Imports. Anytime I import a large set of images, my entire host goes unresponsive. Using Docker on a 8 core, 24gb memory server. I get that it takes awhile to process, but I need a way to throttle it so it doesn't bring the host to it's knees but I couldn't find anything. Any ideas?
22 Replies
try setting all job concurrencies to 1 in the admin page.
oh well gosh, I'm blind. I never even noticed that "Manage Concurrency". LOL. Thanks.
hope your server doesn't burn up again lol
I'd guess specifically the ML jobs are what's causing the problem, so I'd start with those.
Man, even set at 1 for all the concurrency, it's still punishing the server. It's more or less functional - but is there any other throttling possible?
It's the machine learning and microservices containers doing most of the heavy lifting
Need to be able to leave at least 1 core alone . lol
i would suggest pausing either the ml tasks or transcoding
the jobs will still queue up but they won't start until you unpause
make sense. surprised at how hard it's pushing the system. I mean it's old CPU (i7-6700K), but it's not prehistoric
yeah i do wonder why it's pushing it so hard
My wife is not going to be happy when I use this as an excuse to buy a new rig. cough
you could also try changing some of the transcode setttings, e.g.
threads
lmaoYea, there were some mov's in the batch I just uploaded. Maybe it's ffmpeg. I'll set it to like 1 thread and see what happens.
that and be sure the target resolution isn't too high.
4k and original can be pretty intensethe source mov's are pretty small. I'm only in my 2015 year of photos right now and that camera wasn't doing 4k 😄
Little better. machine isn't crying for mercy. That's with all the jobs and ffmpeg threads at 1
Thanks for the tips and guidance.
anytime
also from a performance perspective it'll be a bit faster to pause some of the tasks since it's better for cpu caching
but it's also fine to leave it like this if you don't wanna mess with pausing and unpausing
Got it. I'll look into that as well for the really large folders.
So the gist is that if you don’t limit the CPU for the container/stack, it will use all your the cpu you let it to use
Good to know! I’m running unraid and even with a Xeon 48 core processor, Immich was taking up all compute capacity 😅 things are a lot better after doing core allocations to Immich. Thanks for the suggestion @Alex