Logs of Slow Metadata Extraction on v1.82
As requested by @etnoy , here are the debug logs.
First of all, when clicking the "All" next to metadata, immich microservices seemed to crash because of lack of memory.
See logs (I apologise for the format, I'm running on synology and it is saving the docker logs as a .db file so I'm doing my best here).
Hope this helps!
https://pastebin.com/0RSTwYDD
Pastebin
2023/10/17 13:58:45 stdout [32m[Nest] 7 - [39m10/17/2023, 12:5...
Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.
11 Replies
"JavaScript heap out of memory"
yes, and that's before the metadata extraction actually began
then the microservices service restarts, and doesn't have any more errors
I don't think it has anything to do with memory. My metadata extraction is also slow but it's far from using all available memory
exactly, and after the service restarts, there are no more memory problems
I feel that this may be a non-issue actually. I suspect that for some reason this job prioritises "heavy" files such as videos first and they take longer to process. After my job chewed through first ~2000 assets it got much faster.
it used too process ~1 asset in 10 seconds at first. Now it's 100s per second
There are actually two separate jobs that run per asset the second ones pile up, but are super quick
So the overall stats don't really reflect that
Looking at "waiting" count, 53k to 51k took 40 minutes, 51k to 25k took maybe 5
When one finishes the first job the second is queued so there is no visible "progress"
...and now it's all done!
@avee interesting, I might let mine run and see what happens
@avee you are absolutely right. It seems to start with the large files. Mine stayed about 2 hours super slow, and now it's blazing fast
As jrasm91 said, it was always fast, it was just that every job queued another job so the counter didn't go down much until the end.