Files stuck in upload directory...?
Hey, so full disclosure, I was testing Immich and had just set it up on the laptop, and someone accidentally moved the harddisk that has all the data, temporarily unmounting it. I did the most reasonable thing and restarted the containers; my backups in the mobile app restarted and there were no weird error logs like I'd have expected given the entire drive was suddenly invisible to Immich.... great!
But I noticed a huge discrepancy in the used storage (108 GB in Server Stats and 149 GB in the left sidebar). I checked and saw ~18GB of images and videos still in
/upload/<uuid>/ folders. I tried manually starting the "encode videos" task from the UI as well as others, and left the server running for a day, and then restarted the server again... no dice.
Any clue what's going on?21 Replies
Bumping this, the issue still persists.
Have you looked at the jobs? Is anything marked as failed? Could you try running the storage migration job manually?
No job was marked as failed, and I still reran everything with the "pending" button.
If there's a way to ensure Immich isn't planning to touch those files now, could I just move them somewhere else and import with the cli?
Sorry, did you rerun the "storage migration" job manually?
Yes.
Any logs in the Immich microservices container when you run the job?
Sorry for the delay, but did manage to run the job again. Here's the only log I see:
I ran the microservice twice.
Output of tree immich-app/data/
Clearly, this didn't help. :/
Should I attempt copying one of these files and then making the CLI 'bulk-upload' it? The hash would match, and that might trigger the entire import process again...
I went ahead and did that. Here's what I've found so far:
I copied 6 files to a temp-upload directory and ran
immich upload --key {} --server {} temp-upload, which seemingly succeeded. Then, I checked the logs for the microservices container, which had a bunch of ffmpeg errors.
These are the files I uploaded:
The ffmpeg errors were of the variety "moov atom not found", but crucially, they were on two files:
The full log:
I'm slightly worried now cause there are some partially uploaded jpegs in there too:
Huh, this is very weird. So, all the images out of the 127 files in /upload that I randomly picked (~10ish), are all actually uploaded and showing up properly in the app, and their partially uploaded/currupt versions seem to be left here.
In my
test-upload experiment, this is what the identify check and the uploaded file look like:
I'm totally hypothesizing here, but it seems that for some reason uploads failed, Immich correctly determined that those uploads didn't complete and then restarted the uploads.
Then the new upload got a new UUID associated with it, but for some reason the app didn't clean-up the partially uploaded content.
If I then forcefully reupload one of those partial files, it does get uploaded (as it should).
Thus, the bug appears to be only in the clean-up process when for whatever reason an upload didn't complete.
Thus, I can safely remove all the pending files in /upload
Gonna wait now for you folks' input. /end
Yes that makes sense. The app should delete files when they fail to upload. Although, I could see this happening if the server was restarted during an upload or something like that.
Hmm, maybe a start-up task to look for pending cleanups?
Although it is hard to know for certain if you can ever auto delete anything
Correct me if I'm wrong but there is partial filtering support already..
exifInfo.make|model or smartInfo.objects|tags in the /api/search endpoint, when used with clip:true should essentially filter clip results with those exif filters?Yup there is API support for a lot of filtering already.
Oops, no clue how this ended up here. multiple discord windows lol