Migration Microservice Error
When attempting the new migration seems like there are some issues on my end. One user successfully processed and has no images left in /UID/ all moved to a sub folder. Another user has some folders with images created under /UID/HEX but also still has some remaining items. Looking at the Microservice log I see: Seems like the migration dies after a little while at this step. (Seems like it's trying to copy a file it already copied?)
15 Replies
@amdhunter1989 Not sure if this is a similar issue you are seeing
Similar, but trying to re-run the job on my end to see if I get any errors. I saw some errors here and there, but as far as I can tell the job did run. I need to do a count on the directories the job created to see how many files are actually in the dirs
in trying to re-run the job I see similar errors. I did not see this type of error the first time running the job.
I am watching immich/thumbs dir, and I see the number of items in the root dropping. I am at 8k or so, started with 15k
@schuhbacca how many files were left in your thumbs/UID folder? Can you re-run the job and watch the folder to see the items being moved? I am still getting errors like the above, but the job is running and moving files.
Also, I am seeing the same behavior I described above in the encoded-video folder. Whereas before, i had many items still left in the encoded-video folder.
We've discovered that it actually is possible that there are literally thousands of thumbnails left in the folder which are actually orphaned and not being used anymore
So it isn't unlikely that the job was succesful and those are just old files
My second run of the job finished, and it moved most of the files. I have 241 and 583 files left. I am going to run it one more time, and see what the count is.
If these stay static this time, would it be Ok to delete the files? this is in the /thumbs/UID/ dir. Similar, but smaller number in the encoded-video dir.
If these stay static this time, would it be Ok to delete the files? this is in the /thumbs/UID/ dir. Similar, but smaller number in the encoded-video dir.
In theory, yes. To be sure you can try to verify using https://docs.immich.app/docs/guides/database-queries that those files aren't being referenced in the DB anymore
Database Queries | Immich
Keep in mind that mucking around in the database might set the moon on fire. Avoid modifying the database directly when possible, and always have current backups.
I could do that. But I don't think that querying the DB to check thousands of files would be very doable, LOL
Actually, DB queries are extremely fast
I don't think that's an issue
Will have to look into later tonight with some querying.
But running it seems like it works.
So may just be logs spamming of retrying to copy already moved assets. But will confirm later tonight.
Honestly, there is a known issue with thumbnails not being cleaned up, especially for people. It is pretty safe to delete all files left after running the migration job (without errors)
The thing is (worst case scenario), you just re-generate all the thumbnails from scratch and you'd be fine.
I meant thousands of individual items. But it looks like it's a non issue. I will just delete them and move on and regenerate the thumbs if something is broken. I assume if a single image is broken, I can just re run the thumb generation on that single item anyway.
Unfortunately no, because running all missing only re-runs thumbnail gen for assets/people without a thumbnail path set in the DB. So if you were to delete a thumb that is still referenced in the DB, it wouldn't catch it I think...
Gotcha. Well, after spot checking a bunch I am able to find the image I see orphaned in my library. So I think it's safe to call them orphaned and delete. Appreciate the help!
you can regenreate images for a specific asset, yes. There is a job menu option in the web for that.
Yup, that's what I'm referencing. Looks like it's good to go though.
Oh sorry for the misinformation...