EntityNotFoundError
I originally successfully imported all my files using the CLI tool. Many of the more recent photos were still on my iPhone. The app says there are 389 assets, 117 Backup, 272 Remaining. Everytime I go into the app, it appears to go through my entire Photo library and uploading them. When it finally gets to the first photo taken on the device, it just sits on 100% infinitely.
In addition to that, watching my immich_server logs, each file appears to be outputting something like the following:
49 Replies
Just realized my app wasn't updated, so gonna see if that makes a difference. Just updated it to 1.47.0. Server is already on 1.47.2.
Nope, same errors. The files all appear in the online photo library so it's successfully backing up new items. Maybe it's acting like it's checking if the files are duplicates each and everytime maybe?
About to screen record what the app is doing that's coinciding with those types of logs
While this is recording, I also wanted to add that it always remains at 272 Remaining. If it's a new photo, the total increases and the number backed up increases, the number remaining never decreases.
I think there are some pictures that are failing to back up. Or, are already backed up and the deduplication flow is running into an error. So picture might be backed-up, but that information never makes it back to the phone because of an error in the server.
I'd guess it is related to live photos. Somehow the phone is trying to back up a live photo, the motion part has already been uploaded, but we try to return the asset id for the image picture, which somehow doesn't exist.
That's probably it. They're all HEIC, so very likely related to live photos.
Redundant, but just so you can see at your leisure: https://photos.lahni.com/share/f0ed6323e6fd25fac1654335054281885fdb87127ebace1d0e31272eec66a0a51517dbd6116f777b11ec63ad1ca2bb3c2d4d
Immich
3 shared photos & videos.
Don't expect you guys to watch that entirely, you'll get the gist watching the first 15s and the last 30s, but the 9min video is just uncut from what I see from start to finish every time I load the app
And then the second video is if I cancel and try to start and it just toggles back and forth but doesn't actually re-initiate a connection to the server. I have to close the app/swipe it away.
Also it's the same behavior with or without foreground backup on. I just have it disabled here so you can see how it starts and stops.
I used Immich CLI to import after live photos import was added. Used the docker container but I didn't record exactly what version it used but it was just a few weeks ago.
Yeah, unfortunately, even though live photos support is available in immich, they're only properly imported using the iphone app. Otherwise they're added as two separate assets, an image, and a video (motion portion).
We have a long list of things to work on, better importing support for live photos is one of them. Goal being basically being import from anywhere and auto-link them in immich.
Ah there lies the issue then on how to recreate the issue. Sounds like I should blow my install away and start with phone backups first and let CLI fill in the gaps after the fact?
That would more than likely solve the issue. Your situation is really strange though because the assets being unlinked is different from missing one part of the live photo (picture/motion) combination.
Your issue is specifically - the motion picture has already been backed up and triggers the duplicate flow BUT we can't find the image portion via checksum.
So that leads to entity not found, which throws an error and the phone is just like "idk what happened, ill try to back that one up again later"
Any idea how/why the sha1 for the jpg portion of the live photo would be different on your phone vs the one uploaded via the cli?
Maybe some specific context helps. All the photos were uploaded to Google Photos initially (originals, not compressed). Used Takeout and moved them into Nextcloud. I then continued to backup our devices to Nextcloud. I then used Immich CLI to import some 40k assets from that Nextcloud share. Waited for that to finish, then connected the phone app to prevent competing backups.
Ah, so who knows - anywhere along that path the file might have been touched.
Yup, hard to say.
If you did that whole thing again and started with your phone you wouldn't have backup issues with your phone, but you may end up with "duplicates" a live photo (linked) and probably a slightly different jpg (image portion) of your live photos.
Either that, or maybe the original backup didn't have the jpegs in the first place it is hard to know exactly what went wrong. When you browse immich do you see the videos/photos for the live photos? Or just the motion portions?
Just looking at a few examples, each has a 2s video immediately adjacent to the jpg
So the jpeg and motion parts are both there?
they are but appear to be separated as 2 different files
Yeah, that's to be expected when uploading from the CLI. We're hoping it to make it possible to link them after the fact.
https://github.com/immich-app/immich/discussions/1623
Ok, so nothing new I'm bringing to the table it sounds like. It's TBD.
Two things:
- 1. We have an open feature request to auto-link them when they're uploaded from the CLI
- 2. It's a bug that, even though the motion picture is backed up, we aren't handling the case of returning the right id for the "duplicate"
Perfect way to distill that all down. That sounds accurate to me. It's not preventing the app from accurately backing up new things. Maybe I'll just keep things as is right now so I have an environment I can report back on when that's eventually looked at. I read every set of release notes and will keep an eye on that. No rush but at least I know it's on the radar.
Perfect, yeah that sounds like a good plan. The only question I have is if the jpeg portion on the phone is different from the jpeg portion in immich what should happen...
Do you want to keep the phone's version?
That's probably a fair assumption for me but I could come up with examples for others where they wouldn't want that default. Nextcloud backup allows you to set defaults (to defer to server or client) or even prompt (then remember that answer for that asset)
Perhaps that's an option when a decision has to be made by the end user?
Could do that, just sounds like a lot of work lol.
Yea, have it solve world hunger too while you're at it ;p
booooo
This is very much specific to live photos where there are two portions and a specific case where there is a conflict on one part and not on the other. I'll ask the other devs how we want to handle it. Might just kick it back to the user to manually fix lol
We could make it easier to know which file has an error, and then you can manually remove it in immich and then the phone backup would probably successfully upload that one.
honestly, I have the original photo backed up. and the video. merging them is just a quality of life improvement. as long as it returned the jpg as backed up, i think that would satisfy the issue
True, but there could be a case somehow motion is backedup and image isn't lol.
the last idea you just proposed to remove them so they phone can back them up would be a good idea too
sounds like the safest way is to let the user know where the conflicts are, remove them online and let the phone back those up
Yeah. Honestly, I really dislike that they're uploaded/linked only from the iphone. IMO the iphone should upload two separate assets and dedupe them separately. This problem would basically go away in that case. You may just end up with "duplicates" in immich like you can now with any other set of pictures, which you could just go delete yourself.
I'll work with the team to track/fix this and then get back to you.
That's reasonable too. Live photos are neat, but like most, I imagine we only really care about the jpeg. If I truly wanted video, I would have taken an actual video lol. But err'ing on the side of duplicates is fine by me. I don't have a preference as long as the app properly knows they already exist, whether merged or not.
Or lets me know I need to handle them individually
Part 1 is done - Live photos should be automatically merged in the server now, if you run a extract metadata job (for all assets) it should auto-link them.
Part 2 is pending - mobile devs need to send still/motion as two separate requests.
I was able to confirm Part 1 worked for me before my instance fubar'd. Unfortunately I'm in the process of completely rebuilding my instance now. I re-ran the CLI while on 1.48.0 (before 1.48.1 dropped) and it re-imported all 43k images.
I ended up with .jpg and .heic versions that appeared as duplicates.
NBD as it's easy for me to start clean. I'm still going to import everything from CLI again and get back to the same state I was before.
Not sure I follow - you're re-building now, done a new CLI import, but are seeing some duplicates?
It's already nuked but yes, CLI reimported everything so I doubled the size of my instance. Instance went from 900GB to 1.8TB. The few photos I went through had duplicates that weren't visible before.
Before I send you down a rabbit hole, let me see if I can even reproduce this. I've already removed everything but I can see if I can recreate it.
I'm guessing you have two version of some (all?) of your files meaning sha1 hash is different from phone vs your other copy, which I think was like a 5 step process lol.
Who the hell knows. I've tinkered so much at this point, that's why I figured it's best to see if I end up back in the same state so I can reproduce steps easier for you, so you're not chasing an unnecessary ghost.
I'm still going to CLI import so I can test Part 2 once the mobile devs add in their part of the fix.
But Part 1 definitely merged all movie/still parts
You verified part 1 after the recent release?
Correct, I had 1.48.0 on my instance within about 45mins of it releasing. Saw the movie/stills were merged and confirmed part 2 must still be pending.
Did you re-run the exif job?
yes, that's what ultimately merged them
Cool, that's great to hear.
The original bug from this post still exists and can be reproduced when you try to backup a live photo asset in the mobile app, but the motion part already exists on the server and the still part is missing. That'll be "solved" by Part 2, but your "root" issue is probably that you have different "version" of some of your files, which are causing duplicates, etc.
I then ran the CLI because I needed to pull from my Nextcloud instance where both my wife and I both backup to. Only my stuff is going into Immich from my phone, not hers. So I wanted to pull her latest photos into Immich and ended up with duplicates.
You might want to manually check a few files from your phone and from nextcloud and verify the sha1 hash is the same. That's the only de-dupe method we use right now so if that doesn't match you'll get duplicates.
Yea, that was the first thing I was gonna do if I could reproduce it. Honestly, I think some of stems from having issues running the docker version of the cli. I have all my files on a NFS share and docker cli would just randomly stop with no errors.
Switched to the npm version and no issues
Well, other than the duplicates
Interesting
But like I said, before I have you guys chase any ghosts, let me reimport and see if I can even reproduce the same result
Yeah, I probably won't look at anything else, just follow up on mobile dev + part 2.
Yup, don't bother unless I can give you better details. I'll create a different thread to track that issue if it pops up again.
I'm glad to hear live photo linking worked though, I think it's a lot more robust than relying on the assets being uploaded from an iphone.
Yup, 100% agree. I'm sure a bunch of others are like me and using Nextcloud to backup their photos, so it's very likely they'll be importing from other places than the iphone it originated from.