Reverse geotagging and searching
Hi, I have a question about searching for place names.
Some of my pictures got wrong locations from reverse geotagging. So I corrected the city and state columns in the db manually. But the wrong city name is still listed under the places heading in the explore page. If I open an image, the location name is correct though.
I thought it was because typesense wasn’t aware of the manual metadata change, so I restarted all the containers, but the wrong place is still there. Is it possible to manually trigger a reindexing if my guess is correct? Or it’s because of other issues?
7 Replies
If you run the extract all job your manual changes are going to be overwritten btw.
Typesense has a volume you would have to delete for it to resync the data.
Thank you! By “extract all”, you mean the extract all metadata job, right? It sounds like this job also triggers “pushing” metadata to typesense, besides extracting and inserting metadata to the db. Is it going to be faster if I simply let typesense build a new index by deleting the old docker volume (assuming the metadata extraction job will send the metadata to typesense for every picture)? I’m curious because if I make small changes to the metadata on just a few pictures, running metadata extraction on all assets sounds a bit less efficient. Of course building a new typesense index every time there’s minor changes to metadata is also expensive (plus there may be other data that’s indexed by typesense). Maybe some sort of update mechanism can be considered? Anyways thanks for the quick reply.
Coming back to say deleting the typesense volume works. It was also pretty fast. And embarrassingly I also tried “extract all metadata” even though @jrasm91 clearly said that would overwrite my manual changes. But this leads me to think about how to manage metadata.
Consider this scenario: I want to make changes to the original metadata (eg to correct a wrong time stamp on picture X), I’ll have to run the extract all metadata job, but this will overwrite all my changes to the reverse geotagging in the db (for other unrelated photos). I don’t think the extract missing metadata job will work because none of the photos are missing metadata. So short of an update metadata job, I’ll have to delete photo X and reupload.
This can happen to many photos because apparently I can’t seem to remember to change my camera’s clock every time daylight saving time changes, at least for the first couple of days. Another issue with deleting and reuploading is the expensive ML tasks will have to run again. Of course I should clean up the metadata before uploading them to Immich. But I may also want to make changes to the metadata later on.
So in general, there seems to be some conflict between managing “primary” and “secondary” metadata: when changes are made to the “primary” metadata, in order for them to be picked up by Immich, all “secondary” metadata has to be recalculated as a side effect, potentially erasing manual fixes to the sometimes wrongly calculated “secondary” metadata.
I hope I’m making sense here, and would appreciate the dev’s perspectives.
We added sidecar support for this type of use case.
https://immich.app/docs/features/xmp-sidecars
XMP Sidecars | Immich
Immich can ingest XMP sidecars on file upload (via the CLI) as well as detect new sidecars that are placed in the filesystem for existing images.
Thanks for bringing that up. I manage all my metadata using XMP sidecars. I read the thread here where @hackerman and you were working out how to implement this feature. I saw you were discussing whether to store a hash of the sidecar files in the db so a sync job can pick up changes in the sidecar. But I lost track of the long thread, and wasn’t able to get a sense of what the “discover” and “sync” jobs are exactly doing. It sounds like “discover” finds new sidecars that were not in the file system before, and “sync” will just extract everything from every sidecar from scratch.
If my understanding is correct, neither discover or sync sidecar jobs would pick up my metadata changes while not destroying my manual edits in the db. A solution might be to write my city/state values in the sidecar, and Immich will just ingest them. I think this works in general for other metadata as well. For any piece of information, if Immich finds it in the sidecar, just use that. In the other direction, any metadata worth persisting (face regions, comments) can be saved in the sidecar. Of course there’s the question of which XMP tag maps to which column in the db. And I understand the devs may not want to go this route, which boils down to defining yet another “standard” of how people should manage their metadata. I should stop here, this post is already too long. 😅
Yes, basically. The idea is to avoid manually editing metadata in the database as it will just get overritten the next time you run extract all. Discover = find new xmp file and start tracking them. Sync = re-import all xmp files. Further optimization might happen in the future to be more selective about what we re-import.
City, state, and country are a bit different in that they come from GPS tags and then are calculated via another process. We don't have a way to specify city, state, country directly from exit at the moment.
Yeah, I don’t want to manually change the db either, would rather have all info in the XMP as the single source of truth.
Sometimes the GeoNames db is wrong. For now I can manage with running a SQL script and deleting the typesense volume once in a while.
I looked at the exiftool xmp tag list, there seems to be a tag for location called LocationCreated in the iptcExt namespace, which is a recommended namespace (by its author). I never used it and don’t know if it’s a common one.
I have no idea either.