Image Tagging
When you search for something that is shown in the picture, the search is quite accurate, but I would like to know how I can know what classification the artificial intelligence gave to the picture
For example if there is a flower, grass, sun in the picture
How can I see what the AI calls it?
5 Replies
Hello, it’s more complicated than tagging, the immich search uses clip, you can learn more here : https://openai.com/research/clip
CLIP: Connecting text and images
We’re introducing a neural network called CLIP which efficiently learns visual concepts from natural language supervision. CLIP can be applied to any visual classification benchmark by simply providing the names of the visual categories to be recognized, similar to the “zero-shot” capabilities of GPT-2 and GPT-3.
I read what you sent, thanks for that.
I assume that still the images should have any classification in a database because there is the job of ENCODE CLIP which is intended for
Run machine learning to generate clip embeddings and if for example I changed the artificial intelligence of smart search to ViT-L-14::openai I would have to rerun ENCODE CLIP
So I guess there should be a classification for the pictures somewhere that says what's in them?
Otherwise, how is the operation of the smart search so fast?
when ML runs, it generate a list of number contains the encoded information. When you put in a search phase, it encodes the phases into a list of number, then we use vector comparison to find the closest match and display the photos with the match score closest to the search phase
I understood, that is, it is not possible to know how the ML decided to classify the data inside the image
That is, if there is a picture with a dog sitting on the grass with red clouds, it will not be possible to extract it from this artificial intelligence. I have to provide it with this data and it makes a comparison
Is there any way, even through a third party or through huggingface through Model card, that it would be possible to give him the picture and he will provide me with what the artificial intelligence sees from its side?
I know this tool
https://github.com/HughKu/Im2txt
But it's quite old and probably won't work well with ViT-L's AI
It's not something that's still in Immich right?
Thanks for your answer!
GitHub
GitHub - HughKu/Im2txt: Image captioning ready-to-go inference: sho...
Image captioning ready-to-go inference: show and tell model compatible with Tensorflow r1.9 - GitHub - HughKu/Im2txt: Image captioning ready-to-go inference: show and tell model compatible with Ten...
that is correct