How to mass upload to Immich but through unraid?
Hi there,
I went through the install guide for unraid which went smoothly and it got me to create a separate immich folder inside my photos share which is fine, but my question is how do I mass upload what I have in my photos share into immich since I don't think it has the auto detection features just yet.
189 Replies
Bulk Upload (Using the CLI) | Immich
You can use the CLI to upload an existing gallery to the Immich server
yea i tried that its not really working out for me
tried the docker command and it failed
finding the directory even though i thought it was ment to be a seperate temp container
and it attaches to the main immich container
Can you access the terminal where the directory is mounted somehow?
i can access the directory manually thats fine through terminal but when i put it into the command it couldnt get to it
What's the full command you used?
docker run -it --rm -v $(pwd):/import ghcr.io/immich-app/immich-cli:latest upload --email [email protected] --password password --server http://192.168.1.216:2283/api -d /import
and replaced /import with the images directory
and added in my details
which it worked
but failed at the directory part
As the docs say:
Be aware that as this runs inside a container it mounts your current directory as a volume, and for the -d flag you need to use the path inside the container.
so your saying if i created a immich folder within the images share during my immich setup it basically wont see anything
because immich is mounted inside the immich folder that has nothing in it
because thats all the cli can see
i thought it would be easier to have the "upload" path set to a immich folder inside my photos share like in the unraid guide
since im not sure exactly how the upload process works if it creates versions of my uploads and i dont really want it conflicting with my current folder setup that i have on the share
will i need to move where its sitting then?
For the immich upload folder you want an empty folder wherever you like. Then for the bulk upload, you go to the folder that has all your images and run the upload command
yea well i tried that like i said and it failed
but im asking would it have failed because the upload directory for immich is an empty folder cant see whats behind it
and the cli uses immich set directory like u said
or should it have not mattered
You said you changed the path in the command, but try instead running the original command from the docs in the folder where you have your images (or change the
-v $(pwd):/import to directly mount your folder instead)ok so i didnt change that -v $(pwd):/import
The cli has nothing to do with the upload folder that you configured for immich
il try this now see what i get
i thought the -d was the only thing u changed for directory
so like this
docker run -it --rm -v $(pwd):/mnt/user/photos ghcr.io/immich-app/immich-cli:latest upload --email [email protected] --password password --server http://192.168.10.4:2283/api -d /mnt/user/Photos
the share is photos
and in there has the photo folders layed out
and also the empty immich folder
or is it like | docker run -it --rm -v $(pwd):/mnt/user/Photos ghcr.io/immich-app/immich-cli:latest upload --email [email protected] --password password --server http://192.168.10.4:2283/api -d /Photos
The CLI runs inside the docker container, so you need to use
-d /import where /import is the path inside the container. Then you would mount your folder to /import, either by using -v $(pwd):/import (which mounts the folder you're currently in), or with -v /my/path:/importdocker run -it --rm -v $(pwd):/import ghcr.io/immich-app/immich-cli:latest upload --email [email protected] --password password --server http://192.168.10.4:2283/api -d /mnt/user/Photos
so that then?
is that what your saying since it will be running with immich and immich is already set to its own folder?
and -d is the directory of the photos to upload
Yeah the container needs access to the folder of photos you want to upload so mount that folder as a volume like @bo0tzz said.
so like this then
docker run -it --rm -v $(pwd):/mnt/user/Photos ghcr.io/immich-app/immich-cli:latest upload --email [email protected] --password password --server http://192.168.10.4:2283/api -d /Photos
The -d needs to match the right side of the volume mount.
That looks better. Does it work?
im gonna try it now
You have /mnt/user/Photos doesn't match -d /Photos
The right side of the volume mount needs to match the -d flag.
ok so how i had it before
/mnt/user/Photos for both
does this then upload them into the empty immich folder i created?
technically?
and will it keep the meta data or scan them in like they have been taken today?
Date comes from exif data in the file.
ok cool
Yes, it uses http to "upload" them to immich server, and they'll end up in the UPLOAD_LOCATION
ok cool thanks il run this now
Gl
ok well i got back that it found 0 assets
the directory path set looks fine like i can go to it in terminal and takes me to that spot
What path?
so i must have not done the command right or not made the right adjustment
/mnt/user/Photos
is the path
That on unRAID?
yes
What directory are you running the command in?
just webui terminal general terminal
i dont think its in a directory
its at ~
What does pwd print out?
./root
That's the folder is scanning images for
Left side of volume mount is host, right side is container.
right so now i need to make that adjustment to the command
Or, just cd to that dir and run it again as is
ok yea i could do that as well
good point
To make it less confusing I'd change the right side to be /import and -d /import
whilst in that directory?
i just cd into it
so should i change the command back to /import
instead of my path
Regardless, just so you don't keep confusing /mnt/user, which looks like a host path
Yeah, in the photos dir use a volume mount of pwd to /import and then use -d import and save that command for use in the future.
ok il try that now see how i go
How many images do you have?
ah i would say probably 3000+ i think
oh god no its more like 100k
i says 115k assets
Oh nice
is this a ram or cpu intensive task not that i dont think my machine can handle it but
but just so im aware
seems to be running ok little slow
well it seems also my docker.image seems to be going up
Probably cpu intensive
ive had to stop it
somethings making my docker image go up
It generates a sha1 for each file to prevent duplicates
What do you mean docker image go up?
like im getting notifications about my docker.img going filling
to like 80% from 70% in like a matter of 5 mins
Disk space?
i would say the amount of space i have set for docker
yea
normally it only happens to containers that are quite massive. i dont really fully understand it myself but i know it happens to massive containers
Are there images in your immich folder?
there are some yes
i only got to 115
till i kept seeing the docker size increase
fast
and stopped it
How much space did you allocate? If it includes the volume mount then it needs to be bigger then the size of your library.
how is that though with everything else i run
How big is the immich dir?
the immich folder i have set?
Yeah
as big as my array can take
like 500gb
or more
is available
I mean how full is it?
got to about 8gb
it says
but im not sure why its increasing my docker size
like when i was running the command
Can't help you there.
is there a way to know where the immich container files are being stored?
could it be writing to that place which would be on the docker.img and could explain the increase in filling
"/var/lib/postgresql/data" is where immich is stored?
or just its database?
do you know anyone using unraid and possible may be able to help
There should be a database mount for postgres
yea im just trying to find out what is making the docker.img so high like it must be something writing in there
the images folders and the import folder are outside of the docker.img
Do you have a database mount?
ah only whats there from default so what ever that is
GitHub
immich/docker-compose.yml at main · immich-app/immich
Self-hosted photo and video backup solution directly from your mobile phone. - immich/docker-compose.yml at main · immich-app/immich
i havent set it anywhere else
There should be a mount to that folder you sent
yea it uses those paths from the default dockercompose
and i checked in there and it doesnt look that big of a folder
so i dont know where this increase in size is comging from so dramatically
do you know where the container settings itself get stored?
and those config files or is it just the data base?
Immich data gets put in upload location and the database
So you should have mounts for both of those
like this
/mnt/user/appdata/immich:/var/lib/postgresql/data
is what you mean by mounts?
moving the database outside of the docker?
Yes
What is your upload directory set to?
/mnt/user/Photos/Immich
outside of the docker
Ok
like i would mount anything else
i havent changed the database though
i left it as it was by default
which is what ever is default
What does default mean?
Using pg_data?
means what ever is in the docker.compose.yaml
i never changed the settings
Ok
i just copied from github
Yup
so what would need to be changed there?
database im guessing?
Nothing. Those two volumes are all you need set.
Everything else can be trashed and you don't lose any data.
cause i keep getting MediaServer: Docker high image disk utilization
Warning [MEDIASERVER] - Docker image disk utilization of 80%
1671671881
Docker utilization of image file /mnt/user/system/docker/docker.img
What is the limit?
and it was like at 70% now at 80
in 5 mins from that command i did doing the mass upload
Is that a 5mb increase?
A 50mb increase?
The only other thing actually is a reverse geo location cache on the microservices container. I think it's like 30mb ish
the whole docker image i think has a cap of 80g or something
which is why i was concerned
It doesn't tell you where the increase is coming from?
on the notifications no, which is why im trying to find it but kinda hard to know where to start
just says what i mentioned before about it being "Docker utilization of image file /mnt/user/system/docker/docker.img"
which idk how to monitor what is increasing in there
Do you know how big the logs are for the containers?
Is the utilization still high? I wonder if it is the cli logging or something like that?
i havent had a notification about it going back down
so it could be logs being created
and havent been removed
not sure where they would be though
i did find the .photostructure folder it creates
with all the previews n stuff like that in /import
but its all with the photos outside of the docker so no use
for that
I think bo0tzz uses unRAID so maybe you can ask him tomorrow 😛
is he logged off now?
Yeah i think he's a few hours ahead and is pretty late there
so i have noticed something
since ive spoken last
"immich-immich-machine-learning-1"
is located in my docker img
and that does seem to be increasing in size
is that something that would stay increased over time?
Can you share your docker-compose so I can take a look?
yea im doing that now
on the other one i made just to seperate the issues
No problem
I can help here
thats for the compose of immich
Ok basically default setting
ran this for the upload mass cli
what are you observing?
docker run -it --rm -v $(pwd):/import ghcr.io/immich-app/immich-cli:latest upload --email email --password password --server http://192.168.10.4:2283/api -d /import
this comes up when the cli is going through the process of uploading my images
MediaServer: Docker high image disk utilization
Warning [MEDIASERVER] - Docker image disk utilization of 77%
1671679922
Docker utilization of image file /mnt/user/system/docker/docker.img
it will like increase over a few mins
I am guessing it downloaded the GeoCoding information
thus increasing the img size
what is the maximum you let the image to be?
i just dont know if its something that will keep growing or i need to reset my cap
for the docker.img
its at like 80gb
I will stop at some point
i think
or something
not that high
probably less il try and find out wont let me see unless i stop docker completely
80GB is very high for the microservices since what it does is processing images
77% is like 61GB
can you attach to the microsefice and do
df -h?
yea
my bad
that is the dev one
let me go to the production one
1.6T 446.6G 1.2T 27% /usr/src/app/upload
udev 15.6G 9.0G 6.6G 58% /dev/null
udev 15.6G 9.0G 6.6G 58% /dev/random
udev 15.6G 9.0G 6.6G 58% /dev/full
udev 15.6G 9.0G 6.6G 58% /dev/tty
udev 15.6G 9.0G 6.6G 58% /dev/zero
udev 15.6G 9.0G 6.6G 58% /dev/urandom
tmpfs 15.6G 0 15.6G 0% /proc/asound
tmpfs 15.6G 0 15.6G 0% /proc/acpi
udev 15.6G 9.0G 6.6G 58% /proc/kcore
udev 15.6G 9.0G 6.6G 58% /proc/keys
udev 15.6G 9.0G 6.6G 58% /proc/timer_list
tmpfs 15.6G 0 15.6G 0% /proc/scsi
tmpfs 15.6G 0 15.6G 0% /sys/firmware
so my docker.img must be like 50gbs then
I think the information is a bit not informative
I've never had to mess with docker.img so I am not sure
can you do some experience and see what happen?
yea il mess around and see what i come up with il try and increase it
see how far it goes
and compare
Thanks. I've never had to increase the size of the Vm
no worries
Here is the size of the running disk 18Gb and the size of the mounting disk, for storing images
docker images stay on the root disk
Maybe you can set the same amount storage as I am and it will be fine 🙂
I am not an Unraid user so my support is limited, sorry about that
no your alright i appricate all the help your giving me really do and i love the platform your making so i really want this to work so i can use it
im gonna try now and increase docker size and see how i go
so i increased it from 50gb to 100gb and ran the cli command last night and it seems to gone ok will it got half way and spat this out idk if its just because i was running it in open terminal and wasnt a background process it did this or not but you probably would understand it more then i would
Upload Progress | ███████████████░░░░░░░░░░░░░░░░░░░░░░░░░ | 37% 43459/115971 Current file [/import/.photostructure/previews/000/024/11-sq-n
Failed to upload 1 files [ ...` to show where the warning was created)
{de:1) [DEP0137] DeprecationWarning: Closing a FileHandle object on garbage collection is deprecated. Please close FileHandle objects explicitl.
file: '/import/Immich/184f0361-140a-4e4f-afc3-d3ef239269ae/2020/2020-05-06/IMG_2376-017.MOV.mov',port/.photostructure/previews/000/025/13-fitn
reason: Error: Request failed with status code 504░░░░ | 49% 56870/115971 Current file [/import/.photostructure/previews/000/039/02-fitn
at createError (/usr/src/app/node_modules/axios/lib/core/createError.js:16:15)rrent file [/import/.photostructure/previews/000/091/08-sq-n
at settle (/usr/src/app/node_modules/axios/lib/core/settle.js:17:12)5971 Current file [/import/.photostructure/previews/000/101/03-sq-n
at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/axios/lib/adapters/http.js:322:11)rt/.photostructure/previews/000/119/03-sqn
at IncomingMessage.emit (node:events:539:35)██████ | 100% 115971/115971 || Current file [/import/.photostructure/previews/000/123/99-s]
at endReadableNT (node:internal/streams/readable:1345:12)
at processTicksAndRejections (node:internal/process/task_queues:83:21) {
config: [Object],
request: [ClientRequest],
response: [Object],
isAxiosError: true,
toJSON: [Function: toJSON]
},
response: '<html>\r\n' +
'<head><title>504 Gateway Time-out</title></head>\r\n' +
'<body>\r\n' +
'<center><h1>504 Gateway Time-out</h1></center>\r\n' +
'<hr><center>nginx/1.23.2</center>\r\n' +
'</body>\r\n' +
'</html>\r\n'
}
]
and im checking did you say the machine learning looks for duplicates just wanna know if im creating multiples everytime i run the cli mass upload command
the server will check for duplicate on uploaded, not the machine learning
probably the microservices is taking up all the CPU to handle file creation, thus the server has to wait for the CPU available to keep accepting the incoming file. While waiting, the request is timeout
should i just re run the command again?
its what i was planning to do just wasnt sure if to or not
yes just run the command again, it will scan for files that have not yet upload to upload them
easy thanks il do that again
might check if there is a way to run it in the background instead of having to keep the terminal window open the only thing is i have to enter in y each time to upload which i cant do if i tell it to run in the background
you can use a cronjob to
Bulk Upload (Using the CLI) | Immich
You can use the CLI to upload an existing gallery to the Immich server
ok cool
so i justed finished running the cli mass uploader after multiple attempts and it taking forever
i have my immich running but when i load it up i cant actually see any of my images now?
is it something that maybe i have to restart immich or something?
Ueah try it
can i just compose down and up to do that or will it remove all its meta data and configuration files?
You can comepose down and up
Everything will be persist thanks to Docker’s persistent volume
ok cool
forgot to report back about this but im still having the issue with the webpage althought seems to come through to mine phone ok
Hi all, i have the same issue! I try to upload 723 photos and videos with bulk upload but the totality of photos dont upload in immich.
I use this command and this is the prompt.
and this is 1 fail, but in immich i see only 172 items
the last photo is other test with other album and the same problem, the CLI detect 265 photos and the immich upload 68
the command is this:;
immich upload --email test --password test --server http://192.168.1.35:2283/api -d C:\Users\javi_\OneDrive\Escritorio\Proteccion Civil -al Proteccion
I have unraid 6.11.5 and the last version immich and CLI/NODE
Do you any have duplication in those folders?
What is the number when you hover over this?
How did you get the count of 68? Did you leave the server sometimes to process the background job of generating thumbnails and such?
For the CLI, a total of 285 asseets will be uploaded to the server its the same of the photos in my windows
i dont have duplication
and the number es 17k of photos and 2k videos
and thanks
Does it make any sense if I can stick the files I want to upload right in the Immich directory and have it "upload" them from there?
What you are saying is the "consumer" directory. It is discussed in detailed in this post https://github.com/immich-app/immich/discussions/1006
GitHub
Gallery Watcher - Feature building · Discussion #1006 · immich-app/...
This discussion thread discusses implementing the most-wanted and missing feature of Immich #7 . Gallery watcher Below are some of my initial thoughts. There will be an additional WATCH_LOCATION in...
Hmm.. I just wanted to upload them to Immich, not just have Immich see them. Currently, my library is Google Photos and I mostly view and do minor things with them, so I was just looking for a simpler way to get them all into my Immich library.
You have to link the metadata first before uploading them to Immich
Currently the only way to add media to immich is through the "upload" process, which happens over http, so using the Immich CLI. There is no way to import files "in place" or import files via "copy/move" from another folder on the server.
If using this Gallery Watcher feature? I didn't read anything about that, don't know how to do that.
it is not built yet
The thread is there to gather feedback and thoughts
I see. With unRAID, don't know how I'd install that stuff from CLI repo. Reading through this didn't make me look forward to this process. Upload 50 at a time sounds more ideal, even though that would take awhile. 😳
I see this
023-01-04 09:31:59.328 UTC [1136] ERROR: duplicate key value violates unique constraint "UQ_unique_asset_in_album"
2023-01-04 09:31:59.328 UTC [1136] DETAIL: Key ("albumId", "assetId")=(855d9a6b-1295-4eb8-9b6b-b8d9e8567b9d, ded0f2ce-8e65-4fe5-8fa7-fb8b8edf83ed) already exists.
2023-01-04 09:31:59.328 UTC [1136] STATEMENT: INSERT INTO "asset_album"("albumId", "assetId") VALUES ($1, $2), ($3, $4), ($5, $6), ($7, $8), ($9, $10), ($11, $12), ($13, $14), ($15, $16), ($17, $18), ($19, $20), ($21, $22), ($23, $24), ($25, $26), ($27, $28) RETURNING "id"
but the photos dont upload, and i restart immich and I have the same problem
What are you trying to accomplish?
Upload massive photos.
For the CLI, a total of 285 asseets will be uploaded to the server its the same of the photos in my windows
i dont have duplication
and the number es 17k of photos and 2k videos
The same problem yesterday
You can run the cli in a docker container as well
What do you mean be this? There's a separate docker container I could install and run specifically for uploading the images?
That is correct
Bulk Upload (Using the CLI) | Immich
You can use the CLI to upload an existing gallery to the Immich server
Okay, I see. Can I run this command from the unRAID terminal?
I could run from a Linux machine if need be on the same network
You can run it from any machine. It just needs to be able to communicate with immich via http.
Okay, Will try it. Thanks
I kept it easy by putting files in the same directory I ran the upload from, worked fine.
Some of the dates were lost though and used today's dates, but less than half of what I tested. They were all the same kind of files, so odd that was the case.
I try with unraid terminal but the same problem
the photos cannnot upload
That's why I didn't bother with doing it on unRAID and just used a different machine.
I use unraid termianl and with node in windows
same problem
I used a Linux machine
A VM actually
probably the files have problem
Is it something you can share with for debugging?
I mounted the share on Linux with all my photos and ran immich-cli from there, dates all good, even easier.
how do debuggin? i dont see nothing in logs, can you explain step step?
thanks
I meant sharing the video file for debugging
I already know what the problem is. I uploaded all my photos and then I wanted to upload the albums, but the photos are repeated so they don't upload.
There is an open issue for the cli to support making albums in this scenario
I just noticed that the files uploaded with CLI are all much lower resolution, almost 50% lower. From my Android phone using backup feature they were the same, as well as upload through web GUI being the same, but the CLI is lowering the resolution.
All the processing is done on the server, so the CLI doesn't really do anything regarding the resolution or the quality of the file
So why would the server lower the resolution?
because the thumbnail doesn't need high resolution
I'm not referring to the thumbnail
I even download the images from the web GUI and look at size, res,etc, all different
As an example. File that was 2560x1440, 518.5 KB became 1440x810, 150.9 KB
Finding the file manually in the immich directory, so far 2 of the files that download at lower res, are actually full res, but not if looking at or downloading in the web GUI, even if I view in new tab
When an image get uploaded to the server it generated the jpeg and webp
when viewing an image, it loads the jpeg version
you would only get the original file when download an image
Okay. Playing with a bit more, downloads have been full res now, but they hadn't been. The jpeg versions are purposely lower res? Ones with text looked pretty blurry, but it has only been with CLI uploaded ones.
jpeg version are purposely lower res
it is there for machine learning purpose as well
since most machine learning model reads in jpeg or png
If you right click on an image in the detail viewer and download that image, you don't get the original file. Only the download button from the app would get you the original file
And again, the CLI doesn't touch the original
I am seeing that. For some, they look fine, but some do not. In one case I see, the JPEG version was upscaled to be larger since the OG was smaller.
This is how the jpeg is generated