Media seems broken
A lot of media doesn't appear on the portal. The files aren't broken however.
190 Replies
I uploaded 50k photos and am seeing this
Just started the generate thumbnails job again for all photos to see if it changes anything
Even opening these files has the issue
This indicate your mount at UPLOAD_LOCATION has issue
So what can help me fix it? I used the CLI to transfer the media
Post your docker-compose, .env and logs from server and microservies
Check the jobs, the same happened to me
maybe its just that the thumbnmails are not generated yet
just wait
You can check the jobs queue at the admin panel
Not just the thumbnails but the photo themselves aren't visible
Post your docker-compose, .env and logs from server and microservies and we can look into what the problem is 🙂
Alright give me a moment
Where to find the logs?
@Alex
docker logs immich-server
docker logs immich-microservices
what is the output of your system of
lsblk?Error response from daemon: No such container: immich-server ?!?!?!
Error response from daemon: No such container: immich-microservices ?!
What's happening? Should I restart the server?perform
docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ba29e1851ab9 ghcr.io/immich-app/immich-proxy:release "/docker-entrypoint.…" 4 days ago Up 4 days 0.0.0.0:2283->8080/tcp, :::2283->8080/tcp immich_proxy
a2cabfa4411b ghcr.io/immich-app/immich-machine-learning:release "tini -- python -m a…" 4 days ago Up 4 days immich_machine_learning
992e1a85a0aa ghcr.io/immich-app/immich-server:release "tini -- /bin/sh sta…" 4 days ago Up 4 days 3001/tcp immich_server
1953c9a9dc69 ghcr.io/immich-app/immich-server:release "tini -- /bin/sh sta…" 4 days ago Up 25 hours 3001/tcp immich_microservices
1b0c414d832d ghcr.io/immich-app/immich-web:release "tini -- /bin/sh ent…" 4 days ago Up 4 days 3000/tcp
try
docker logs immich_serverThat's long
How can I send it
Lots of red text
i.e. Errors
capture a screenshot
Just a part of it
what is
lsblk output?
is that /media your mount point for the storage?
Yup
can you go into
/media/ghar/PHOTOS/ and perform ls
can you go into thumb library and check if there are files in it?
so rry
library directory
There's one folder for a user
Here's what's inside
go into one of the year and drill down to see if you can see the files
Which files? The ones causing the error?
no just any file,
I just want to make sure the files make it so the location
Yup lots of them
ok that is good
Should I scp one of them?
To check?
now do this,
docker-compose down then docker-compose up
feel free to do that
to check the files are not corruptedWill do this
I'm on my phone so will do this later
ok no problem
I did compose up. Looks like the command does stop
Should I Ctrl+C and use the -d option?
don't use it so you can monitor the log
The problem still exists
Some videos load when opening them but don't have a thumbnail
But the majority of photos I uploaded 2d ago (50k) don't have the thumbnail and load when clicked
I am suspecting something wrong with the mount when you upload the files that is why the thumbnails weren't generated
if you download some photos on the top of the timeline now, then delete and reupload them, does the thumbnail get generated now?
I'll try this
Yes you're right
Some of them do show the thumbnail now
But not all 🤔
any log?
One sec
Which one? immich-server or immich-microservices?
microservice
docker logs immich-microservices
immich_microservices
underscoreOh
Also one of them that I reuploaded also has the wrong date. Perhaps coz it's still indexing them and queued? 🤔
perhaps
do you mind nuke this instance and try again?
50k photos?
I suspect this instance is FUBAR
How to do that?
While keeping photos?
try with a small set first to make sure it works
Just reupload them like you did initially
I did scp to get the media back from the server earlier once and a few media caused errors and didn't get received. I don't want that to happen again. How do you suggest me to do this again in a safer manner?
so you remove your original already?
One thing you can try before nuking the instance is to generate thumbnail for all again
it would ensure all thumbnails get generated correctly
Would that help in loading the files even after clicking them?
yes
Alright I'll do this
because it would use the higher resolution thumbnail for viewing in detail
🤞
Started it. Hopefully that should fix it
okay, let me know how it goes
Then there's some videos. That codec issue is still unsolved
that is the story for later, let's try one problem of a time
no space left on device
How the heck?
check
df -h
media looks good
so I suspect the mount isn't actually mount into the container somehow
pretty weird
I don't see anything abnormal on your system, so I am not sure what is going on here
🤦🏻♂️
What can I do to solve the issue?
I'm ready to plug in and copy the entire user folder
And recreate the instance
I think it is best to start from scratch now, and in a new instance, before making the bulk upload, try a small set of photos first to make sure that thumbnails and videos get generated correctly
Sorry man, I am unsure what is going with your setup here, everything looks correct
It's alright
Should I be sending these files while connected to the server or wireless it from another computer?
How did you upload those files initially?
Wirelessly
I mean your original library, where did it located?
uploaded from the old hard drive, where all the old photos were located
connected to another laptop
Using the cli tool
You do that again
using the CLI tool
so that you new instance is clean
FAQ | Immich
What is the difference between the cloud icon on the mobile app?
Thanks
Do you remember if you use the --import flag?
Using the cli? No
okay, just want to make sure
FYI, I uploaded the files using WSL
Mounting the hard drive
What is your server spec?
the one that runs Immich
An old dell i3
One sec for exact specs
Intel i3-3227U (4) @ 1.900GHz
thank you
No worries. Did that help with something?
no, I am just wondering if the low spec can choke the system and cause issue like what we see, but I remember we have many people running RPi4 so your server shouldn't have issue
Also one more thing. When I uploaded via the CLI, I encountered an issue I don't remember. I redid the upload to see 1528 media more. Redoing the upload was again asking me to upload those 1528 files. What could've been the issue there?
thost 1528 are duplicated files
So reuploading them by mistake wouldn't be an issue?
correct, the server reject them
that is why it keeps asking to be uploaded 😛
since there is no persistent memory on the CLI client
But it should just NOT show them right? While indexing and comparing, it already checks for the transferred files right?
the CLI client is stateless so it doesn't know which file is duplicated, it only knows which file has been uploaded since it compare the ID with the server
Is it okay to reupload using WSL?
That shouldn't be causing the issue right?
I believe so
After purging and remove all folders in /media/ghar/PHOTOS/ should I just create another instance using the existing .env and docker-compose.yml?
yes that should be fine
Do you have any suggestions for which video encoding to use?
Or I'll do that after uploading a few files
I use the default one, you can tweak it to your liking
Hmm... Cause I was thinking whether h264 < vp9
I have a Windows, Mac, iphones, and one Android in the home
I'll leave the default for now
you might be right, I am not very well-versed with ffmpeg 😄
Alright I changed that to vp9
I'll upload a few media now
Uploaded about 800 files
And can see their thumbnails and they open when clicked as well
ok so the mount is good this time then
Great
So now should I upload the remaining 50k too?
At once?
Yeah I don't think there should be any issue now
I'll probably export the output to a log file in case there are any issues
Thanks @Alex you were of great help. This discord channel is a blessing 🙂
no problem man, sorry for the inconvenience
It's fine. Still in beta. Over time it should get sorted out
What do you think was the issue though?
Too many uploads at once? Or a broken upgrade probably on my end?
I think the issue is the mount somehow not mounted properly in the container during your initial upload
Hopefully shouldn't cause problems this time
A lot of the photos I sent have lost the metadata of dates
For some reason they're showing up as 12th aug '23
About 300 of them
Even the exiftool shows 12th aug
But it was fine on the web app earlier
if exiftool shows 12th then there is issue I believe
Yeah but how was Immich putting them in the right places earlier?
do you use the same original files to upload this time?
I uploaded those files from a phone probably. And this time I did it using the cli. But all I did was download the images from the server and now reupload them
Are the jobs finished running yet?
if not let's wait until everything settle and we can come back to this
Sure. They're still uploading
I'm encountering the issue again 😦
A lot of files don't have their thumbnails
hmm
Are all the job finished?
can you go to Administration > Jobs and take a screenshot of the page?
The upload is still going on
But the issue is the same as mentioned above in immich_microservices
This?
Yes a lot of this
@jrasm91 Do you have any recommendation for the issue that we are encountering here. His system after a while running into the issue with error message above, but from the report, the mount drive still have plenty of space
can you attach the shell into microservices container and perform
df -hattach the shell?
docker exec -it immich_microservices shHere is what we want to see
yeah plenty of space left
can you capture the screenshot of microservices logs again?
It's too long to be captured in one screen
Yo this is a long thread
Long problems require long solutions 😭
Just a snippet. It's still printing the output
basically no thumbnail get generated due to "no space left on device" error
Sounds like you don't have any space on the device /s
Despite having half the disk space vacant
Can you access this file system through the host file explorer?
What type of file system is running on that mount? Can I see the fstab entry for it? Can you restart the machine and then try uploading a single (new) file and see if it generates a thumbnail for it?
No space left on device can sometimes mean the device is busy and unavailable. I'm guessing it is related to the filesystem and not being able to support opening new file descriptors for writing because it's out of handles/at some limit.
I/O bottleneck?
Not specifically i/o bottleneck, but the underlying filesystem implementation might be running into some issues.
Downloading a bunch of files and then uploading them again somehow works
/dev/sda1 466G 171G 296G 37% /media
What's /dev/sda1?
it's a partition (/dev/sda is the hard drive itself)
what's the output of
df -i?
Here 👆
inodes look fine
actually isn’t it kinda weird that sda1 has 0 inodes?
What's inodes?
The issue still persists and I have no idea what to do
Probably do one folder at a time instead of all 50k?
What format is /media? Is it ext4, btrfs, zfs, NTFS, etc.?
Which command to identify that?
they’re special files that store metadata about files. if there are no free inodes left, it will give the same error as if the disk is full because it can’t write to any inode
lsblk -f
That's probably the reason for the errors?
It's FAT32
fat32 actually doesn’t use inodes so that’s normal then
how many images did you upload?
fat32 has a cap of 65,534 files in a single directory
since we store all thumbnails in the same folder and generate multiple thumbnails for the same image, you’re probably hitting that limit
the actual originals wouldn’t be affected because they’re stored hierarchically by date
But so many thumbnails are missing. Even after opening it doesn't show up
the large image in the image viewer is also generated and stored in the thumbnail folder
I get it now
I'll let you know the number of files
13107 is the number of files
That's not much 😦
In the thumbs folderI'm no expert, but I know fat32 has a few limitations and I'm guessing you are running into one.
It shows total usage as 49k. That should be originals. You should have 2x that in the UPLOAD_LOCATION/thumbs directory.
Hi there, I have the same problem, following Alex suggestions for debuging I found that immich want to download something named buffalo or somthing like that. My PI-Hole DNS was rejecting that dns so file was not able to download. (sorry for my english) 🙂
that is the model for facial recognition, you will need to allow that dns so that facial recognition would work
but at same time my thumbs start to generate...
the dns blocking the doamin for facial recognition model doesn't related to thumbnail generation process
i got 100 % the same problem as universalhecker, didn't do any changes except bypassing pihole
alex, are you saying that facial recongnition model is not done on local device?
yeah facial recognition has always been done on the server
sorry, i mean on local server where immich is running or remote server?
local server
but it needs to download the model from a remote repository in order to have the model to perform the inferencing
But the problem is that if I download and reupload the files then it works fine
The thumbs directory has 13107 photos as mentioned above
Do most of your photos not have thumbnails then?
Yeah 😭
Can you try on a different filesystem?
13107 is pretty much exactly 65534 / 5. 65534 is the max number of files, assuming short file names for FAT32. We use uuids, so longer names means less than space in the directory. Tldr you need to switch file systems or wait until we implement something like this
https://github.com/immich-app/immich/issues/3314
GitHub
[BUG] Thumbs dir size becomes too big · Issue #3314 · immich-app/im...
The bug Since all thumbnails are stored in a single directory (./uploads/thumbs/$user_uuid), this directory accrues many files. I uploaded my whole collection of photo's and albums, and now thi...
How can I do that? It's a mounted drive so I'm okay doing that but can I change it with the system switched on?
I'll format it 👍
Which file system to use?
ext4 should be good
Does it work on Windows and MacOS?
Ahh it doesn't
Which is the most suitable one that can be accessed on each OS?
Is exFAT fine?
Oh it's not supported on Linux. Because I'll do regular backups of the library folder in another hard drive that I want to be able to access on a Windows or Mac system
I think ntfs is supported on linux, mac and windows. Probably your best cross-platform option
That's read-only on mac
But all I want to do is copy the files and open them in another system. The files don't change themselves so it's alright to use ext4
I've reset my system because of an issue in my linux installation
And have also installed immich along with formatting the drive with ext4 format
will get back after uploading media
Alright there's some real issue
I formatted the hard drive to ext4
And the problem persists
I did a folder of a photos of 3241 photos and 156 videos but 4 of those videos still don't load
But all photos do load
Looks like it's these specific ones that are causing an issue. They don't load even on the pc
Btw is ext4 good for large number of files? A lot of them also have long filenames
Like the photos downloaded from WhatsApp that have a long string of characters separated by dashes
Should be fine afaik
Till now everything's working fine except a few errors
Thanks for the support
Is there a way to print only the errors from immich_microservices?
What do you mean?
There are a few red texts in the logs of immich_microservices
I only want to see them
The errors
There is a log level variable you can set to only see errors moving forward.
Is there a way to show the past errors too?
I'm not sure. You can use docker logs to get all the logs. Then probably there is a way to filter the messages.
Np thanks 👍