New Image uploads are all corrupted.
Not sure what's going on here. When I upload from my mobile app, instead of seeing images in any other client, every single one appears as a broken link. If I try to download the image, the resultant file is corrupted and won't open. Deleting them and reuploading them results in the same thing. This appears to apply to ALL my new uploads to the system.
My current server version is v1.81.1
127 Replies
Uploading from my desktop seems to work okay. So it may be limited to the android upload.
Can you check the microservices to see if it has any error?
[Nest] 8 - 10/10/2023, 1:51:51 AM ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
[Nest] 8 - 10/10/2023, 1:51:51 AM ERROR [JobService] Error: Input file contains unsupported image format
[Nest] 8 - 10/10/2023, 1:51:51 AM ERROR [JobService] Object:
Looks like whatever my phone is spitting up is suddenly unsupported>
Are you taking photos in any special raw format?
No, the photos are in jpg format.
(they're also showing as jpg inside the immich client app, where they display properly)
This is an example image which I uploaded from the phone
And the downloaded "upload" from immich
Looks like it got cut off at 64Kb?
Doesn't appear to be a problem with my proxy. Results are the same when connecting directly to the server.
hmm
uploading the file from the web works I susepct?
Can you help pulling the log from the mobile device?
...actually no, it does not appear to work if I upload images sourced from that phone
From desktop.
Very weird.
That above dog photo will not work from desktop upload on my end.
Can you help zip the image and send it here?
So this works if I upload from the web
Okay, interesting. Still not working on my end.
Hmm
I uploaded your image and that also failed.
Can you check microservices log?
Maybe my attempt at uploading an image earlier was a fluke
[Nest] 8 - 10/10/2023, 5:53:22 PM ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
[Nest] 8 - 10/10/2023, 5:53:22 PM ERROR [JobService] Error: Input file contains unsupported image format
[Nest] 8 - 10/10/2023, 5:53:22 PM ERROR [JobService] Object:
{
"id": "a246efe6-8def-4ecb-a4e6-f7cb754a1dfa",
"source": "upload"
}
Hmm
try bring down the stack, and bring it back up
then removes all the error images and reupload
Same error
ok, try this, bring down the stack, repull the images and restart it
Yep
Did that earlier, same results. Just trying again now, with this command:
Same result
does upload other things work or just this image?
This image works. Everything else I've tried right now doesn't
Can you try this?
Does not work
Hmm weird
Can you help check if the UPLOAD_LOCATION mount can be seen in microservices container?
I can see files created in the NAS that the instance is connected to, but they're all corrupted
Which file system are you using? NFS?
Yep
Ok now try this, delete all photos/videos from the instance
bring the stack down, then change the upload location to be relative to the VM that run Immich, instead of using the NAS mount
then try upload again
From in the microservices container it has the correct path
Just checked.
I'll do this a little later.
But there is no VM. All of this is bare metal. Bare metal Ubuntu server running docker, hardwire connected to a TRUNAS CORE system, mounted under CIFS to the user directory on the Ubuntu System
I have been using Immich for a few months now with the configuration with no issues, but it seems this issue cropped up recently, must've been with a watchtower update.
No luck. I changed UPLOAD_LOCATION to a spot on the local drive, still can't upload properly.
still unsupported image format msg?
Hmmm. I did the build force command again, same from earlier, and now it's uploading to the local
From phone and computer.
I'm going to try repeating the proceedure and swapping it back to the network drive.
Back to broken images.
then probably network drive mount issue
how do you mount the drive?
sudo mount //{$IP}/Storage/ /home/$LOCALUSER/$REMOTEDIR -t cifs -o iocharset=utf8,credentials=$CREDPATH, file_mode=0777,dir_mode=0777,uid=1000,gid=1000,cache=strict
and it has been working before, correct?
cant you mount as ext4?
That's the command that's been in place for the last few years.
Works on all my other services. For some reason, Immich is the stickler.
hmm very strange here
Immich is definitely creating files and directories over there, but for some reason, everything gets corrupted.
@Contributor any idea about this issue?
All of the corrupted files, regardless of their original size, are 24Kb or less
Could you upload one of the corrupt files?
I wonder if it's something weird with https://github.com/andrewrk/node-mv/blob/645d8f4c783abf84204be704098fdd41f36ab195/index.js#L58
You said you also tried it on a local filesystem? That would imply this isn't involved
Could it be a docker thing?
Ooooh... That's interesting.
I can see the files in immich, but I can't download them.
Immich doesn't ever really touch the original file, other than to move it around. So it would have to be somewhere else
Images that were uploaded prior to this issue display fine in immich, but are corrupted when I try to download them from the web ui
😬
That's real neat
how about the actual file
are they ok in the
library directory?Yeah
Everything prior to and including October 3rd is fine.
As in, the files in the library directory are good and uncorrupted, and they display at full resolution in immich (and play, if they're videos) but attempting to download them results in a corrupted file.
would it be possible to mount a sub directory in your nas as ext4 and try?
How would I go about that? are you meaning a different protocol? Ext4 is just the filesystem...
@64B1T You could exec into an immich container and inspect the filesystem from there. I think that would give a clearer picture
Microservices a good place to look?
I think immich-server is best
It can see the drive just fine
Do the files look okay?
can you create a file and write something to it?
That too
Yes and yes
Wrote an 80Kb text file through the container to the drive and verified on another host.
You said that exisiting files would download corrupt?
Yep
Could you try to read one of those existing files from within the container?
identify -v /path/to/file should be enoughYeah that works fine
How do you access Immich? Do you have a reverse proxy somewhere?
Yes I do, however, this issue is also occuring without the proxy enabled - over a direct IP connection.
Though I haven't tried downloading images without the proxy, I'll do that now.
Thanks
Still corrupt
What is the direct IP to?
The server immich is hosted on, which has the immich proxy on it mirroring port 2283
What if you try the docker local IP to immich-server?
so if the URL is https://demo.immich.app/api/asset/download/299f3662-4823-4565-9e0b-8ab1549f8d05
you would want to use http://immich-server/asset/download/299f3662-4823-4565-9e0b-8ab1549f8d05
note
/api is droppedNot able to get to the immich server IP with any computer that has a GUI 😦
hmm.
You can use
curl -LOfs <url>Perhaps I'm doing something wrong.
1) Identify IP address of immich_server with
2) Identify the asset of a known good image by openng the image in a new tab.
3)
Not able to connect to it, though I can ping it properly
Are your port forwarding the containers?
If so, you can just
localhost:3001
It may be worth using curl -Lfv for more infoOnly the proxy port is exposed.
2283
Ah, I think you would have to expose the immich-server container
I guess you could also just run the command inside the immich-server container
should produce the same result
Service is rebooting
Ugh.
Opened up 3001 which seems to be the port that immich-server uses. No dice though. Going to try something else.
You can try running curl in the immich-server container instead, which would not need port changes
Still no dice. Is my command wrong?
Results in a 404
Running inside immich-server
hah
why on earth
try
curl -X POST <url>Ah, it wants auth
In GUI land, that has auth:
Makes sense
Not sure how to feed it auth over cmd. Set an API key?
-H 'Authorization: abc123'
or, I guess for immich (sorry) it's probably a cookie so🙃
I appreciate all the help
Sorry this is such a weird issue.
--cookie 'immich_auth_type=password; immich_access_token=TaDGJplsMtKoF8V3Hem1Mj3vSVn0mTPFHbjUcqldoo'
You're welcome, sorry I don't have an easy solutionLmao, using a fresh API token, I'm getting an Invalid user token unauthorized.
ah for api tokens I think you need
-H 'x-api-key: <token>'Or you can just add a query param ?apiKey=key
This works... But I'm now getting an asset not found
So the symptoms are the following?
- no thumbnails
- microservice error like invalid/unsupported file during thumbnail generation
files on the filesystem are also corrupt, and file downloads for previously okay files are corrupt too
And you cannot download the original either? You get a partial file that is much smaller than the original upload?
Oh, the actual upload, moved to library/ is corrupt?
Did you update/change anything in the system recently?
- No thumbnails
- Can't upload images from phone. Computer sometimes works, but I'm not sure why. Results in partial file on drive and no thumbnail
- Can't download existing images from immich, even if they display and interact properly in the web ui. Results in a partial file.
imo it sounds like not an immich thing, and probably something somewhere else (like a proxy)
wanted to verify though
Sounds like a problem with the drive to me
- With or without proxy enabled
well, I wanted to bypass even the immich nginx proxy
Everything was working up until october 3rd, which is when 1.81 came out
May try regressing to 1.80 later
See if that works
We did move to node 20 in 1.81
GitHub
createWriteStream has problems in a CIFS mount path with node 20.x....
Version 18.18.0 and 20.x.x Platform Linux iob-node20 6.2.16-5-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-6 (2023-07-25T15:33Z) x86_64 GNU/Linux Subsystem iobroker What steps will reproduce the bug? writ...
Looks very similar
Ah, look at that.
I'm guessing that rolling back to 1.80 would fix it then
@64B1T weird suggestion but try setting
UV_USE_IO_URING=0 in your .env, restarting the stack and uploading an image
With the latest version of immichDing ding ding
damnnnnnn
Deleted all the corrupted images. Nature is healing.
Thank you!
What is this black magic
apparently the bug is specific to io_uring, so disabling it forces libuv to fallback to a normal (in this case non-buggy) thread pool
Likely this will be fixed in the next minor node version. Do we add this to the faq in the meantime?
Faq and release notes imo
Agreed
Hello everyone
I'm sorry to re open something that appared to be close if needed I can open my own question for my own case.
I'm in the same case of @64B1T and since 1.81 all my new image are conrrupt (passing by the application or the web).
I see a env config have done the trick. The sad part for me if I'm on a TrueNas ix application and I don't see a way to configure a special setting after 😦
So here are my questions :
Do you plan to fixe this issues in the futur release and in the meatime I have to look how to rollback to 1.80 ?
Or maybe does one of you know how to configure that part in TrueNas cause I may miss a way to do it ?
Thank you all in advance for all your answer and support given.
Hello, I am monitoring the nodejs version to see if we can update to the new one to resolve this issue
OK thank you I hop it will happen soon.... sadly I think I'm stuck on 1.82 version and I don't see a way to configure my env on TruNas... :/
By the way when I read again all the discussion the solution seem to appear from nowhere. Can I have a way to check if I'm really in the same case and it's not something else that is breaking all my download ?
I havent used TrueNAS, but you should be able to set env variables based on their docs https://www.truenas.com/docs/scale/scaleuireference/apps/installcustomappscreens/#container-environment-variables-settings
Install Custom App Screen
Provides information on the Install Custom App screen and configuration settings.
Thank you for the help unfortunately I'm not on a custom app but I'm on the official TrueNas Immich app.
Here a glance of what it's look like
Normaly it should be there where I got a way to put my env setting but from the official app I don't get a way to add it and if I m correct it's not something I can add after the creation of the 8 dockers. :/
It is also there when I can show I have mount my uploads library and thumbs storage to a mnt path
Those mount point are done on my TrueNas Scale server using CIFTS to my other NAS where I store my photo.
This is where based on the discussion I read here I have my problem I think 😦
To only part that remmains for me would be to have a log to check if indeed I got the same problem are not cause I can't roll back to an older version and when creating a new Immich app using local docker storage the image upload work.