Issue with large videos?
So i uploaded 2 files, 1 around 1gig, and the other around 40gigs. Both of them uploaded, but only one of them shows in the library. If i try to reupload the 40gig one, the CLI says that the asset has been backed up. Any idea why?
96 Replies
I think we went over this at some points
the video file is probably too large for the hash calculation
So there’s no way to upload it? I need to split it?
I believe so
Ok
Any idea what would be the max file size?
Or can I increase it in the env file?
not sure what it would be
do you use your internal ip to upload?
the default is 50GB maximum for the internal proxy
Yes, I am using my internal up
Ip
192.168.0.6:port
So…?
🙂
Try this ?
So the videos get uploaded but never shows up in the timeline?
I am not sure. Haven’t validated this scenario
Based on the symptoms, I'd guess that the 504 is from the sha1 hash taking longer than the proxy timeout to compute, but it did upload. Combined with the cli getting assets by device id and saying it doesn't need to upload it again indicates that it did indeed successfully uploaded. The reason for it now showing in the timeline could be a problem with ffmpeg generating the "resize path", right?
So it's filtered out in the select assets due to some condition, probably related to transcoding or thumbnail stuff?
Yes, I think this is very well the issue. Because the hashing calculation has to be done first before return the status code
I think we encountered this issue before
We should make the proxy timeout longer.
Or, do hashing on upload, not read the whole file from disk again after 😛
how can i make the proxy timeout longer?
We'd have to add them on our side in the product container. It's not the root cause for you, just gives you a false error message even though the asset uploads successfully.
I think your issue is related to the video not being transcoded. Can you try running the thumbnail job and maybe the extract exif job and then see if there any errors in the microservices container logs?
whats wierd is that even in server status page
videos = 1
storage = 16gigs so far
when running the jobs o only get no missing exif/thumbnail found
No errors in the microservices logs after running the jobs?
nope
i'm trying to reupload the file now
using the immich webui
got a few minutes left
nope, nothing
ran the jobs
no errors
after upload, immich said that file was uploaded succesfully
refresh to view file
If you do that, the network tab should show you the asset id in the network response which might be helpful for troubleshooting.
what do u mena by network tab?
In the dev console there is another tab called network that shows the http requests. There will be one for each upload and the response would have the asset id in it.
Basically we should verify the asset info / date and make sure it got processed correctly, which might be easier to do with the id of it.
hmm, and what should i be looking for in the network tab?
should i try uploading the file again
considering the network tab is recoding now?
i just noticed
before i uploaded, my storage was 183gb of 5.5tb
now its 195gb out of 5.5tb
but the file i am uploading is 41kb
gb*
You didn't re-upload in this recording did you?
Can you connect to the database? We could run a few queries to try to find the video that way?
if you could point me in the right direction as how to connect to the db, i would be more then happy to do that 😦
i have adminer installed
if u could point me on the server/user/pass/database
what to write there
docker exec -it immich_postgres bash
this command will get you too the database container
psql -U postgres -d immich
That command will get you to the database name immich under user postgres (default setting in .env)when doing the docker exec
then you can start using the queries that Jason supplied above
ok
doing now the queries
4abf25b3-afdd-4208-b307-95c94184647b | AYAN - Ziua Botezului | VIDEO | 1773125532
only 1 video showsIs that the 40gb one? Looks like it was uploaded from the web.
nope
Can you run the second one? That should list assets by size.
its a 5 minute one
i've ran it like 3-4 times allready
and the only thing it does
is add like 3-4gb to my server
of data
so fater a 40gb file upload is finished
the difference between before and after upload
this
i started today with the tries at 183gb
after a few tries, it just went to 200.2 gb
i haven't uploaded anything else
hmm
but this
hasn't changed
still 16gigs showing
Is this the video you uploaded via the web?
Did you ever run this sql?
It should list assets sorted by size.
If the large videos aren't in this list, then they haven't been successfully uploaded. I wonder if it would be worthwhile to just do some testing myself with arbitrarily large video files from the CLI and see if I can get it to work or not.
Nope
The video in question is not in the list
What’s wierd is imo
The file isn’t successfully uploaded, but some part of it of like 4gigs does get uploaded
…
Weird. I wonder if the connection gets broken play way through
Do you have some random big ish partially uploaded video files in the upload directory?
It would under your user id originals then device id i think.
And it should be a random uuid like filename
We should maybe add some utility that can report files in the upload folder that aren't in the database
seems so yes
can i delete these directly?
Yup, none of those are in use by immich
So it must be aborting during the upload.
but why? i checked my router
i'm not dropping my internet connection
I have no idea lol
Were there any server logs?
where would those be?
Docker logs command
any idea why i can't delete these?
when i try to delete them, they just change their name
just the ending of the name
Is that a trash thing?
no idea
fuse_hiddenXXXX files. It means a file was deleted but there is at least one software which is still using it, so it can't be removed permanently. It will be done automatically when the relevant software stops using the file or exists. Such files are always gone after umount/reboot
probably immich?
Probably
Super weird.
trying to compose down immich
they are gone now
right after immich closed
So it had an open file handle but the upload was aborted? Sounds like a bug
yeah, seems so
but in the web interface
it tells me that the file gets uploaded succesfully
all the 40gigs
it doesn't stop midway or anything
But you don't have a 40gb file anywhere do you?
on the server? no
nowhere
How long did it take to upload?
didn't keep tabs tbs
but i'm doing it again
and it did 10% of 40gb in around <30 seconds
Would you mind trying again with the network tab open and see if you see the /api/asset/upload request?
i'm uploiading with 50Mb
Ok
ffs
when i hit record
the upload stopped
😄
Lol
You don't have to record, just find the upload request
And then send a screenshot of it when it finishes and copy the response.
i don't see any /api/asset/upload
Did you start an upload after opening the tab?
yes
The third one is upload
And it is pending
Click on it to see details
Err 4th one. I can't count
this one?
Yup
Request URL: http://192.168.0.6:2283/api/asset/upload
Referrer Policy: strict-origin-when-cross-origin
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Content-Length: 44363641728
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryoulPdhTfXdZkSVVN
Cookie: adminer_permanent=; immich_auth_type=password; dockerFolder_view_mode=advanced; my_apps=startedOnly; SonarrAuth=CfDJ8L3QXZ_GbXFBkbvnc1Y7npYCca746LLICL9NpcyssVKeuMUuysli5HTlr0XOm-aPk8XT57jKL_Hl5-Xha5Sz0QgqwgjRP4kVHS8IsGPnhSpMUwfku_XSanoykAdm__kpoIeMPv_29pmu0PD2wYU69LjRhXJow-lBZBEN9ZVuxT1Zic0eAf7H7ai1jsPQ9ru9r2eYJ3wIl-FHjimskjhSIA0yY-g7eTx5l9Zv2yb7Y-ENVjDKLLhQ0zyzCXxfd_Tbkja0czucGWWvq-xWxsbiCei8JciNSsrS9XtLMtaXVoU3CjO1bsm2teaS5Y1C7amV9GJY_T-ITRRoBWyzbrzUG89GhjCrm3L3V5JtjdDAPyjEodBVFx1ZWg4GkfVLZ05s5h-16yKxal7HcfZ5g6hvKwD4nGElSIMvW24GX2W2RM0g9dY3ZdTKnjCvgxp3euLVuA; connect.sid=s%3A90Lruye3iKuP7hjU7YOe3rOKEVWTp_Jw.sIkKDc5Z7p%2F4BB01ceP3PSuDLK46oifNWgTLsh2hrzs; nerdtools_checkall=no; nerdtools_packages_uninstall=--uninstall; docker_listview_mode=basic; dockersettings_view_mode=advanced; csrftoken=93S7uF9XAzdufXUViRWLes51IDu9fdjaZmgUb40v9DT5UuJgbptZt28cLUOYltZI; immich_access_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiIzY2IzOTc5ZS05N2ZkLTQ5NDAtYjBkOC04MDI1MTZmMGYzZjQiLCJlbWFpbCI6InBhaG9udHVzZXJnaXVAZ21haWwuY29tIiwiaWF0IjoxNjcxNTc4MTkxLCJleHAiOjE2NzIxODI5OTF9.iwGEtz7TYM0odtU0uTaJi9-WGi3N4pcWoIWv8DIauV4; unraid_f23c168fe4b03627754c52f632c547e2=96cebc3ae6970c75d86b263afb0d9e81; ud_reload=UD_reload; adminer_sid=4kd9jddkcrm1rn2u4q0qr85slm; adminer_key=f165284f846491e738e71c47f2a95f73; adminer_version=4.8.1; col=2; dir=0
Host: 192.168.0.6:2283
Origin: http://192.168.0.6:2283
Referer: http://192.168.0.6:2283/photos
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36When it finishes you should see something under "response"
ok
will check
@40% now
Do you know how to get the docker logs for the server container?
sadly no
How is your running immich?
composed
followed to guide on ur wiki
upload finished
still file is not on immich, but what should i check now
Did you get a response on the network request?
nope
nothing
Is it still pending? Can you send another screenshot?
i closed it, sadly
tbh, i think i'm just gonna save my bigger files somehwere else or maybe just split it in 2
Sounds good. I will test some bigger files next week and get back to you if i find anything.
sounds good
❤️
ty for all the help 🙂
Sorry we didn't figure anything out! Lol.
no worries man 😄