Unable to access web UI proxy errors in startup logs. [error] 41#41: *1 connect() failed (111
I know I have had a lot of support questions lately, but I am still very new to immich and I have just CLI imported a library of 160k files. i apologise if my questions are too frequent.
I was getting issues with the ML services failing but I managed to get into the UI enough to get to the settings page and turn off the ML features until I sort out the cliploader amazon thing.
BUT now, even with ML services off, I cant use the web UI and it appears to be related to the proxy server. I have docker compose down and back up and also restarted docker engine.
Attached is the full set of logs on startup and you can see the proxy errors at the bottom.
117 Replies
for clip loading amazon thing, please refer to this comment https://github.com/immich-app/immich/issues/4117#issuecomment-1772790612
try
docker-compose down then docker-compose up again, it should workI saw this but as I am running Docker on windows, I don't believe I can get inside the vm volume to put the files in /var/lib/docker/volumes/ ... or can I?
yes you can
hmm
actually
you might need to do some research 🙂
Way cool - have made some discoveries
Ok - thanks Alex - that was a success things are running much more smoothly now thank you
My web UI is now getting stalled loading a websocket call. the page loads but wont scroll. I believe I have caught the culprit in the attached screen shot. This also appears in the proxy logs.
`
immich_server has no complaints in its log:
Should this not say ?
No, that is fine, I believe it is the denote of the localhost
I have a feeling the proxy container has issue
docker on Windows always behave like that, with lots of issues
:/
hmm this only started today though..
i have made no system changes today prior to changing the ML files as per your advice (which solved that issue)
yeah I am not sure why it is running fine and then having issue
is there anything still running in the background to process upload?
i.e is the upload folder empty?
or are there any active jobs?
is there any active jobs?
Nope. all clear
go back to the timeline
is the issue still persist?
yep
it works fine until i scroll a bit
then when it freezes I see that socket connection stalled in teh console
Do you see any red error in the console tab?
nope
if you refresh the page and don't scroll
wait a bit then scroll does it work
just want to make sure that not some issue with rendering the assets causing the issue but websocket connection
how long should i wait before attempting to scroll
like a minute or 2
ok
ok yes i can scroll now 😕
still can see that stalled connection in the network console but its working like normal
and does it freeze after a while again?
so what can you conclude from that?
let me try freeze it
no i can not freeze it
strange
should i reload the photos page and then try to scroll more or less right away again?
ok so it gets weird... I opened /admin/server-status in a separate tab to see the size of my library.. and when i went back to the tabe i had open with the timeline view - its totally frozen now
What is the server CPU's load?
nothing
pressing refresh on the photos page i am just seeing
and
in the footer
nothing in console
immich_web log
Hmm I have no clue now why it happens
refresh that log view and the end is now 2023-10-21 03:45:59 data: { statusCode: 500, message: 'Internal server error' }
2023-10-21 03:45:59 }
2023-10-21 03:45:59 }
2023-10-21 03:52:10 Listening on 0.0.0.0:3000
so 52 is now
so the web container just restarted
all the admin pages load like lightning too btw\
like they are not dead
@jrasm91 Do you have any thoughts on this issue? Hmm
could it be related to the size of the gallery?
overlaoding the DOM or soemthing
No we use lazy loading on the web
so it only query for what is on the screen
Ah
let me try another browser like Edge or something
Ok do that and just a thoughts
is there any one month that you have more than 6-7000 photos taken
Can you filter oput
time-buckets in the network tab?
hmm in a single month? possibly i had some years with 20,000 or more
i cant be sure though
years should be fine
try to filter out that and grab me the preview
yep trying to get it to load again... it locked up in edge too
same behaviour
let's get the time-buckets request to see if you have any month that have absurd amount of assets
proxy seems to have auto shut down
it auto started
this is just bizzare - all the admin pages are loading fine, and yet i cant get the photos page to render ANYTHING like its a white page with a spinning loading in the tab
Can you help me with the request above?
no, becuase i cant get the photo page to load at all
thats what i am seeing
now do this, bring down the whole stack with
docker-compose down then docker-compose pull and docker-compose upok
ok so pull pulled nothiong so I was up to date
lol I forgot the -d on the compose command so now I know what that does 🙂
log spam
ok photo page
photo page up and running
shit i forgot to open dev tools before I loaded it
so then i opend dev tools - pressed refresh and its locked up again
🤔
i cant get it to load at all now
have down down and up again no avail
I am goin gto reboot the pc
going to
back in 5
okay
preview is blank
HMm
can you try create a new Immich instance, upload a few dozen assets and see if it ever freeze
good idea
can i be sure my backup is sound?
is there a way to integrity check it?
it's a 1.8GB .sql file
I just keep this instance and not removing anything
(i didn't gzip it)
I think by setting the project name in docker-compose file, it will split it regarding the volume
oh - how do I have more than one instance?
just change the port in docker-compose for web container to something different than 2283
you have to create a new folder on your machine with a new docker-compose and .env files though
yep
ok I get you
Just to make sure, what is the output of your
docker volume ls?
the docker UI gives some more insight into those volumes
hmm so they are not splitted by project, did you use the project feature in docker-compose?
i dont believe so, but I did need to put COMPOSE_PROJECT_NAME=immich in my .env
becuase remember I am running it in the root of a hdd and so it didnt have a folder name to assume as the project name
Okay, let create a different project name for the test instnace
all those volumes are immich instances btw, i dont run anything else in docker
yes I will set COMPOSE_PROJECT_NAME=test-immich
and run from a different hdd\
oops i forgot to set a different port
should be ok though
ok clean isntall is up
should i use the CLI upload to send up some files or just use the web UI
Just try browsing /api/asset/time-bucket
have a new install up now
https://demo.immich.app/api/asset/time-buckets?size=MONTH
This is what Alex is looking for.
I would guess your disk io is maxed out.
on the new install i see that
disk i/o maxed out at what point? all resources were idle pretty much when things were dead
Ok so to confirm the new install has 578 photos, is working fast as ligtening with no issues.....
it should be noted, that even though it is most likely unrelated to the issues I was having, that this is still in the proxy log
So whre does thaat leave us? disk i/o maxed out at what point? i don't follow as disk resources were not high at all when the page was not loading I am heading to bed 5am here.. I will bring the old instance back up so we can trouble shoot more later. ok have the old troublesome instance back up successfully - can confirm disk activity is zero when loading fails
So whre does thaat leave us? disk i/o maxed out at what point? i don't follow as disk resources were not high at all when the page was not loading I am heading to bed 5am here.. I will bring the old instance back up so we can trouble shoot more later. ok have the old troublesome instance back up successfully - can confirm disk activity is zero when loading fails
If I load /api/asset/time-buckets?size=MONTH before trying to load the UI then I do get a result...
Looks like 4k assets in the first bucket
when I load the web UI, before touching it I see three time buckets in the console
all of them are empty when i click "preview"on them
one has status 304 and the other two 200
Ok thanks both ttyl need to zzz
5am oh boy lololol
Good afternoon, has anyone had any thoughts on next steps to try and resolve my issue?
@Alex any further ideas? I can say that my library is working super fast and flawlessly from the IOS app
I think you need to narrow down the issue. It is hard to troubleshoot such a general problem. Can you start with a smaller library and add more photos until it stops working?
well it was flawless with 570 photos, and my library is 160,000 so i am not sure how many increments I would need to try?
are there other users with large librarys and no such issue?
also is it definately unrelated to this error can we say?
We have other people with 200k+ and it works fine. So it could be a photos distribution problem (a lot of photos on one day), or a hardware problem (disk read speed is insufficient, although caching should mitigate this), or something else, but I'm not sure where to look. We really need some profiling to figure out if it is a rendering problem, a network problem, a disk problem, a memory problem, or something else
ok well the disk when i copy to the c:drive is reading at 450MB/S
Is your immich-server crashing?
So I dont think its disk speed. The maachine is an i9 12th gen with 32gb ram so its plenty beefy...
immic-server no it stays up
the only thing that crashes is the browser
Did that write happen while it was up and your accessed the photos page?
have tried chrome and edge and same behaviour for both.. Have also tried on other machines on my local network - same issue
Sounds like maybe a rendering problem.
that was me copying the database backup from the USB SSD to the SSD on the local machine
I meant the connection error
Chrome has some profiling built in, maybe you can run that and share it, although it will be very large
that connection error is contiinually there in the proxy log
Like repeatedly? That does seem odd
yes repeatedly, Alex was suggesting it might be a symptom of docker on windows
Hmm.
i have another i7 windows machine here, I am going to do a setup on that
same setup
and see how that behaves, to eliminate any hardware issues.
Based on what you said it doesn't sound like a hardware issue
Can you record the network tab and how long requests are taking?
i wish but nothing even appears in the network tab half the time
it's just bizzare
Your time buckets looks normal
I am not sure what we are experimenting here
It's it a problem for all web clients?
He tried On Chrome and Edge I believe
Can also try on Firefox please?
ok will try firefox
You should be able to go to the Admin page then open the network tab and start recording and then go to the main timeline. It would be good to see what is happening with the requests
ok am on a new machine now
migration done
testing...
Same thing happening on Firefox
I think the fact that it is running flawlessly on android and ios apps supports the theory that the proxy server errors are the issue am I correct in assuming that the apps are not using the immch_proxy service?
What url did you use on the mobile apps?
If it has /api then it goes through the proxy