K8S - HEML Chart Setup Issue
I'm trying to run Immich on Kubernetes cluster. First, I created my own deployments, but after struggling with some issues, I decided to use your Helm Chart. Turns out, it didn't solve one of the issues that I had.
What works:
- The web interface
- iOS app can connect to the server
- Persistant storage is mounted properly under
/usr/src/app/upload on every container besides immich-web, and has permissions set correctly
- 777 for the debugging
- I can create files from each container's shell, and their content is visible on other containers
- Immich connects to Postgres DB. Everything there looks fine; there're some tables and I can see my user in one of them.
What doesn't
- Uploading files results in…nothing?
- iOS app just pushes all of the photos without any error, but the photos never show up in the web interface, and their state in the mobile app doesn't change
- Selecting a photo in the web interface does nothing as well. Nothing changes in the persistent storage (it's still an empty directory)
In logs, I can see:
And that's the only log generated by the entire namespace upon file upload on the web interface. That log comes from immich-web.126 Replies
My Helm Chart values:
Can you grab the full logs of immich-server and immich-microservices and post them?
I'm gonna restart all of the containers and send that. Because when I interact with the web interface, those lines I sent above are the only logs generated by all of the containers
Those are logs from the namespace. After a minute after I restarted all of pods I navigated to the web interface, which is visible as that sw.js error on the top
The
sw.js is your browser trying to refresh a cached file. Did you previously run another application at the address you're using for Immich? That shouldn't cause any problems fwiw, it's just a 404
I suspect your microservices are failing to start. There's a DNS issue with loading the geocoding data that we haven't managed to fix yet. Setting DISABLE_REVERSE_GEOCODING should bypass that and let microservices start up properly (https://github.com/bo0tzz/kube/blob/main/apps/immich/immich.yaml#L61)
Oh, I hadn't realised the log is backwards. Seems like geocoding does apparently initialise correctly (which also seems weird to me lol)yeah, that's how Loki formats logs
I did actually. but I also tried to use a different browser and it still doesn't work
Did you still get the sw.js error with the other browser?
I think the biggest issue is that Immich doesn't actually log much
good question to which I don't remember the answer. let me check
Hm actually, setting LOG_LEVEL=verbose might help
Wait, that's not quite it 🤔
okay, so I removed the namespace, removed everything remaining, deployed it again and now I can upload photos through the web interface. I also reinstalled the iOS app, but those photos still don't show up
Did you also clear out the database?
I always forget that it actually uses a db, let me try that
that's interesting: the iOS app uploaded one photo -- the latest one, which is the logs screenshot above
I'm gonna wait a few minutes, because maybe it's just a bit slow to index those photos, but the iOS app reports only 1 asset backed up (even though the "Uploading file info" below seems to be uploading a few files a minute all the time; so just like before the purge I just did)
Try going into administration->jobs and rerunning those. Could be it uploaded stuff but just failed to process them
it doesn't detect anything
do all containers run as root?
could be that one container creates a file with chmod 761, and another can't read it
Could be - I believe there was a pr changing that a while ago, but I don't remember if we merged it (or reverted it)
should leave a trace in logs, though
I put
LOG_LEVEL: verbose in common_env, but it didn't change anything I thinkI've got something just now:
3001 would mean it's immich-server, right?
so, immich-web timeouted on connecting to immich-server
weird
I executed that on immich-web container
Is there a concept of docker network in your setup?
no, it's k8s
So you have to setup the network so that the containers can communicate with each other
and they can, as you see
They're using the Helm chart, same as me
It should just be working lol
Hmm
This cmd gets me the same result but my setup does work
immich-web def can connect to immich-server
maybe immich-server is just too slow?
iOS is probably spamming it with requests
Check the cpu usage of server?
It should respond even if it's busy I believe
The 404 is expected actually
wget immich-server.immich:3001/server-info -O - does work
You're just hitting an api path that doesn't existnothing much happening there
yeah, it works
all I get from immich-server is occasional
probably the iOS app
Yeah that's all normal
basically, what's currently not working is the iOS app. let me try to run it on macOS. maybe there's some pesky cache that's persistent even after uninstalling the app
On android there's buttons to clear app data etc. Does iOS have an equivalent?
no, I looked for that
okay. it's the same on macOS
If you upload a new file via the web right now, does it work?
interesting enough, I tried just now and the photo doesn't show up. but a moment ago I did upload two photos and it worked fine (I didn't change anything in the deployment in the meantime)
Even if you give it a moment and then reload the page?
yep
and tried with another photo and it worked
so weird
Very weird 🤔
And it wasn't a file you've ever uploaded before ?
oh, the one that failed had a weird filename
no, a new screenshot
Screenshot 2022-12-04 at 9.25.50 AM (2).png
maybe it failed because of
(2)?
yeah. tested with a bunch of other files and that one seems to be the only one that doesn't show up
so it's probably a separated issue not connected to iOS app not workingHuh, interesting. Should be unrelated though
Can you open an issue for that filename thing?
I gotta go have lunch, back in a bit
sure will do
Earlier you wiped your setup. Did you end up clearing the database as well, or only the library folder?
The web can't connect to the server. The actual error was failure to connect to this URL.
http://immich-server:3001/user/me
That hostname was different from what you tested with wget.
The wget hostname was
immich-server.immich, which in a kubernetes environment && in this case is equivalent
.immich is indicating the namespace to look inAre you sure it would work without it?
Yes
Ah I missed that one.
How come server.layout.ts had a connection timeout error?
AxiosError: connect ETIMEDOUT 10.43.16.65:3001
2022-12-11 12:05:16 at
No idea. Seems like that wasn't happening anymore later though
I think there's a few different things going on
Gotcha. Seems like a fresh install might be the best thing but. Who knows what state anything is in at this point.
I wiped everything
the only thing that remains the same between those setups is the DB credentials. everything else had been removed
oh, and TLS secret, but that shouldn't influence it either
I did, it didn't help
And what doesn't work now?
as I mentioned, uploads from the iOS app
Can you upload from the web?
yes
The pictures show up in the web as well?
pictures uploaded via web show up in the web interface and iOS app
Ok, that's good.
What exactly is not working from the iOS side? New pictures there do not backup to immich?
yup. I can see them being uploaded as the progress bar moves and uploads new files, but it's like the server ignores it
let me see if my phone generates any traffic on the network
Do you see any change in available space on the mount?
nothing from iOS gets uploaded to the PV
Are you certain you cleared the database? Feels to me like deduplication is kicking in
I removed all tables
(immich will refuse uploads of files it's already got the hash of in the db)
Hmmmm
oh! redis
I didn't touch redis
does it use redis for dedup?
Redis is just the queues for ingest. I can't think of a way old state in there would break things, but it probably doesn't help 😛
it would be much easier to debug it if logs showed a bit more. do you plan on adding some more info to logs?
if deduplication is using
assets table, then I don't think that's it
also, my phone seems to not be generating any traffic on the network. same with the k8s node -- no significant traffic reaching it
Then how did you login? Or are you saying just for upload? In which case it sounds like it is not uploading.
I mean the uploads
That'd probably be good yeah
Then my suspicion is the app has old state and thinks it's already uploaded the files so it skips them. I don't know enough about the app to troubleshoot that though
@Alex Tran any ideas?
Yes the app would save the duplicated asset in the local storage
reinstalling the app would clear that storage though
that's how it looks like
does the progress bar display both upload and hashing time?
hasing process is on the server
there is no hashing on the client
hm, then I don't know what's happening. something clearing takes some time, but there isn't any significant traffic from the phone or to the server
Hmm, do you have a docker instance setup that you can try?
I was running immich a few weeks ago and it ran fine on a slightly different setup
probably something wrong with the current setup that the asset isn't uploaded
the cues are from the count not updating and the traffic is not much
I'm gonna give up on this. I'm gonna see in a few weeks/months/releases. without logs, I can't do much really
sounds good, we don't work a lot with k8s besides the community support for HELM, k8s
but that's fine. Immich is progressing fast, and I love it!
I don't think this is related to k8s, since the web works fine
are you accessing through your local ip?
I'm accessing through my ingress, via https
try local ip
I have a valid domain, etc
it doesn't really have an IP
It's either (1) problem with the app after the backend is reinstalled or (2) a networking issue from the app.
it goes through nginx ingress to the container
I guess that might be the source of the issue
but it works fine for the web interface
everything works there
it's just the iOS app that doesn't work. I don't think ingress is the problem
Do you have anything limiting the upload size?
client_max_body_size settingI'm gonna add some annotations, but since @bo0tzz is running the same helm chart, it should work
also, if it was upload size limit, the web interface would struggle just as much, given those files are similarly sized
yeah, agree. For last effort, maybe just add that in?
oh shit
folks
HEIC usually has larger size though
it uploads
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
that was all I added
the question is why @bo0tzz's ingress didn't have this issue
but anyway. thanks a lot!
I'm gonna do a PR to the helm chart repo mentioning this as a possible issue if you want
I do have the body size set
I really should've caught that one lol, sorry
Yes please
(those particular annotations won't work if somebody is using something else than nginx, so they shouldn't get included in the yaml itself)
This had to resolve in an error somewhere, no? Like a http exception on the app side? We could probably make sure this is reported in the logs.
immich-proxy could report it
oh wait, no
Usually it shows in the proxy container but we disabled the proxy log since it is too chatty
No I mean the app
I don't think it's visible to you at all at proxy level. ingress won't forward anything to proxy in this case (I think)
Should show a http exception
I can try
There should be a 413 error on upload if this is cut off by nginx
If the logs stated the error it'd be easy to tell people to update their web server config
I'm pretty sure it cut off by ingress, before it reaches immich-proxy
but I might be wrong
Yes and it cuts it off by sending a 413 to the mobile app http client.
So showing 413 in the mobile app logs
yeah. mobile app reports nothing
I can check the mobile app in this case
there is an error reporting on upload error
Yeah, it should be possible to log this all I'm saying.
so somehow this doesn't get caught
Just checking there are no errors in the app logs @hhes ?
No errors
Yeah, so my take away from this is just - this is a know issue, but we can make bug/track an issue that mobile app doesn't log upload errors.
in corpo terms I'd create an epic to improve logging in general, but I don't know how you manage your project
You can open an issue on Github
GitHub
[Feature]: Log more informations · Issue #1093 · immich-app/immich
Feature detail Recently, I had an issue where I couldn't upload files via the mobile app. The resolution was to increase file upload limits on Immich's ingress. That problem could h...
@hhes I'm gonna update the helm-chart after christman, did you create the issue? I think this should be part of the docs
I forgot! I'm gonna add myself a reminder to create a PR with that ingress issue
also, it'd be great if the helm chart would allow setting cgroups (I don't remember how k8s calls it), because if I run immich-microservices on raspi instead of an x86 node, the node explodes due to high load
I'm gonna add an issue for that
You mean resource limits? That should be possible afaik
yeah, limit cpu time and memory
it's totally possible
also, it'd nice to be able to set node affinity for same reasons