Immich•3y ago

Trouble with kubernetes deployments. Getting random `ECONNREFUSED` and 404s.

Been trying to get the latest version deployed in my k3s cluster. I'm not using the helm charts, but instead crafted some simple service & deployment manifests from comparing the helm releases in the k8s-at-home community charts and the docker-compose manifests provided by immich in the main docs. That said, I've got everything in a running state, listening in the cluster. I think, however, I'm not applying the my ingress resources correctly? I'm really unsure, but I've attached a screen of the errors I get when I port-forward on port 3000 to the immich-web pod and use the browser to sign up. When refreshing on that page, I either get ECONNREFUSED or it appears to let me sign up, but 404s when attempting to submit. Happy to provide any additional detail, and appreciate any advice!

35 Replies

shrinedoggOP•3y ago

My ingress manifest I think is the issue, but I'm unsure what the backend should be when it's not using the /api prefix.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: immich-ingress
  namespace: immich
  annotations:  
    nginx.ingress.kubernetes.io/configuration-snippet: |
      rewrite /api/(.*) /$1 break;
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
spec:
  ingressClassName: nginx
  rules:
  - host: photos.domain.thang
    http:
      paths:
      - path: /
        pathType: Prefix
        backend: 
          service:
            name: immich-proxy-service
            port:
              number: 8080
      - path: /api
        pathType: Prefix
        backend: 
          service:
            name: immich-server-service
            port:
              number: 3001

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: immich-ingress
  namespace: immich
  annotations:  
    nginx.ingress.kubernetes.io/configuration-snippet: |
      rewrite /api/(.*) /$1 break;
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
spec:
  ingressClassName: nginx
  rules:
  - host: photos.domain.thang
    http:
      paths:
      - path: /
        pathType: Prefix
        backend: 
          service:
            name: immich-proxy-service
            port:
              number: 8080
      - path: /api
        pathType: Prefix
        backend: 
          service:
            name: immich-server-service
            port:
              number: 3001

jrasm91•3y ago

Non API request should be sent to immich-web Are you still running the immich-proxy? If so then just send all request to that. It'll forward them between the immich-web and immich-server

shrinedoggOP•3y ago

Ah I am using the proxy Got it, I'll give that a shot Should I still be applying the rewrite annotations to the ingress when using the proxy-service?

bo0tzz•3y ago

Nah, the proxy should take care of that

jrasm91•3y ago

I think you can either ignore all of that stuff and send everything to the proxy (recommended) or try to add all of the immich proxy stuff here and then not run the proxy at all.

bo0tzz•3y ago

Indeed. Proxy is the easy way out

jrasm91•3y ago

At the end of the day your can treat immich proxy port 8080 as a single point of entry for the immich website and API, so just send everything there and you'll be golden.

shrinedoggOP•3y ago

Hm, still getting different errors, but it seems I'm hitting the proxy backend -

2023/02/26 19:32:46 [error] 31#31: *51 connect() failed (111: Connection refused) while connecting to upstream, client: 10.42.0.64, server: , request: "GET / HTTP/1.1", upstream: "http://10.43.15.120:3000/", host: "photos.domain.thangz"

2023/02/26 19:32:46 [error] 31#31: *51 connect() failed (111: Connection refused) while connecting to upstream, client: 10.42.0.64, server: , request: "GET / HTTP/1.1", upstream: "http://10.43.15.120:3000/", host: "photos.domain.thangz"

2023/02/26 19:40:17 [error] 32#32: *59 connect() failed (111: Connection refused) while connecting to upstream, client: 10.42.0.64, server: , request: "GET /_app/immutable/chunks/preload-helper-41c905a7.js HTTP/1.1", upstream: "http://10.43.15.120:3000/_app/immutable/chunks/preload-helper-41c905a7.js", host: "photos.domain.thangz", referrer: "https://photos.domain.thangz/_app/immutable/start-a9228c3e.js"

2023/02/26 19:40:17 [error] 32#32: *59 connect() failed (111: Connection refused) while connecting to upstream, client: 10.42.0.64, server: , request: "GET /_app/immutable/chunks/preload-helper-41c905a7.js HTTP/1.1", upstream: "http://10.43.15.120:3000/_app/immutable/chunks/preload-helper-41c905a7.js", host: "photos.domain.thangz", referrer: "https://photos.domain.thangz/_app/immutable/start-a9228c3e.js"

Those are logs from the proxy container. The web container just says Listening on 0.0.0.0:3000 The only the other thing interesting so far is the server container logs have a few entries of...

missing 'error' handler on this Redis client

missing 'error' handler on this Redis client

🤷‍♂️

bo0tzz•3y ago

This happens when you're running redis 7 instead of 6 iirc

shrinedoggOP•3y ago

I am pretty sure that checks out

bo0tzz•3y ago

This one I'm not sure about. If you ping and curl the web service from inside proxy, does it work?

shrinedoggOP•3y ago

Hm, interesting, I can't curl http://immich-web-service.immich.svc.cluster.local:3000 from the proxy...

shrinedoggOP•3y ago

ClusterIP is at least... there... and the container says listening.. 🤔

bo0tzz•3y ago

Do you have networkpolicies or anything like that in your cluster?

shrinedoggOP•3y ago

Nope, it's pretty barebones k3s on baremetal ubuntu

bo0tzz•3y ago

Hmmm

shrinedoggOP•3y ago

Gonna try the IP next Of the svc Worked. It's always DNS!

bo0tzz•3y ago

dig immich-web-service?

shrinedoggOP•3y ago

I have dnsPolicy: ClusterFirst in the deployment yaml for each of these pods...

bo0tzz•3y ago

That's the default as well, so that shouldn't be an issue

shrinedoggOP•3y ago

No dig or nslookup on the proxy container

bo0tzz•3y ago

Ah, dangit

shrinedoggOP•3y ago

Seems it's actually intermittent, actually, with IP 😞 Do pretty much the same behavior I was seeing in the browser where about 1/10 would get a response from the backend Should I rollback to Redis 6? Could that be breaking the whole experience?

bo0tzz•3y ago

Redis shouldn't be involved on the web side, so that's unlikely Was this with the ingress still, or through immich-proxy? It looks to me like this is a networking issue, not an immich one 🤔

shrinedoggOP•3y ago

Just wanted to circle back on this one, it was a problem with Label/Selectors. 😅 Got into the admin panel for the first time just now.

shrinedoggOP•3y ago

https://giphy.com/gifs/borat-great-success-a0h7sAqON67nO

Giphy

shrinedoggOP•3y ago

Another question for you fine gents - should I be using the release or v1.49.0 tag for image? Upon getting into the mobile app, it complained about my server version Figured that out as well - same hash.

jrasm91•3y ago

Yeah, i think everyone uses the release tag

shrinedoggOP•3y ago

I'm onto what I'm thinking a permissions issue, perhaps?

node:internal/fs/utils:345
    throw err;
    ^

Error: ENOENT: no such file or directory, mkdir 'upload/5472bc26-36ce-4acd-9767-4c6526809c90/original/9a21da1923cb1ab201f776d36a9de42c9bb54399a1984c790c96c15d9fc03717'  
    at mkdirSync (node:fs:1349:3)
    at DiskStorage.destination [as getDestination] (/usr/src/app/dist/apps/immich/apps/immich/src/config/asset-upload.config.js:64:28)
    at DiskStorage._handleFile (/usr/src/app/node_modules/multer/storage/disk.js:31:8)
    at Object._handleFile (/usr/src/app/dist/apps/immich/apps/immich/src/config/asset-upload.config.js:25:21)
    at /usr/src/app/node_modules/multer/lib/make-middleware.js:137:17
    at fileFilter (/usr/src/app/dist/apps/immich/apps/immich/src/config/asset-upload.config.js:48:9)
    at wrappedFileFilter (/usr/src/app/node_modules/multer/index.js:44:7)
    at Multipart.<anonymous> (/usr/src/app/node_modules/multer/lib/make-middleware.js:107:7)
    at Multipart.emit (node:events:527:28)
    at HeaderParser.cb (/usr/src/app/node_modules/busboy/lib/types/multipart.js:358:14) {
  errno: -2,
  syscall: 'mkdir',
  code: 'ENOENT',
  path: 'upload/5472bc26-36ce-4acd-9767-4c6526809c90/original/9a21da1923cb1ab201f776d36a9de42c9bb54399a1984c790c96c15d9fc03717'
}

node:internal/fs/utils:345
    throw err;
    ^

Error: ENOENT: no such file or directory, mkdir 'upload/5472bc26-36ce-4acd-9767-4c6526809c90/original/9a21da1923cb1ab201f776d36a9de42c9bb54399a1984c790c96c15d9fc03717'  
    at mkdirSync (node:fs:1349:3)
    at DiskStorage.destination [as getDestination] (/usr/src/app/dist/apps/immich/apps/immich/src/config/asset-upload.config.js:64:28)
    at DiskStorage._handleFile (/usr/src/app/node_modules/multer/storage/disk.js:31:8)
    at Object._handleFile (/usr/src/app/dist/apps/immich/apps/immich/src/config/asset-upload.config.js:25:21)
    at /usr/src/app/node_modules/multer/lib/make-middleware.js:137:17
    at fileFilter (/usr/src/app/dist/apps/immich/apps/immich/src/config/asset-upload.config.js:48:9)
    at wrappedFileFilter (/usr/src/app/node_modules/multer/index.js:44:7)
    at Multipart.<anonymous> (/usr/src/app/node_modules/multer/lib/make-middleware.js:107:7)
    at Multipart.emit (node:events:527:28)
    at HeaderParser.cb (/usr/src/app/node_modules/busboy/lib/types/multipart.js:358:14) {
  errno: -2,
  syscall: 'mkdir',
  code: 'ENOENT',
  path: 'upload/5472bc26-36ce-4acd-9767-4c6526809c90/original/9a21da1923cb1ab201f776d36a9de42c9bb54399a1984c790c96c15d9fc03717'
}

It looks like it can't mkdir in my NFS share PVC

jrasm91•3y ago

Yeah it'll definitely need to create a few directories on the upload folder path.

shrinedoggOP•3y ago

Seems legit; would there be anything beyond making sure the PGID/PUID match perms on the filesystem?

shrinedoggOP•3y ago

I see the share in an exec ls on the pod

jrasm91•3y ago

That should be all you need i believe. What are the permissions on the upload folder?

shrinedoggOP•3y ago

I think it's a path issue, I'm onto to it. When I ls that "upload" directory you see in that shot, it actually points at the base directory of the share, not the upload folder I suspected it would point to Awesome, first backup in progress. Thanks again for all the assistance, hopefully I can make time to contribute in some meaningful way in the near future!

bo0tzz•3y ago

We don't use puid/pgid at all, I believe?

Gaming

Programming

Trouble with kubernetes deployments. Getting random `ECONNREFUSED` and 404s.

Did you find this page helpful?