Immich•2y ago

Traefik doesn't allow large size upload

Following these messages https://discord.com/channels/979116623879368755/994044917355663450/1131786157361999972 https://discord.com/channels/979116623879368755/994044917355663450/1131862468034035764 I was not able to upload large files via CLI Bulk upload. The file I have isolated is a MOV file 2.5Gb in size. When I tried with immich-proxy port as a temporary solution, it works, but not when I use Traefik. Since this is a reverse proxy problem, my phone likely suffers from the same issue.

35 Replies

alittlebeeOP•2y ago

I've tried this but it did not help. https://discord.com/channels/979116623879368755/1131862468034035764/1131866743384571994

etnoy•2y ago

Does traefik cache requests on disk? What is the free disk space of the traefik system?

Allram•2y ago

Do you happen to use Cloudflare? Traefik does not cache as default to disk. Only on Enterprise version or by plugins

etnoy•2y ago

I'm not familiar with traefik, but I had that issue with nginx.

Allram•2y ago

This is my Immich Traefik settings:

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${LIBRARY_LOCATION}:/usr/src/app/upload/library
      - ${UPLOAD_LOCATION}:/usr/src/app/upload/upload
      - ${THUMBS_LOCATION}:/usr/src/app/upload/thumbs
      - ${PROFILE_LOCATION}:/usr/src/app/upload/profile
      - ${VIDEO_LOCATION}:/usr/src/app/upload/encoded-video
      - /tmp/Postgresql/postgresql/:/var/run/postgresql
      - /tmp/Redis/Redis/:/var/run/redis
      - ${IMPORT_LOCATION}:/import
    env_file:
      - .env
    depends_on:
      - typesense
    restart: always
    networks:
      immich:
        ipv4_address: 192.168.30.17
    labels:
    - "TZ=Europe/Oslo"
    - "traefik.enable=true"
    - "traefik.http.routers.immich-api.entryPoints=https"
    - "traefik.http.services.immich-api.loadbalancer.server.port=3001"
    - "traefik.http.routers.immich-api.rule=Host(`photos.domain.org`) && Pathprefix(`/api`)"
    - "traefik.http.middlewares.service-immich-api-strip.stripprefix.prefixes=/api"
    - "traefik.http.routers.immich-api.middlewares=middlewares-secure-headers@file,service-immich-api-strip"
    environment:
      - TZ=Europe/Oslo

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${LIBRARY_LOCATION}:/usr/src/app/upload/library
      - ${UPLOAD_LOCATION}:/usr/src/app/upload/upload
      - ${THUMBS_LOCATION}:/usr/src/app/upload/thumbs
      - ${PROFILE_LOCATION}:/usr/src/app/upload/profile
      - ${VIDEO_LOCATION}:/usr/src/app/upload/encoded-video
      - /tmp/Postgresql/postgresql/:/var/run/postgresql
      - /tmp/Redis/Redis/:/var/run/redis
      - ${IMPORT_LOCATION}:/import
    env_file:
      - .env
    depends_on:
      - typesense
    restart: always
    networks:
      immich:
        ipv4_address: 192.168.30.17
    labels:
    - "TZ=Europe/Oslo"
    - "traefik.enable=true"
    - "traefik.http.routers.immich-api.entryPoints=https"
    - "traefik.http.services.immich-api.loadbalancer.server.port=3001"
    - "traefik.http.routers.immich-api.rule=Host(`photos.domain.org`) && Pathprefix(`/api`)"
    - "traefik.http.middlewares.service-immich-api-strip.stripprefix.prefixes=/api"
    - "traefik.http.routers.immich-api.middlewares=middlewares-secure-headers@file,service-immich-api-strip"
    environment:
      - TZ=Europe/Oslo

And Immich-web:

immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - .env
    restart: always
    networks:
      immich:
        ipv4_address: 192.168.30.16
    labels:
    - "TZ=Europe/Oslo"
    - "traefik.enable=true"
    - "traefik.http.routers.immich-http.entryPoints=https"
    - "traefik.http.services.immich-http.loadbalancer.server.port=3000"
    - "traefik.http.routers.immich-http.rule=Host(`photos.domain.org`)"
    - "traefik.http.routers.immich-http.middlewares=middlewares-secure-headers@file"
    environment:
      - TZ=Europe/Oslo

immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - .env
    restart: always
    networks:
      immich:
        ipv4_address: 192.168.30.16
    labels:
    - "TZ=Europe/Oslo"
    - "traefik.enable=true"
    - "traefik.http.routers.immich-http.entryPoints=https"
    - "traefik.http.services.immich-http.loadbalancer.server.port=3000"
    - "traefik.http.routers.immich-http.rule=Host(`photos.domain.org`)"
    - "traefik.http.routers.immich-http.middlewares=middlewares-secure-headers@file"
    environment:
      - TZ=Europe/Oslo

` I also run:

        - crowdsec-bouncer-plugin@file
        - middlewares-compress@file
        - middlewares-rate-limit@file

        - crowdsec-bouncer-plugin@file
        - middlewares-compress@file
        - middlewares-rate-limit@file

As plugins in Traefik, can you send your traefikk.yml and relevant config from fileconfig.yml @alittlebee ?

alittlebeeOP•2y ago

Thanks @Allram , I will test out the configuration and let you know about the result. That looks promising ❤️

alittlebeeOP•2y ago

@Allram I have a very simple traefik configuration. All configurations are done through this docker-compose file and no traefik.yml

traefik_docker-compo...

alittlebeeOP•2y ago

And this is my immich config, which works for smaller files.

immich_docker-compos...

alittlebeeOP•2y ago

Would you mind to share your traefik.yml config and how to load it from the docker container?

alittlebeeOP•2y ago

I updated to this and Traefik throws errors

alittlebeeOP•2y ago

Allram•2y ago

Try to remove middleware 'middlewares-secure-headers@file' That is in my fileconfig.yml so it might be that yours are named different or not exist I can share my files later when I'm in front of my PC 👍

alittlebeeOP•2y ago

Thank you. Removining middlewares-secure-headers@file returns my config to the original state, so it works. Looking forward to see your config 🙂

Allram•2y ago

@alittlebee Traefik.yml: https://jpst.it/3jDid Fileconfig.yml: https://jpst.it/3jDjt

alittlebeeOP•2y ago

Thanks, @Allram . Wow you have a hardcore configuration. I'll try it out and keep you posted 🙂

Allram•2y ago

Trying to keep it as secure as I can 😅

alittlebeeOP•2y ago

Did you follow this guide to setup? https://docs.ibracorp.io/traefik/master/docker-compose/config-files-explained This is the only resource I found mentioning fileConfig.yml

Config Files Explained

alittlebeeOP•2y ago

Btw, it is mentioned that traefik.yml is a "static config" and fileConfig is "dynamic config". What do they mean? Ah, I think I start to understand it now.

Allram•2y ago

Yes, followed that and a few other I found around the internet. The static traefik.yml, Traefik will only incorporate the changes when it's restarted. The dynamic fileconfig.yml will be read/updated often by Traefik, so changes can be adapted "on-the-fly" without restarting Traefik.

alittlebeeOP•2y ago

Thanks a lot for the explaination. Now I understand. ❤️ At this point, I am pretty sure I configured everything correctly but none helped. Immich now runs as normal. I loaded a new middleware middlewares-immich-secure-headers

    middlewares-immich-secure-headers:
      headers:
        accessControlAllowMethods:
          - GET
          - OPTIONS
          - PUT
        accessControlMaxAge: 100
        hostsProxyHeaders:
          - "X-Forwarded-Host"
        # sslRedirect: true #replaced with middlewares-https-redirectscheme for v2.5.x
        stsSeconds: 63072000
        stsIncludeSubdomains: true
        stsPreload: true
        forceSTSHeader: true
        # frameDeny: true #overwritten by customFrameOptionsValue
        customFrameOptionsValue: "SAMEORIGIN" #CSP takes care of this but may be needed for organizr.
        contentTypeNosniff: true
        browserXssFilter: true
        # sslForceHost: true # add sslHost to all of the services
        # sslHost: "{{env "DOMAINNAME_SHB"}}"
        referrerPolicy: "same-origin"
        # Setting contentSecurityPolicy is more secure but it can break things. Proper auth will reduce the risk.
        # the below line also breaks some apps due to 'none' - sonarr, radarr, etc.
        # contentSecurityPolicy: "frame-ancestors '*.{{env "DOMAINNAME_SHB"}}:*';object-src 'none';script-src 'none';"
        # Line below, featurePolicy, was deprecated in v2.5.x in favor permissionPolicy
        # featurePolicy: "camera 'none'; geolocation 'none'; microphone 'none'; payment 'none'; usb 'none'; vr 'none';"
        permissionsPolicy: "camera=(), microphone=(), geolocation=(), payment=(), usb=(), vr=()"
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex,"
          server: ""
          # https://community.traefik.io/t/how-to-make-websockets-work-with-traefik-2-0-setting-up-rancher/1732
          X-Forwarded-Proto: "https"

    middlewares-immich-secure-headers:
      headers:
        accessControlAllowMethods:
          - GET
          - OPTIONS
          - PUT
        accessControlMaxAge: 100
        hostsProxyHeaders:
          - "X-Forwarded-Host"
        # sslRedirect: true #replaced with middlewares-https-redirectscheme for v2.5.x
        stsSeconds: 63072000
        stsIncludeSubdomains: true
        stsPreload: true
        forceSTSHeader: true
        # frameDeny: true #overwritten by customFrameOptionsValue
        customFrameOptionsValue: "SAMEORIGIN" #CSP takes care of this but may be needed for organizr.
        contentTypeNosniff: true
        browserXssFilter: true
        # sslForceHost: true # add sslHost to all of the services
        # sslHost: "{{env "DOMAINNAME_SHB"}}"
        referrerPolicy: "same-origin"
        # Setting contentSecurityPolicy is more secure but it can break things. Proper auth will reduce the risk.
        # the below line also breaks some apps due to 'none' - sonarr, radarr, etc.
        # contentSecurityPolicy: "frame-ancestors '*.{{env "DOMAINNAME_SHB"}}:*';object-src 'none';script-src 'none';"
        # Line below, featurePolicy, was deprecated in v2.5.x in favor permissionPolicy
        # featurePolicy: "camera 'none'; geolocation 'none'; microphone 'none'; payment 'none'; usb 'none'; vr 'none';"
        permissionsPolicy: "camera=(), microphone=(), geolocation=(), payment=(), usb=(), vr=()"
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex,"
          server: ""
          # https://community.traefik.io/t/how-to-make-websockets-work-with-traefik-2-0-setting-up-rancher/1732
          X-Forwarded-Proto: "https"

with the same content as you. The error is still there 😦

alittlebeeOP•2y ago

This is my traefik for the immich_server

Allram•2y ago

This is weird 😅 and you don't have Cloudflare or anything in front? Might it be the timeouts that need to be increased?

jrasm91•2y ago

What is the actual error message?

alittlebeeOP•2y ago

It is the same as my original error, this one. https://discord.com/channels/979116623879368755/1131876653442813992/1131876653442813992 I only use CloudFlare to issue certificate/DNS challenge. Not sure if it needs to be configured with something.

Allram•2y ago

No, was just thinking if you use Cloudflare's proxy service, since that will limit to 100MB transfers 🙂

alittlebeeOP•2y ago

Ah, no. I ran cloudflared before and ditched it as soon as I discovered the limitation.

jrasm91•2y ago

I don't see an error. You just say "unable to upload large video". Did you get an actual error somewhere?

alittlebeeOP•2y ago

I linked the error in the message. But here you go.

jrasm91•2y ago

Sorry I clicked through a bit but could not locate it. Can you send a screenshot of the error further down? The "response" section might have some more details.

alittlebeeOP•2y ago

Of course, here you go

Immich_error_logs

jrasm91•2y ago

 data: '<html>\r\n' +
        '<head><title>500 Internal Server Error</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>500 Internal Server Error</h1></center>\r\n' +
        '<hr><center>cloudflare</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'

 data: '<html>\r\n' +
        '<head><title>500 Internal Server Error</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>500 Internal Server Error</h1></center>\r\n' +
        '<hr><center>cloudflare</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'

There is a reference to cloudflare here. Hmm

A2251•2y ago

I have the same problem now. Any larger videos do not get uploaded via cloudfared due to size limitation so perhaps some type of chunking can be enabled OR "trusted / local" wifi AP can be added to the list in which the immich instance is routed to a local ip vs external. Similar to how home assistant does the implementation When i say now - this is because i lost an ability to open ports & have to use cloudfared tunnels to expose anything out of my local network due to internet provider change.

jrasm91•2y ago

This gets asked quite a bit. I don't really care either way, but you'd have to convince a mobile dev to add it. cc: @Alex

alittlebeeOP•2y ago

I don't use Cloudflared. I use Traefik and Cloudflare is only used to verify DNS

jrasm91•2y ago

Then how are you getting a 500 error from cloudflare?

Gaming

Programming

Traefik doesn't allow large size upload

Did you find this page helpful?