Immich•3y ago

CLI Import - Gateway Timeout

I was able to successfully import most of my files but there are about 200 or so that the CLI import tool isn't happy about. I'm going to venture a guess it has something to do regarding the size of the files. I cd to my directory of folders/files I want to upload and run:

docker run -it --rm -v "$(pwd)":/import ghcr.io/immich-app/immich-cli:latest upload --key <redacted> --server https://photos.domain.com/api -t 10

. It runs for awhile and eventually outputs a bunch of `504 Gateway Time-out. The server didn't respond in time." Any ideas how I can fix this? It's probably hitting an upload size limit.

23 Replies

cykotixOP•3y ago

Example output:

  {
    file: '/import/2022/12/22-12-25 08-45-56 0483.mov',
    reason: Error: Request failed with status code 504
        at createError (/usr/src/app/node_modules/axios/lib/core/createError.js:16:15)
        at settle (/usr/src/app/node_modules/axios/lib/core/settle.js:17:12)
        at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/axios/lib/adapters/http.js:322:11)
        at IncomingMessage.emit (node:events:539:35)
        at endReadableNT (node:internal/streams/readable:1345:12)
        at processTicksAndRejections (node:internal/process/task_queues:83:21) {
      config: [Object],
      request: [ClientRequest],
      response: [Object],
      isAxiosError: true,
      toJSON: [Function: toJSON]
    },
    response: '<html><body><h1>504 Gateway Time-out</h1>\n' +
      "The server didn't respond in time.\n" +
      '</body></html>\n'
  },
  {
    file: '/import/2021/12/20211225_083842.mp4',
    reason: Error: Request failed with status code 504
        at createError (/usr/src/app/node_modules/axios/lib/core/createError.js:16:15)
        at settle (/usr/src/app/node_modules/axios/lib/core/settle.js:17:12)
        at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/axios/lib/adapters/http.js:322:11)
        at IncomingMessage.emit (node:events:539:35)
        at endReadableNT (node:internal/streams/readable:1345:12)
        at processTicksAndRejections (node:internal/process/task_queues:83:21) {
      config: [Object],
      request: [ClientRequest],
      response: [Object],
      isAxiosError: true,
      toJSON: [Function: toJSON]
    },
    response: '<html><body><h1>504 Gateway Time-out</h1>\n' +
      "The server didn't respond in time.\n" +
      '</body></html>\n'
  }

  {
    file: '/import/2022/12/22-12-25 08-45-56 0483.mov',
    reason: Error: Request failed with status code 504
        at createError (/usr/src/app/node_modules/axios/lib/core/createError.js:16:15)
        at settle (/usr/src/app/node_modules/axios/lib/core/settle.js:17:12)
        at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/axios/lib/adapters/http.js:322:11)
        at IncomingMessage.emit (node:events:539:35)
        at endReadableNT (node:internal/streams/readable:1345:12)
        at processTicksAndRejections (node:internal/process/task_queues:83:21) {
      config: [Object],
      request: [ClientRequest],
      response: [Object],
      isAxiosError: true,
      toJSON: [Function: toJSON]
    },
    response: '<html><body><h1>504 Gateway Time-out</h1>\n' +
      "The server didn't respond in time.\n" +
      '</body></html>\n'
  },
  {
    file: '/import/2021/12/20211225_083842.mp4',
    reason: Error: Request failed with status code 504
        at createError (/usr/src/app/node_modules/axios/lib/core/createError.js:16:15)
        at settle (/usr/src/app/node_modules/axios/lib/core/settle.js:17:12)
        at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/axios/lib/adapters/http.js:322:11)
        at IncomingMessage.emit (node:events:539:35)
        at endReadableNT (node:internal/streams/readable:1345:12)
        at processTicksAndRejections (node:internal/process/task_queues:83:21) {
      config: [Object],
      request: [ClientRequest],
      response: [Object],
      isAxiosError: true,
      toJSON: [Function: toJSON]
    },
    response: '<html><body><h1>504 Gateway Time-out</h1>\n' +
      "The server didn't respond in time.\n" +
      '</body></html>\n'
  }

jrasm91•3y ago

What version of the server are you using?

cykotixOP•3y ago

1.43.1

[1] Pinging server...
Server status: OK
[2] Logging in...
Login status: OK
Connected to Immich with user [email protected]
[3] Checking directory...
Directory status: OK
[4] Indexing files...
Indexing file: OK
Found 43920 assets in specified directory
[5] Gathering device's asset info from server...
A total of 198 assets will be uploaded to the server
Do you want to start upload now? (y/n)

[1] Pinging server...
Server status: OK
[2] Logging in...
Login status: OK
Connected to Immich with user [email protected]
[3] Checking directory...
Directory status: OK
[4] Indexing files...
Indexing file: OK
Found 43920 assets in specified directory
[5] Gathering device's asset info from server...
A total of 198 assets will be uploaded to the server
Do you want to start upload now? (y/n)

It uploaded the majority of those without issue. Having issues with the last 198. It was at 202 before so it was able to randomly make it through 4 this last pass.

jrasm91•3y ago

And the one from the error appeared to be a video. Are you communicating through any proxy or connecting directly to let 2283?

cykotixOP•3y ago

going through haproxy. let me attempt directly. So that solved the initial issue. I also rectified it with HAProxy by increasing Server Timeout from the default 30s. For now I increased it to 5m and I no longer get the 504 Gateway Time-out errors. However now I've got a new issue. Anyway to increase the verbosity of the output?

docker run -it --rm -v "$(pwd)":/import ghcr.io/immich-app/immich-cli:latest upload --key <redacted> --server http://immich.xxx.com:2283/api
[1] Pinging server...
Server status: OK
[2] Logging in...
Login status: OK
Connected to Immich with user [email protected]
[3] Checking directory...
Directory status: OK
[4] Indexing files...
Indexing file: OK
Found 43920 assets in specified directory
[5] Gathering device's asset info from server...
A total of 175 assets will be uploaded to the server
Do you want to start upload now? (y/n) y
Start uploading...
Upload Progress | ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ | 0% || 75/43920 || Current file [/import/2022/01/22-01-09 08-21-24 3B72.jpg](node:1) Warning: Closing file descriptor 24 on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
(node:1) [DEP0137] DeprecationWarning: Closing a FileHandle object on garbage collection is deprecated. Please close FileHandle objects explicitly using FileHandle.prototype.close(). In the future, an error will be thrown if a file descriptor is closed during garbage collection.
(node:1) Warning: Closing file descriptor 25 on garbage collection
(node:1) Warning: Closing file descriptor 26 on garbage collection
Upload Progress | ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ | 0% || 175/43920 || Current file [/import/2022/12/22-12-25 08-45-56 0483.mov]
Failed to upload 0 files  []

docker run -it --rm -v "$(pwd)":/import ghcr.io/immich-app/immich-cli:latest upload --key <redacted> --server http://immich.xxx.com:2283/api
[1] Pinging server...
Server status: OK
[2] Logging in...
Login status: OK
Connected to Immich with user [email protected]
[3] Checking directory...
Directory status: OK
[4] Indexing files...
Indexing file: OK
Found 43920 assets in specified directory
[5] Gathering device's asset info from server...
A total of 175 assets will be uploaded to the server
Do you want to start upload now? (y/n) y
Start uploading...
Upload Progress | ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ | 0% || 75/43920 || Current file [/import/2022/01/22-01-09 08-21-24 3B72.jpg](node:1) Warning: Closing file descriptor 24 on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
(node:1) [DEP0137] DeprecationWarning: Closing a FileHandle object on garbage collection is deprecated. Please close FileHandle objects explicitly using FileHandle.prototype.close(). In the future, an error will be thrown if a file descriptor is closed during garbage collection.
(node:1) Warning: Closing file descriptor 25 on garbage collection
(node:1) Warning: Closing file descriptor 26 on garbage collection
Upload Progress | ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ | 0% || 175/43920 || Current file [/import/2022/12/22-12-25 08-45-56 0483.mov]
Failed to upload 0 files  []

Just to be clear,the above output is connecting directly, not through HAProxy just to remove that from the equation.

jrasm91•3y ago

Hmmm. Did it successfully upload, but just print out that warning?

cykotixOP•3y ago

It seems to think there are 175 assets that haven't been uploaded

jrasm91•3y ago

I don't think there is any logging in the CLI

cykotixOP•3y ago

Doesn't appear to have uploaded either

jrasm91•3y ago

Are they all movies/videos?

cykotixOP•3y ago

Hard to tell what the 175 are it thinks need to be uploaded

cyko@immich:/mnt/immich/e8482e01-ae0f-4cab-8a84-348c6542f3b2$ find . -name '22-12-25 08-45-56 0483.mov'
cyko@immich:/mnt/immich/e8482e01-ae0f-4cab-8a84-348c6542f3b2$

cyko@immich:/mnt/immich/e8482e01-ae0f-4cab-8a84-348c6542f3b2$ find . -name '22-12-25 08-45-56 0483.mov'
cyko@immich:/mnt/immich/e8482e01-ae0f-4cab-8a84-348c6542f3b2$

didn't find that mov, let me try that jpg next. those are the only 2 file names that show up in the output, other than when it's iterating through files. What I pasted is literally all I see in the output now. Same with the jpg, so no, not all movies/videos it seems:

cyko@immich:/mnt/immich/e8482e01-ae0f-4cab-8a84-348c6542f3b2$ find . -name '22-01-09 08-21-24 3B72.jpg'
cyko@immich:/mnt/immich/e8482e01-ae0f-4cab-8a84-348c6542f3b2$

cyko@immich:/mnt/immich/e8482e01-ae0f-4cab-8a84-348c6542f3b2$ find . -name '22-01-09 08-21-24 3B72.jpg'
cyko@immich:/mnt/immich/e8482e01-ae0f-4cab-8a84-348c6542f3b2$

Other things from that day uploaded with no issue:

cyko@immich:/mnt/immich/e8482e01-ae0f-4cab-8a84-348c6542f3b2$ find . -name '22-01-09*'
./2022/02/22-01-09 09-20-29 5726.mov
./2022/02/22-01-09 10-46-20 BC56.mov
./2022/02/22-01-09 08-49-49 73FF.mov
./2022/02/22-01-09 10-11-13 F13C.mov
./2022/02/22-01-09 10-10-31 6A4D.mov
./2022/02/22-01-09 08-51-17 8672.mov
./2022/01/22-01-09 12-19-04 3667.jpg
./2022/01/22-01-09 08-19-31 3644.jpg
./2022/01/22-01-09 08-51-17 3650.mov
./2022/01/22-01-09 08-19-40 3645.jpg
./2022/01/22-01-09 08-01-22 3643.jpg
./2022/01/22-01-09 08-19-59 3647.jpeg
./2022/01/22-01-09 08-19-42 3646.jpg
./2022/01/22-01-09 08-21-24 3648.jpeg
...<snip>

cyko@immich:/mnt/immich/e8482e01-ae0f-4cab-8a84-348c6542f3b2$ find . -name '22-01-09*'
./2022/02/22-01-09 09-20-29 5726.mov
./2022/02/22-01-09 10-46-20 BC56.mov
./2022/02/22-01-09 08-49-49 73FF.mov
./2022/02/22-01-09 10-11-13 F13C.mov
./2022/02/22-01-09 10-10-31 6A4D.mov
./2022/02/22-01-09 08-51-17 8672.mov
./2022/01/22-01-09 12-19-04 3667.jpg
./2022/01/22-01-09 08-19-31 3644.jpg
./2022/01/22-01-09 08-51-17 3650.mov
./2022/01/22-01-09 08-19-40 3645.jpg
./2022/01/22-01-09 08-01-22 3643.jpg
./2022/01/22-01-09 08-19-59 3647.jpeg
./2022/01/22-01-09 08-19-42 3646.jpg
./2022/01/22-01-09 08-21-24 3648.jpeg
...<snip>

jrasm91•3y ago

Hmm, usually i have a few guesses of what could be going on but this is pretty new for me. What version of node are you running out of curiosity? And is the CPU maxed during upload?

cykotixOP•3y ago

 node --version
v12.22.9

 node --version
v12.22.9

This lives in a VM and given it 10 cores. Running the process again and I'd say it's averaging about 50% cpu

jrasm91•3y ago

The immich-server process?

cykotixOP•3y ago

36-40% no combination of ffmpeg, node and nginx

jrasm91•3y ago

I'd definitely try with node 16. I didn't think it would even complete with version 12

cykotixOP•3y ago

but isn't node running inside the container? probably should see what version is running inside there

docker exec -it immich_server node --version
v16.15.1

docker exec -it immich_server node --version
v16.15.1

 docker ps
CONTAINER ID   IMAGE                                        COMMAND                  CREATED        STATUS       PORTS                                       NAMES
63e1dfa90e7c   altran1502/immich-proxy:release              "/start.sh"              18 hours ago   Up 4 hours   0.0.0.0:2283->8080/tcp, :::2283->8080/tcp   immich_proxy
5b363331b0b4   altran1502/immich-machine-learning:release   "/bin/sh ./entrypoin…"   18 hours ago   Up 4 hours                                               immich_machine_learning
efa61f160019   altran1502/immich-server:release             "/bin/sh ./start-ser…"   18 hours ago   Up 4 hours   3001/tcp                                    immich_server
b189173f0184   altran1502/immich-server:release             "/bin/sh ./start-mic…"   18 hours ago   Up 4 hours   3001/tcp                                    immich_microservices
0558620f58ff   altran1502/immich-web:release                "/bin/sh ./entrypoin…"   18 hours ago   Up 4 hours   3000/tcp                                    immich_web
b825f846ef4e   redis:6.2                                    "docker-entrypoint.s…"   32 hours ago   Up 4 hours   6379/tcp                                    immich_redis
afe6361962e0   postgres:14                                  "docker-entrypoint.s…"   32 hours ago   Up 4 hours   5432/tcp                                    immich_postgres

 docker ps
CONTAINER ID   IMAGE                                        COMMAND                  CREATED        STATUS       PORTS                                       NAMES
63e1dfa90e7c   altran1502/immich-proxy:release              "/start.sh"              18 hours ago   Up 4 hours   0.0.0.0:2283->8080/tcp, :::2283->8080/tcp   immich_proxy
5b363331b0b4   altran1502/immich-machine-learning:release   "/bin/sh ./entrypoin…"   18 hours ago   Up 4 hours                                               immich_machine_learning
efa61f160019   altran1502/immich-server:release             "/bin/sh ./start-ser…"   18 hours ago   Up 4 hours   3001/tcp                                    immich_server
b189173f0184   altran1502/immich-server:release             "/bin/sh ./start-mic…"   18 hours ago   Up 4 hours   3001/tcp                                    immich_microservices
0558620f58ff   altran1502/immich-web:release                "/bin/sh ./entrypoin…"   18 hours ago   Up 4 hours   3000/tcp                                    immich_web
b825f846ef4e   redis:6.2                                    "docker-entrypoint.s…"   32 hours ago   Up 4 hours   6379/tcp                                    immich_redis
afe6361962e0   postgres:14                                  "docker-entrypoint.s…"   32 hours ago   Up 4 hours   5432/tcp                                    immich_postgres

just for giggles, i'm going to restart everything and see if something is just "stuck"

jrasm91•3y ago

Oh, i didn't realize you were using the docker container

cykotixOP•3y ago

Seems there were some ffmpeg processes that might have been orphaned due to the original HAProxy issue. Now it's sitting at <3% while running the process. Outcome was the same. Yea, sorry, probably should have included that for ya... Now I'm starting to wonder if I should nuke the volume and start from scratch. Wondering if the server timeout from the proxy caused some files to be in purgatory

jrasm91•3y ago

Yeah, sorry I couldn't be more help. Not sure what else to try

cykotixOP•3y ago

I'm happy to dig into the db for you if you think it would be helpful, but seems most prefer nginx for the proxy of choice here vs haproxy. I might be an edge case.

jrasm91•3y ago

The proxy timeout shouldn't really be an issue It just means the server didn't respond soon enough, but everything else would have worked fine

cykotixOP•3y ago

shrug i'll wipe and re-upload. not a big deal. however, for those that may search, you definitely need to increase your Server Time Out >30s when using HAProxy. Normally the default would be fine. Only becomes an issue for large files, which is why it seems to mostly affect videos.

Gaming

Programming

CLI Import - Gateway Timeout

Did you find this page helpful?