Immich•3y ago

Microservices crash on reverse geocoding

Upgraded to the latest version. Migrated storage successfully. Started generating all thumbnails. Jobs are currently stuck - GENERATE THUMBNAILS 7/25,280; TAG OBJECTS 3/37,498; ENCODE CLIP 2/17,957. Looking at "htop" no immich processes seem to do anything, CPU usage is close to idle. How do I resume the process?

24 Replies

MavorOP•3y ago

According to "docker ps" immich_microservices container seems to got into restart loop. Logs:

[Nest] 1  - 03/30/2023, 1:12:24 AM     LOG [MetadataExtractionProcessor] Initializing Reverse Geocoding
/usr/src/app/node_modules/local-reverse-geocoder/index.js:746
            throw err;
            ^
CsvError: Invalid Record Length: expect 19, got 15 on line 6447
    at Object.__onRecord (/usr/src/app/node_modules/csv-parse/dist/cjs/index.cjs:940:11)
    at Object.parse (/usr/src/app/node_modules/csv-parse/dist/cjs/index.cjs:903:36)
    at Parser._flush (/usr/src/app/node_modules/csv-parse/dist/cjs/index.cjs:1336:26)
    at Parser.final [as _final] (node:internal/streams/transform:112:25)
    at callFinal (node:internal/streams/writable:694:27)
    at prefinish (node:internal/streams/writable:723:7)
    at finishMaybe (node:internal/streams/writable:733:5)
    at afterWrite (node:internal/streams/writable:504:3)
    at onwrite (node:internal/streams/writable:477:7)
    at Parser.Transform._read (node:internal/streams/transform:245:5) {
  code: 'CSV_RECORD_INCONSISTENT_FIELDS_LENGTH',
  bytes: 1048576,
  comment_lines: 0,
  empty_lines: 0,
  invalid_field_length: 0,
  lines: 6447,
  records: 6446,
  columns: false,
  error: undefined,
  header: false,

  index: 15,
  raw: undefined,
  column: 15,
  quoting: false,
  record: [
    '2159045',
    'Maitland',
    'Maitland',
    'MTL,Maitland,Mehjtlend,Mejtland,Mejtlend,West Maitland,mei te lan,mytlnd  yyny gwnyy wlz,Мейтленд,Мејтланд,Мэйтленд,میئٹلینڈ، نیو ساؤتھ ویلز,میتلند، یئنی گونئی ولز,میٹلینڈ,مێیتلەند، نیو ساوت وێلز,მეიტლენდი,梅特兰',
    '-32.73308',
    '151.5574',
    'P',

    'PPLA2',

,
    '',

,

    '15050',
    '',
    '',
    ''
  ]
}

[Nest] 1  - 03/30/2023, 1:12:24 AM     LOG [MetadataExtractionProcessor] Initializing Reverse Geocoding
/usr/src/app/node_modules/local-reverse-geocoder/index.js:746
            throw err;
            ^
CsvError: Invalid Record Length: expect 19, got 15 on line 6447
    at Object.__onRecord (/usr/src/app/node_modules/csv-parse/dist/cjs/index.cjs:940:11)
    at Object.parse (/usr/src/app/node_modules/csv-parse/dist/cjs/index.cjs:903:36)
    at Parser._flush (/usr/src/app/node_modules/csv-parse/dist/cjs/index.cjs:1336:26)
    at Parser.final [as _final] (node:internal/streams/transform:112:25)
    at callFinal (node:internal/streams/writable:694:27)
    at prefinish (node:internal/streams/writable:723:7)
    at finishMaybe (node:internal/streams/writable:733:5)
    at afterWrite (node:internal/streams/writable:504:3)
    at onwrite (node:internal/streams/writable:477:7)
    at Parser.Transform._read (node:internal/streams/transform:245:5) {
  code: 'CSV_RECORD_INCONSISTENT_FIELDS_LENGTH',
  bytes: 1048576,
  comment_lines: 0,
  empty_lines: 0,
  invalid_field_length: 0,
  lines: 6447,
  records: 6446,
  columns: false,
  error: undefined,
  header: false,

  index: 15,
  raw: undefined,
  column: 15,
  quoting: false,
  record: [
    '2159045',
    'Maitland',
    'Maitland',
    'MTL,Maitland,Mehjtlend,Mejtland,Mejtlend,West Maitland,mei te lan,mytlnd  yyny gwnyy wlz,Мейтленд,Мејтланд,Мэйтленд,میئٹلینڈ، نیو ساؤتھ ویلز,میتلند، یئنی گونئی ولز,میٹلینڈ,مێیتلەند، نیو ساوت وێلز,მეიტლენდი,梅特兰',
    '-32.73308',
    '151.5574',
    'P',

    'PPLA2',

,
    '',

,

    '15050',
    '',
    '',
    ''
  ]
}

Alex Tran•3y ago

We are aware of this issue and in the progress of working on a fix. At the moment, you can work around this by destroy the immich_microservices container and recreate it

MavorOP•3y ago

Thanks The workaround seems to be working, I see numbers being updated. Thanks again

Alex Tran•3y ago

Thank you for reporting back!

MavorOP•3y ago

Well, it worked for 15 min before getting into the same state again. Logs show different record now.

Alex Tran•3y ago

hmm interesting Can you post your .env file?

MavorOP•3y ago

DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=<masked>
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis
UPLOAD_LOCATION=<masked>
UPLOAD_THUMBS_LOCATION=<masked>
JWT_SECRET=<masked>
PUBLIC_LOGIN_PAGE_MESSAGE=<masked>
IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003
TYPESENSE_API_KEY=<masked>

DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=<masked>
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis
UPLOAD_LOCATION=<masked>
UPLOAD_THUMBS_LOCATION=<masked>
JWT_SECRET=<masked>
PUBLIC_LOGIN_PAGE_MESSAGE=<masked>
IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003
TYPESENSE_API_KEY=<masked>

This time it lasted 7 min, so I guess I will have to wait until the fix This is the error before it gotten into the restart loop.

[Nest] 1  - 03/30/2023, 1:50:49 AM     LOG [ImmichMicroservice] Running Immich Microservices in PRODUCTION environment - version 1.52.1 - Listening on port: 3002
[...]
[Nest] 1  - 03/30/2023, 1:57:14 AM     LOG [MediaService] Generating Video Thumbnail Success 4548b8d3-cc42-4b2c-a193-4c8f5a512e52
/usr/src/app/node_modules/pg/lib/client.js:132
      const error = this._ending ? new Error('Connection terminated') : new Error('Connection terminated unexpectedly')
                                                                        ^

Error: Connection terminated due to connection timeout
    at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
    at Object.onceWrapper (node:events:641:28)
    at Connection.emit (node:events:527:28)
    at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:57:12)
    at Socket.emit (node:events:527:28)
    at TCP.<anonymous> (node:net:709:12)

[Nest] 1  - 03/30/2023, 1:50:49 AM     LOG [ImmichMicroservice] Running Immich Microservices in PRODUCTION environment - version 1.52.1 - Listening on port: 3002
[...]
[Nest] 1  - 03/30/2023, 1:57:14 AM     LOG [MediaService] Generating Video Thumbnail Success 4548b8d3-cc42-4b2c-a193-4c8f5a512e52
/usr/src/app/node_modules/pg/lib/client.js:132
      const error = this._ending ? new Error('Connection terminated') : new Error('Connection terminated unexpectedly')
                                                                        ^

Error: Connection terminated due to connection timeout
    at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
    at Object.onceWrapper (node:events:641:28)
    at Connection.emit (node:events:527:28)
    at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:57:12)
    at Socket.emit (node:events:527:28)
    at TCP.<anonymous> (node:net:709:12)

There are also quite a few errors like that

[Nest] 1  - 03/30/2023, 1:57:13 AM     LOG [MediaService] Start Generating Video Thumbnail
Request #1680141409969: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
Request #1680141409969: Sleeping for 4s and then retrying request...

[Nest] 1  - 03/30/2023, 1:57:13 AM     LOG [MediaService] Start Generating Video Thumbnail
Request #1680141409969: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
Request #1680141409969: Sleeping for 4s and then retrying request...

ahbeng•3y ago

I got it to be more stable by also recreating the Immich-Server and Immich-Microservices somehow. Maybe its luck but both of those containers does share the same image. Might as well recreate both then IMO.

etnoy•3y ago

Ah, so this is the reason my microservices are crashing This is my error message:

immich-microservices_1     |
immich-microservices_1     | /usr/src/app/node_modules/local-reverse-geocoder/index.js:746
immich-microservices_1     |             throw err;
immich-microservices_1     |             ^
immich-microservices_1     | Error downloading GeoNames cities500 data: Error: aborted
immich-microservices_1     | (Use `node --trace-uncaught ...` to show where the exception was thrown)

immich-microservices_1     |
immich-microservices_1     | /usr/src/app/node_modules/local-reverse-geocoder/index.js:746
immich-microservices_1     |             throw err;
immich-microservices_1     |             ^
immich-microservices_1     | Error downloading GeoNames cities500 data: Error: aborted
immich-microservices_1     | (Use `node --trace-uncaught ...` to show where the exception was thrown)

After disabling reverse geocoding in env I get

immich-microservices_1     |
immich-microservices_1     | /usr/src/app/node_modules/csv-parse/dist/cjs/index.cjs:940
immich-microservices_1     |           new CsvError('CSV_RECORD_INCONSISTENT_FIELDS_LENGTH', [
immich-microservices_1     |           ^
immich-microservices_1     | Error: Invalid Record Length: expect 19, got 4 on line 174673
immich-microservices_1     |     at Object.__onRecord (/usr/src/app/node_modules/csv-parse/dist/cjs/index.cjs:940:11)
immich-microservices_1     |     at Object.parse (/usr/src/app/node_modules/csv-parse/dist/cjs/index.cjs:903:36)
immich-microservices_1     |     at Parser._flush (/usr/src/app/node_modules/csv-parse/dist/cjs/index.cjs:1336:26)
immich-microservices_1     |     at Parser.final [as _final] (node:internal/streams/transform:112:25)
immich-microservices_1     |     at callFinal (node:internal/streams/writable:694:27)
immich-microservices_1     |     at prefinish (node:internal/streams/writable:723:7)
immich-microservices_1     |     at finishMaybe (node:internal/streams/writable:733:5)
immich-microservices_1     |     at afterWrite (node:internal/streams/writable:504:3)
immich-microservices_1     |     at onwrite (node:internal/streams/writable:477:7)
immich-microservices_1     |     at Parser.Transform._read (node:internal/streams/transform:245:5)

immich-microservices_1     |
immich-microservices_1     | /usr/src/app/node_modules/csv-parse/dist/cjs/index.cjs:940
immich-microservices_1     |           new CsvError('CSV_RECORD_INCONSISTENT_FIELDS_LENGTH', [
immich-microservices_1     |           ^
immich-microservices_1     | Error: Invalid Record Length: expect 19, got 4 on line 174673
immich-microservices_1     |     at Object.__onRecord (/usr/src/app/node_modules/csv-parse/dist/cjs/index.cjs:940:11)
immich-microservices_1     |     at Object.parse (/usr/src/app/node_modules/csv-parse/dist/cjs/index.cjs:903:36)
immich-microservices_1     |     at Parser._flush (/usr/src/app/node_modules/csv-parse/dist/cjs/index.cjs:1336:26)
immich-microservices_1     |     at Parser.final [as _final] (node:internal/streams/transform:112:25)
immich-microservices_1     |     at callFinal (node:internal/streams/writable:694:27)
immich-microservices_1     |     at prefinish (node:internal/streams/writable:723:7)
immich-microservices_1     |     at finishMaybe (node:internal/streams/writable:733:5)
immich-microservices_1     |     at afterWrite (node:internal/streams/writable:504:3)
immich-microservices_1     |     at onwrite (node:internal/streams/writable:477:7)
immich-microservices_1     |     at Parser.Transform._read (node:internal/streams/transform:245:5)

I'm trying to find where we call that code. Could it be an issue with the csv data we download? OK, issue confirmed and fixed. Deleting the microservices volume will clear the geoname cache and make it work again If you don't want to delete the volume, just delete the .reverse-geocoding-dump folder in the microservices container

ahbeng•3y ago

cool... thinking of volumizing that onto one server's folder and delete that folder.... (yes, i'm that reckless lol!)

jrasm91•3y ago

Reminds me of this PR, which was to make it an anon volume: https://github.com/immich-app/immich/pull/1234/files One step further is to just map it to a physical folder and then delete the contents when needed. There's no issue with doing that, it's simply a cache folder and the app will work as expected when the file is deleted. The issue, in fact, comes from the fact that is does a fresh/new download every day. If the download gets interrupted and the file becomes incomplete/corrupted it will start causing the microservices container to crash. I think because it is downloading daily for all immich users the chance of somebody having the issue is pretty high. It's an old library and the code is hard to modify/improve, so it's kind of just how it is, although I think there's a pull request upstream that's supposed to try to detect a corrupted download and self-heal. We'll see 😄

ahbeng•3y ago

i wish I can code - but I would say low hanging fruit is to not download daily. can I set it as readonly folder after - once content gets populated once? volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /mnt/immich/micro:/usr/src/app/.reverse-geocoding-dump:ro let see how it goes after this lol redeploy after that folder has all it needs and stable then made it RO

jrasm91•3y ago

I'd guess it errors out, but not 100% sure how it would respond to that.. It's another library that i'm this close 🤏 to just rewriting 🙂

ahbeng•3y ago

yep.. lets see tonight it if works manual band aids for now even if it restarts the container without that hard break reboot loop. I'll be happy

jrasm91•3y ago

It looks for a file based on the date in the name of it.

ahbeng•3y ago

oh.. ouch. so much for that maybe...

jrasm91•3y ago

Basically 2023_03_30.csv or whatever It's terrible

ahbeng•3y ago

basically I got like 500GB of photos that I have yet to upload... so been watching it like a hawk when I upload in small batches. that takes like watching it for weeks. time wasted when it reboot loops while I sleep.

jrasm91•3y ago

Yeah, super annoying. I wonder if you could copy/paste the file with tomorrows date

ahbeng•3y ago

i'm ok with cron scripts... thats an idea

jrasm91•3y ago

You could either do that at midnight utc or just maintain current day + tomorrows day I haven't looked at the download code in awhile, but I think that would work.

ahbeng•3y ago

sooo... docker-compose on this: - /mnt/immich/micro:/usr/src/app/.reverse-geocoding-dump:ro seems to be working... too early to tell. but I left it as it is and recreated the container seems to be holding lemme leave it for another day b4 I declare "semi-victory" for a workaround i didn't rename the files with the new date name

jrasm91•3y ago

Have you uploaded a new images with gps? It only runs reverse geocoding on demand.

ahbeng•3y ago

I just did on a new pic upload. it worked with GPS details. quite honestly this is like really a hacked hack. so I won't technically recommend as I'm just messing around it. for now it won't bootloop so I am happy!

Gaming

Programming

Microservices crash on reverse geocoding

Did you find this page helpful?