Immich•2y ago

Starting Typesense is extremely slow

Does anyone else have this issue? When I start the Typesense container, it take around 10-20 minutes to start, and Immich doesn't load until it's completely loaded. Log output when I start container: I20230807 17:46:55.262866 1 typesense_server_utils.cpp:357] Starting Typesense 0.24.1 ... then, much later:

I20230807 18:04:52.545059   543 batched_indexer.cpp:446] Restored 3 in-flight requests from snapshot.
I20230807 18:04:52.545084   543 raft_server.cpp:498] Finished loading collections from disk.
I20230807 18:04:52.616158   543 raft_server.h:273] Configuration of this group is 192.168.160.10:8107:8108
I20230807 18:04:52.835302   543 snapshot_executor.cpp:264] node default_group:192.168.160.20:8107:8108 snapshot_load_done, last_included_index: 104420 last_included_term: 85 peers: "192.168.160.10:8107:8108"
I20230807 18:04:52.940532   505 raft_meta.cpp:521] Loaded single stable meta, path /data/state/meta term 87 votedfor 192.168.160.20:8107:8108 time: 24336
I20230807 18:04:52.940585   505 node.cpp:576] node default_group:192.168.160.20:8107:8108 init, term: 87 last_log_id: (index=106235,term=87) conf: 192.168.160.20:8107:8108 old_conf:
I20230807 18:04:52.940639   505 node.cpp:1549] node default_group:192.168.160.20:8107:8108 term 87 start vote and grant vote self
I20230807 18:04:53.067860   505 raft_meta.cpp:546] Saved single stable meta, path /data/state/meta term 88 votedfor 192.168.160.20:8107:8108 time: 125755
I20230807 18:04:53.067901   505 node.cpp:1783] node default_group:192.168.160.20:8107:8108 term 88 become leader of group 192.168.160.20:8107:8108
I20230807 18:04:53.068109   505 raft_server.cpp:129] Node last_index: 106235
I20230807 18:04:53.068123   505 typesense_server_utils.cpp:306] Typesense peering service is running on 192.168.160.20:8107

I20230807 18:04:52.545059   543 batched_indexer.cpp:446] Restored 3 in-flight requests from snapshot.
I20230807 18:04:52.545084   543 raft_server.cpp:498] Finished loading collections from disk.
I20230807 18:04:52.616158   543 raft_server.h:273] Configuration of this group is 192.168.160.10:8107:8108
I20230807 18:04:52.835302   543 snapshot_executor.cpp:264] node default_group:192.168.160.20:8107:8108 snapshot_load_done, last_included_index: 104420 last_included_term: 85 peers: "192.168.160.10:8107:8108"
I20230807 18:04:52.940532   505 raft_meta.cpp:521] Loaded single stable meta, path /data/state/meta term 87 votedfor 192.168.160.20:8107:8108 time: 24336
I20230807 18:04:52.940585   505 node.cpp:576] node default_group:192.168.160.20:8107:8108 init, term: 87 last_log_id: (index=106235,term=87) conf: 192.168.160.20:8107:8108 old_conf:
I20230807 18:04:52.940639   505 node.cpp:1549] node default_group:192.168.160.20:8107:8108 term 87 start vote and grant vote self
I20230807 18:04:53.067860   505 raft_meta.cpp:546] Saved single stable meta, path /data/state/meta term 88 votedfor 192.168.160.20:8107:8108 time: 125755
I20230807 18:04:53.067901   505 node.cpp:1783] node default_group:192.168.160.20:8107:8108 term 88 become leader of group 192.168.160.20:8107:8108
I20230807 18:04:53.068109   505 raft_server.cpp:129] Node last_index: 106235
I20230807 18:04:53.068123   505 typesense_server_utils.cpp:306] Typesense peering service is running on 192.168.160.20:8107

Is this normal and expected?

21 Replies

sircharloOP•2y ago

Here's the relevant portion of my docker-compose.yml file:

typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    cpus: 0.3
    labels:
      - "com.centurylinklabs.watchtower.enable=false"
    env_file:
      - /immich/.env
    entrypoint: >
      /bin/bash -c "
        dpkg --configure -a &&
        apt-get update &&
        apt-get install -y curl &&
        /opt/typesense-server
      "
    depends_on:
      redis:
        condition: service_started
      database:
        condition: service_started
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl --silent --fail localhost:8108/health || exit 1"
        ]
      interval: 300s
      timeout: 10s
      retries: 10
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
      - TZ=${TZ}
    volumes:
      - /immich/data/tsdata:/data

typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    cpus: 0.3
    labels:
      - "com.centurylinklabs.watchtower.enable=false"
    env_file:
      - /immich/.env
    entrypoint: >
      /bin/bash -c "
        dpkg --configure -a &&
        apt-get update &&
        apt-get install -y curl &&
        /opt/typesense-server
      "
    depends_on:
      redis:
        condition: service_started
      database:
        condition: service_started
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl --silent --fail localhost:8108/health || exit 1"
        ]
      interval: 300s
      timeout: 10s
      retries: 10
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
      - TZ=${TZ}
    volumes:
      - /immich/data/tsdata:/data

Alex Tran•2y ago

How many assets do you have on your instance?

bo0tzz•2y ago

And, just as a curiosity: If you remove the cpu limit, does it still take a long time to start?

Alex Tran•2y ago

good catch

sircharloOP•2y ago

/api/asset/statistics:

{
  "images": 30482,
  "videos": 1219,
  "total": 31701
}

{
  "images": 30482,
  "videos": 1219,
  "total": 31701
}

Not much faster last time I tested, but I'll try again. The thing is, all the other services on this server slow to an absolute crawl if Typesense is left to its own devices and gobbles up my CPU for multiple minutes...

bo0tzz•2y ago

That amount shouldn't cause problems, I think

sircharloOP•2y ago

I removed the CPU limit. I20230807 18:30:44.004426 1 typesense_server_utils.cpp:357] Starting Typesense 0.24.1 I'll post when Typesense shows that it's ready

Alex Tran•2y ago

I have around 27000 assets and it takes about 30 seconds to start up

sircharloOP•2y ago

I20230807 18:32:53.243768 539 raft_server.cpp:498] Finished loading collections from disk. A bit over two minutes. Hm.

Alex Tran•2y ago

How many CPU cores do you allocate for this VM?

sircharloOP•2y ago

its not a vm, just a really old machine lol

Alex Tran•2y ago

well then 2 mins is not too bad 😄

sircharloOP•2y ago

Does this seem normal, even 20 mins after Typesense is started?

sircharloOP•2y ago

It seems like Typesense just never stops reading and writing, even though I'm not scanning anything or adding files to the library

Alex Tran•2y ago

Might not be normal, I haven't seen this on order system usually typesense will need to indexing upon starting up I wonder with the lower CPU spec, it takes a while to finish indexing

sircharloOP•2y ago

Maybe In any case I've disable Typesense for now My poor server can't handle 30 minutes of sustained IO like that In case anyone is interested:

            .-/+oossssoo+/-.               user@my-server
        `:+ssssssssssssssssss+:`           ----------------
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu 20.04.6 LTS x86_64
    .ossssssssssssssssssdMMMNysssso.       Host: Inspiron 545 00
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Kernel: 5.4.0-155-generic
  +ssssssssshmydMMMMMMMNddddyssssssss+     Uptime: 4 days, 13 hours, 58 mins
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Packages: 3534 (dpkg), 4 (snap)
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Shell: bash 5.0.17
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Terminal: /dev/pts/1
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   CPU: Intel Core 2 Quad Q8200 (4) @ 2.331GHz
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   GPU: AMD ATI Radeon HD 4350/4550
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Memory: 3864MiB / 7943MiB
.ssssssssdMMMNhsssssssssshNMMMdssssssss.
 /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/
  +sssssssssdmydMMMMMMMMddddyssssssss+
   /ssssssssssshdmNNNNmyNMMMMhssssss/
    .ossssssssssssssssssdMMMNysssso.
      -+sssssssssssssssssyyyssss+-
        `:+ssssssssssssssssss+:`
            .-/+oossssoo+/-.

            .-/+oossssoo+/-.               user@my-server
        `:+ssssssssssssssssss+:`           ----------------
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu 20.04.6 LTS x86_64
    .ossssssssssssssssssdMMMNysssso.       Host: Inspiron 545 00
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Kernel: 5.4.0-155-generic
  +ssssssssshmydMMMMMMMNddddyssssssss+     Uptime: 4 days, 13 hours, 58 mins
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Packages: 3534 (dpkg), 4 (snap)
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Shell: bash 5.0.17
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Terminal: /dev/pts/1
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   CPU: Intel Core 2 Quad Q8200 (4) @ 2.331GHz
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   GPU: AMD ATI Radeon HD 4350/4550
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Memory: 3864MiB / 7943MiB
.ssssssssdMMMNhsssssssssshNMMMdssssssss.
 /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/
  +sssssssssdmydMMMMMMMMddddyssssssss+
   /ssssssssssshdmNNNNmyNMMMMhssssss/
    .ossssssssssssssssssdMMMNysssso.
      -+sssssssssssssssssyyyssss+-
        `:+ssssssssssssssssss+:`
            .-/+oossssoo+/-.

bo0tzz•2y ago

Might be worth asking the typesense folks about this

sircharloOP•2y ago

I tried using a new Docker volume for the Typesense data. Maybe that will help

Snuupy•2y ago

I have a Celeron N5095 (15w tdp) and it takes about 2 mins with 60k+ assets. 100% CPU the entire time fyi

jrasm91•2y ago

A new volume will require all you assets to be reindexed from postgres, so it makes sense for it to be busy. Subsequent restarts should be much faster.

sogan•2y ago

if it's really old then its cpu may not support avx2 or related instruction sets. building the hnsw index for vectors is cpu-intensive, so this will take longer if it can't accelerate it. as far as i can tell, the N5095 also does not support avx2.

Gaming

Programming

Starting Typesense is extremely slow

Did you find this page helpful?