Immich•2y ago

500 getaddrinfo ENOTFOUND immich-server

I upgraded via Portainer | Stacks | Editor | "Update the stack", and am now getting this 500 error. Of course I committed the cardinal sin of forgetting to read the release notes for the 2+ "major" versions that I was jumping, and now my _redis and _server containers are bouncing. Things I've tried already: -entering a random value for TYPESENSE_API_KEY in Stack | Environment variables -updating the stack again

9 Replies

DarrenOP•2y ago

Here's a snippet from my redis log:

1:C 07 Aug 2023 20:29:42.392 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 07 Aug 2023 20:29:42.392 # Redis version=6.2.13, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 07 Aug 2023 20:29:42.392 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 07 Aug 2023 20:29:42.393 * monotonic clock: POSIX clock_gettime
1:M 07 Aug 2023 20:29:42.394 * Running mode=standalone, port=6379.
1:M 07 Aug 2023 20:29:42.394 # Server initialized
1:M 07 Aug 2023 20:29:42.394 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 07 Aug 2023 20:29:42.394 # Fatal: Can't initialize Background Jobs.

1:C 07 Aug 2023 20:29:42.392 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 07 Aug 2023 20:29:42.392 # Redis version=6.2.13, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 07 Aug 2023 20:29:42.392 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 07 Aug 2023 20:29:42.393 * monotonic clock: POSIX clock_gettime
1:M 07 Aug 2023 20:29:42.394 * Running mode=standalone, port=6379.
1:M 07 Aug 2023 20:29:42.394 # Server initialized
1:M 07 Aug 2023 20:29:42.394 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 07 Aug 2023 20:29:42.394 # Fatal: Can't initialize Background Jobs.

Here's a snippet from server:

node[7]: ../src/node_platform.cc:68:std::unique_ptr<long unsigned int> node::WorkerThreadsTaskRunner::DelayedTaskScheduler::Start(): Assertion `(0) == (uv_thread_create(t.get(), start_thread, this))' failed.
 1: 0xb83f50 node::Abort() [node]
 2: 0xb83fce  [node]
 3: 0xbf14ee  [node]
 4: 0xbf15d1 node::NodePlatform::NodePlatform(int, v8::TracingController*, v8::PageAllocator*) [node]
 5: 0xb41ed3 node::InitializeOncePerProcess(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, node::ProcessFlags::Flags) [node]
 6: 0xb4256b node::Start(int, char**) [node]
 7: 0x7f09aa7f51ca  [/lib/x86_64-linux-gnu/libc.so.6]
 8: 0x7f09aa7f5285 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
 9: 0xac1fee _start [node]
Aborted (core dumped)

node[7]: ../src/node_platform.cc:68:std::unique_ptr<long unsigned int> node::WorkerThreadsTaskRunner::DelayedTaskScheduler::Start(): Assertion `(0) == (uv_thread_create(t.get(), start_thread, this))' failed.
 1: 0xb83f50 node::Abort() [node]
 2: 0xb83fce  [node]
 3: 0xbf14ee  [node]
 4: 0xbf15d1 node::NodePlatform::NodePlatform(int, v8::TracingController*, v8::PageAllocator*) [node]
 5: 0xb41ed3 node::InitializeOncePerProcess(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, node::ProcessFlags::Flags) [node]
 6: 0xb4256b node::Start(int, char**) [node]
 7: 0x7f09aa7f51ca  [/lib/x86_64-linux-gnu/libc.so.6]
 8: 0x7f09aa7f5285 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
 9: 0xac1fee _start [node]
Aborted (core dumped)

Also, when I try to look at the typesense log, I get Failure: Unable to retrieve container logs

Alex Tran•2y ago

Can you first check the docker-compose file to make sure it is up to date then pull the images and perform docker-compose down and docker-compose up again?

jrasm91•2y ago

Whwat version of docker are you running? Also, what hardware are you running this on?

koolkid•2y ago

Is this what you did?

DarrenOP•2y ago

I double-checked that I'm using the latest docker-compose file. I hadn't been, apparently - the big giveaway was the use of "entrypoint" vs "command". So I updated that, and then forced an update of the stack again, re-pulling and redeploying images. Running Docker 19.03.13 on Ubuntu 5.4.0-150-generic on x86/x64 hardware. After that I still had the same problem, so I stopped and started the stack, and again the same 500 error. Now: the Typesense container is bouncing, and the log seems to be empty (I get "No log line matching the '' filter", which I believe is Portainer not finding anything in the log). The MicroServices container is bouncing (with the same log error as the Server container above, except node[6] rather than node[7]). The Server container is bouncing (same node[6] error as MicroServices).

jrasm91•2y ago

Not 100% sure if it is the issue, but your docker version is quite old. It might be worth updating that (regardless if it is related to the issue or not)

DarrenOP•2y ago

I'll do that and report back. Thanks!

StandDown•2y ago

maybe this helps: stop the stack, so not the containers. And then start the stack so it starts using the dependencies order as set in the docker-compose.yml

DarrenOP•2y ago

Upgraded docker to 24.0.5 seems to have done the trick. Thanks! Is there a way/need to mark this as "solved"?

Gaming

Programming

500 getaddrinfo ENOTFOUND immich-server

Did you find this page helpful?