immich_postgres exited with code 0/135
I messed up after the null-checksum issue in 1.58. I was unable to run the fix for it, so I decided to just restart everyting.
I have used the docker compose down -v command to purge everything.
I have updated my OS (apt update, apt upgrade).
I have rebooted my computer.
I have made sure that I have the newest docker-compose.yml.
I have made sure that I have the newest .env file.
I have done all this multiple times.
I have kept my upload folder as is, hoping that it can be re-used by the new installation.
I'm running Ubuntu 22.04.
Still, after running docker compose up, I keep getting weird errors. I am a complete newbie with docker, so I haven't been able to figure out what's wrong, but I believe it has to do with the database.
I keep seeing these messages in the log (when running docker compose up without -d):
immich_postgres exited with code 0
immich_postgres exited with code 135
(They repeat regularly, sometimes with code 0, sometimes with code 135.)
And then immich_microservices prints this, but I assume that's because the database isn't running:
immich_microservices | Error: connect ETIMEDOUT
immich_microservices | at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
immich_microservices | at Object.onceWrapper (node:events:627:28)
immich_microservices | at Socket.emit (node:events:513:28)
immich_microservices | at Socket._onTimeout (node:net:570:8)
immich_microservices | at listOnTimeout (node:internal/timers:569:17)
immich_microservices | at process.processTimers (node:internal/timers:512:7) {
immich_microservices | errorno: 'ETIMEDOUT',
immich_microservices | code: 'ETIMEDOUT',
immich_microservices | syscall: 'connect'
immich_microservices | }
And immich-server:
immich_server | [Nest] 1 - 06/10/2023, 6:30:32 PM ERROR [TypeOrmModule] Unable to connect to the database. Retrying (7)...
Any pointers would be very much appreciated!
51 Replies
Just to be sure, I tried making a new upload directory (completely empty), but I still get the same.
docker logs immich_postgres is empty.
Can you provide your .env and docker-compose setup
if you want to start clean I suggest starting from here https://immich.app/docs/install/docker-compose
Docker Compose [Recommended] | Immich
Docker Compose is the recommended method to run Immich in production. Below are the steps to deploy Immich with Docker Compose.
Sure!
That's the link I've been using when starting clean. And I have tried that three times now, but I always get this error, so I believe there must be some remnants from my old setup somewhere?
Here are my files:
.env:
docker-compose:
The latter is an exact copy of the one from https://immich.app/docs/install/docker-compose, while the .env file has custom values on UPLOAD_LOCATION and TYPESENSE_API_KEY.
To be completely sure, I just tried running the docker-compose tutorial from a completely different folder on my server, with a completely different upload_location. Still the same! :/
Here is the full output the first time I run docker compose up:
Are there any other logs I can extract? For example from the database container, as it looks like that's the root of the problem? As mentioned, docker logs immich_postgres seems to be completely empty, but perhaps there's more somewhere else?
Also, just to be completely clear: I'm more than happy to start completely fresh -- it just looks like I'm unable to. I can easily re-upload the photos from my phone, so that's not a concern. I just want my immich back!
Hello, sorry for the late reply you can purge data by following this guide https://immich.app/docs/FAQ#how-can-i-purge-data-from-immich
FAQ | Immich
What is the difference between the cloud icon on the mobile app?
How ever before doing that, what happen if you perform docker compose down then up?
No worries, better late than never! (And given that this is free support for a free system, I wouldn't even call it "late" š .)
But unfortunately, that FAQ item is what I meant when I wrote that I had purged everything. So I have already done that -- multiple times. And tried to start anew with the docker-compose tutorial, also multiple times, both from different locations on my server, and with different upload locations. But still, the postgres container keeps quitting with error code 0 or 135.
(And docker compose down then up too, doesn't change anything.)
So, as mentioned, I am a complete docker-newbie, but when I have used down -v, and even changed the location from where I run docker compose, I don't understand how anything from my old installation is left. And maybe it isn't, but then something else must have changed, because I have been running immich for months without issue.
So in summary, based on the log when purging then re-installing and using the up-command (the file attached above), it looks to me that there's something wrong with the postgres container. But with docker logs immich_postgres being empty, I am unable to find any information about it. So I think that's the most important question: are there any other places I can look for logs?
When you use docker compose down -v it removes all the persistent data aka your database
Are you on a pi? Does the hello world docker image work fine?
Yes, that was my understanding too. So maybe the problem isn't remnants from the previous installation. But what is it then?
No, not on a pi. Ubuntu 22.04 on an Intel NUC. Other docker images work fine, and immich did too, until I messed up with the null-checksum issue in 1.58, and had to purge and restart. After that, I keep getting the errors shown above.
Can you verify if you can load v1.57 or 1.58 currently?
I think that will be helpful in figuring out if it's something that might have changed in a recent image, or if there is a setup issue/configuration issue with your machine/installation.
Sure thing! How do I do that?
If you use the latest docker compose file there is an env to specify a tag to you.
IMMICH_VERSION=v1.58.0 for example
In the docker-compose file you should see something like this:
image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
It defaults to release, but there should be tags for each version still available to download and use.Ok, I did that now. But just a quick question: shouldn't that make docker download new containers? It is a bit suspicious if it just starts containers immediately after I changed IMMICH_VERSION, isn't it?
Unless you already had those images downloaded on your machine
Oh, okay. That could be -- I have used all immich versions since around Summer last year.
The release tag points to an image identifier. Your previously used
release tag would match the current id used by the v1.58.0 tag.So I'm running with v.1.58.0 now, and see the same issue. (immich_postgres exited with code 135). Any way to confirm that I'm actually running the 1.58 version from the log output of docker compose up?
Changed it to 1.57 now. It's pulling containers this time.
The thing that is strange is the postgres container is just the official postgres image, it's not an immich specific one
image: postgres:14Are there other ways of getting more logs than docker logs immich_postgres?
If postgres is exiting with an error, that seems to imply something outside of immich is causing problems.
If 1.57 used to work and now doesn't that also seems to imply something else is going on.
Still pulling 1.57, but 1.58 gave the same errors, yes. I'll let you know once 1.57 is done, looks like my internet is slow today.
You aren't running dropbox or some of file watching program in your UPLOAD_FOLDER are you?
Er, maybe not upload folder, but the postgres volume mount?
I am running Nextcloud on the same server, but I don't think it knows anything about the immich folders. It's located in its own folders. But I don't actually know what/where the postgres volume mount is, so I can't say that for sure?
Okay, 1.57 just finished. Similar, but not identical messages. Does still include immich_postgres exited with code 135, but it starts with something else from immich_microservices:
immich_microservices | /usr/src/app/start-microservices.sh:3
immich_microservices | if [ "$DB_HOSTNAME_FILE" ]; then
immich_microservices | ^
immich_microservices |
immich_microservices | SyntaxError: Unexpected token '['
immich_microservices | at internalCompileFunction (node:internal/vm:73:18)
immich_microservices | at wrapSafe (node:internal/modules/cjs/loader:1176:20)
immich_microservices | at Module._compile (node:internal/modules/cjs/loader:1218:27)
immich_microservices | at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
immich_microservices | at Module.load (node:internal/modules/cjs/loader:1117:32)
immich_microservices | at Module._load (node:internal/modules/cjs/loader:958:12)
immich_microservices | at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
immich_microservices | at node:internal/main/run_main_module:23:47
immich_microservices |
immich_microservices | Node.js v18.16.0
More log attached.
Can you perform,
docker compose down then docker compose pull and finally docker compose up?Well, that $DB_HOSTNAME_FILE is basically a breaking change we had in docker compose, that you'd have to revert for it to work again.
The other thing that might be worthwhile trying it so delete the postgres image and trying running it by itself to see if it can come up and load successfully.
Like this should work:
I have to leave the computer for a while now, but I'll try all of this later! Thanks a lot for your suggestions!
You could always try deleting the postgres image and re-pulling it too
No change, I'm afraid. But I now realized that it's not only postgres, it's a lot of the containers! (And it was before too, I found the same in the log I posted yesterday.)
immich_web | Listening on 0.0.0.0:3000
immich_postgres exited with code 0
immich_machine_learning exited with code 0
immich_redis exited with code 0
immich_proxy | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
[7 more proxy rows ...]
immich_proxy | /docker-entrypoint.sh: Configuration complete; ready for start up
immich_postgres exited with code 0
immich_redis exited with code 0
immich_machine_learning exited with code 0
immich_proxy exited with code 0
[5 more proxy rows ...]
immich_proxy | /docker-entrypoint.sh: Configuration complete; ready for start up
immich_postgres exited with code 135
immich_redis exited with code 135
immich_proxy exited with code 135
immich_machine_learning exited with code 135
This seems to work, but there is no output except for a long hash (container ID?). Any way I can check that it actually works?
Docker logs some-postgres
Completely empty. :/
(base) erlend@erlendserver:~/immich-docker$ sudo docker logs some-postgres
(base) erlend@erlendserver:~/immich-docker$
docker ps -a
Does it show that it exit-ed with an error?
Oh, yes it does!
55c25f0c6f7e postgres:14 "docker-entrypoint.sā¦" 9 hours ago Exited (135) 9 hours ago some-postgres
How do I do this?
Ah, docker image rm, I suppose. I'll do that with all of them, and try again
You can try with just postgres until it is working
I mean you can delete everything, but I'd just pull and run postgres until you get that container working.
Yeah, agreed. Still working on deleting everything related to immich. Turns out I had a lot of them, both from altran1502/immich-* and some ghcr.io/immich-*. And the postgres one, of course.
Should free up a bunch of space š
Could try removing an re-installing docker too if you're not making any progress.
Will do, thanks! I'll let you know how it goes a little later.
Alright! Looks like we're getting somewhere now. The some-postgres command now works. immich_postgres, and most of the immich containers also work. But immich_proxy and immich_machine_learning are now restarting with error code 135, and I cannot access the immich gui at all (as a consequence of that, I guess).
The immich_machine_learning log is empty, while the immich_proxy log looks like this:
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-set-env-variables.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-set-env-variables.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
So it's basically just repeating the bolded section over and over again.
Does that tell you anything, and can that also be the cause of immich_machine_learning restarting, or is all of this caused by whatever caused postgres:14 to fail earlier?
The bolded stuff looks like normal start up logs for nginx. With the exception of the container exiting unexpectantly everything looks normal.
If deleting the postgres image and then re-pulling it magically fixed the issue that seems very strange. Is there a chance you have a failing harddrive or something?
Crap, hope not. Will try to check that out.
Alright, looks like I got it back up now. SMART tests showed nothing wrong with the harddrive, so I redid all the image removals, and reset immich once again. Looks like that did the trick. It's possible that I missed some containers or images the last time. So my (on very thin ice) conclusion is that there was something wrong with one or more of my images.
So, again, thank you so much for all your help, guys! I would never have figured this out without all the help.
Do you think it would be helpful if I did a write-up of how I solved it (for an issue/wiki/faq/whatever)? Or is this such an edge case that it's not relevant for others anyway?
You could definitely do a write on github discussions saying exit code 135 or whatever it was could be fixed by deleting the images and re-pulling them. Hopefully nobody else runs into it š¤. This is the first time in almost a year I have seen this issue by someone in immich
That, and the fact that I couldn't find any reference to that error code online still has me wondering what the problem is lol
Yeah, it's extremely weird. Hope it's gone for good now, and let's cross our fingers it doesn't happen to anybody else.
I'll do a write-up on github discussions tomorrow, describing the symptoms and how to fix it.
Or how to maybe fix it, that is. š
thank you
Lol nice!
Done. Please let me know if I used the wrong tag or location or need to fix anything!
https://github.com/immich-app/immich/discussions/2760
GitHub
How to fix immich_[container] exited with code 135 (and/or code 0) ...
After struggling with an Immich update, I probably corrupted my containers somehow, and kept getting logs like this when I ran docker compose up: immich_postgres exited with code 0 immich_machine_l...