Immich•3y ago

Invalid request body

Upon updating from 1.51.1 to 1.51.2 I receive an error message from the web and mobile app which reads "Invalid request body".

89 Replies

EnochPrimeOP•3y ago

Thinking this might be related to #2019 I am not using the provided proxy service.

MicheI•3y ago

Please check for error logs in the server container and your browser/app

bo0tzz•3y ago

~~Is your app also up to date?~~ NVM, you mentioned web too

EnochPrimeOP•3y ago

Web report 400 bad request Don't see anything in the server container logs.

bo0tzz•3y ago

Check the web container

I am not using the provided proxy service.

Did you set up the routing correctly?

EnochPrimeOP•3y ago

Well I did notice there was a change from #2019 and I have yet to adapt that. I'm thinking that is the likely cause at the moment.

bo0tzz•3y ago

That's not a change that should break anything It's just a small improvement

EnochPrimeOP•3y ago

Okay, well the proxy I have was working with 1.51.1

bo0tzz•3y ago

Is the IP of your proxy, as seen by immich-server, one of

loopback - 127.0.0.1/8, ::1/128
linklocal - 169.254.0.0/16, fe80::/10
uniquelocal - 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, fc00::/7

loopback - 127.0.0.1/8, ::1/128
linklocal - 169.254.0.0/16, fe80::/10
uniquelocal - 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, fc00::/7

EnochPrimeOP•3y ago

Would I be able to find that in the logs?

bo0tzz•3y ago

I don't think so, but docker inspect should do it

EnochPrimeOP•3y ago

I run 2 replicas of the webserver with the proxy and I see the address as 10.0.1.239 and 10.0.1.240

MicheI•3y ago

Why do you think #2019 is related? Haven't seen anything that points towards that. Does everything work again if you revert back to v1.51.1? Also, can you please provide more details about your setup (deployment method, version, which proxy, etc.), check if all your containers are running, if they contain error logs and post screenshots of any errors

EnochPrimeOP•3y ago

Was thinking it was related as the default proxy config changed. Everything does work again if I revert to 1.51.1

MicheI•3y ago

But you aren't using Immich's proxy, so I don't see how that's causing issues

EnochPrimeOP•3y ago

Compose file: https://pastebin.com/K3wxPJdb

Pastebin

version: "3.9"# swarm deployment extensionx-deploy-swarm: &deploy_s...

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

EnochPrimeOP•3y ago

I deploy with docker swarm and use the linuxserver.io swag container for the proxy

EnochPrimeOP•3y ago

Proxy config https://pastebin.com/kW7w5hfE

Pastebin

Version 2023/01/04server { listen 443 ssl; listen [::]:443...

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

bo0tzz•3y ago

I thought the

app.set('trust proxy', ['loopback', 'linklocal', 'uniquelocal']);

app.set('trust proxy', ['loopback', 'linklocal', 'uniquelocal']);

Might not include their proxy address Ofc that shouldn't be the case, because the proxy shouldn't be on a public address internally

EnochPrimeOP•3y ago

All the services are running.

bo0tzz•3y ago

If you go to https://demo.immich.app/api/server-info/ping (replace the host with your own, of course), do you get a pong response?

EnochPrimeOP•3y ago

I don't see errors in the logs, except the database has some suspicious messages related to my setup.

{"res":"pong"}

{"res":"pong"}

bo0tzz•3y ago

🤔 @jrasm91 I don't think this has to do with your refactor in https://github.com/immich-app/immich/pull/2038, but can I get your eyes on this anyways just in case?

jrasm91•3y ago

What's the actual error and for what request?

EnochPrimeOP•3y ago

EnochPrimeOP•3y ago

bo0tzz•3y ago

That's an error coming from your proxy, not from Immich So check your proxy logs

jrasm91•3y ago

Can you send a screenshot of the network request from the web?

EnochPrimeOP•3y ago

From the web dev tools console?

jrasm91•3y ago

Yeah

Alex Tran•3y ago

https://immich.app/docs/administration/reverse-proxy @EnochPrime can you try follow the info mentioned in the docs

Reverse Proxy | Immich

When deploying Immich it is important to understand that a reverse proxy is required in front of the server and web container. The reverse proxy acts as an intermediary between the user and container, forwarding requests to the correct container based on the URL path.

jrasm91•3y ago

Like what if the actual response code and did it say what server is sending it.

EnochPrimeOP•3y ago

Unfortunately I cannot send a screenshot at the moment. But it is error code 400

bo0tzz•3y ago

What are the response headers etc?

EnochPrimeOP•3y ago

Connection: keep-alive Content-Length: 20 Date: <as expected> Server: nginx

jrasm91•3y ago

And the response?

bo0tzz•3y ago

"Invalid request body", presumably

jrasm91•3y ago

What url is this for?

EnochPrimeOP•3y ago

bo0tzz•3y ago

@EnochPrime can you pastebin the full logs of * your proxy container * immich-server * immich-web

EnochPrimeOP•3y ago

immich-web

immich_app.1.u11tn6rpblnj@pumpkinpi    | Listening on 0.0.0.0:3000

immich_app.1.u11tn6rpblnj@pumpkinpi    | Listening on 0.0.0.0:3000

EnochPrimeOP•3y ago

immich-server https://pastebin.com/6pADAnRD

Pastebin

immich_server.1.mp33o1xn2dck@node-hp | [32m[Nest] 1 - [39m03/...

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

EnochPrimeOP•3y ago

proxy 1 https://pastebin.com/EWMuaWL6

Pastebin

core_webserver.1.1hx0w6b7gc8b@pumpkinpi | [mod-init] Attempting ...

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

Alex Tran•3y ago

i dont see anything suspicious from the logs

bo0tzz•3y ago

@EnochPrime is it possible to up the log verbosity in your immich proxy config? Ideally so it prints out the full request/response That should immediately spit out what's going on

EnochPrimeOP•3y ago

I think I may found something deeper in the proxy logs, but I'm having difficulty redacting stuff on the phone. Here is a sample

2023/03/23 08:34:53 [error] 419#419: *177 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.2, server: photos.*, request: "GET /api/server-info/ping HTTP/2.0", upstream: "http://10.0.1.156:3001/server-info/ping", host: "photos.<redacted>", referrer: "https://photos.<redacted>/photos"

2023/03/23 08:34:53 [error] 419#419: *177 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.2, server: photos.*, request: "GET /api/server-info/ping HTTP/2.0", upstream: "http://10.0.1.156:3001/server-info/ping", host: "photos.<redacted>", referrer: "https://photos.<redacted>/photos"

2023/03/23 00:26:46 [error] 415#415: *174 immich_server could not be resolved (3: Host not found), client: 10.0.0.2, server: photos.*, request: "GET /api/socket.io/?EIO=4&transport=websocket HTTP/1.1", host: "photos.<redacted>"

2023/03/23 00:26:46 [error] 415#415: *174 immich_server could not be resolved (3: Host not found), client: 10.0.0.2, server: photos.*, request: "GET /api/socket.io/?EIO=4&transport=websocket HTTP/1.1", host: "photos.<redacted>"

jrasm91•3y ago

Seems like the proxy can't connect and proxy to the immich server container

bo0tzz•3y ago

Since the ping endpoint worked, it could be a problem specifically with proxying the websocket connection?

jrasm91•3y ago

The the ping test go through the proxy?

EnochPrimeOP•3y ago

Yes

jrasm91•3y ago

How come the log says connection refused while connecting to upstream with the ping url

EnochPrimeOP•3y ago

Is this possibly related? https://github.com/immich-app/immich/pull/2059

GitHub

chore: bump openapi version to v1.51.2 by michelheusschen · Pull R...

Can't this be automated?

bo0tzz•3y ago

That's just a routine version number change That'll do absolutely nothing to anything meaningful 😛

EnochPrimeOP•3y ago

Is there a way to make the webserver log more verbose?

bo0tzz•3y ago

LOG_LEVEL=verbose for immich-server

jrasm91•3y ago

It seems likely unrelated to immich-server as you get the same error on the mobile app and that is not using the immich-web container at all. The same error in both places would imply it has to be related to your proxy server proxying to the immich-server

bo0tzz•3y ago

(or to immich-server itself, but if that had any errors it would log them)

EnochPrimeOP•3y ago

From the app: https://domain -> invalid request body https://domain/api -> no error Also exposed the webserver port and directly connected without the proxy with issue. Without issue*

bo0tzz•3y ago

In the app, do you have your domain with /api at the end or not?

EnochPrimeOP•3y ago

without

bo0tzz•3y ago

Try with That'll work

EnochPrimeOP•3y ago

With works

bo0tzz•3y ago

So it is some issue with the web container or the proxying to it

EnochPrimeOP•3y ago

Perhaps from my webserver environment?

IMMICH_SERVER_URL: http://server:3001

IMMICH_SERVER_URL: http://server:3001

jrasm91•3y ago

Navigating to the web by ip works?

EnochPrimeOP•3y ago

Web via host IP directly to the webserver container works.

jrasm91•3y ago

like you can login and everything? That implies web -> server is good and mobile -> server is good. Looks like proxy -> web is bad But proxy -> server is good. The mobile app hits the web server if you don't put in the /api, since it makes a request to .well-known/immich

EnochPrimeOP•3y ago

I thought it took me to the page already logged in, but I have to double check. Mobile app, domain name, with /api, logs in without issue Web, direct IP, displays login page, reports invalid login

jrasm91•3y ago

sounds like maybe an issue with the web to server communication then? The happens via your proxy right?

EnochPrimeOP•3y ago

That's what it seems like at the moment. Should this url work? http://192.168.0.132:3000/api/server-info/ping If I roll the webserver back to 1.51.1 then everything seems works. Even with the server and everything else on 1.51.2.

jrasm91•3y ago

There is this change in the web server export PROTOCOL_HEADER=X-Forwarded-Proto (entrypoint.sh) Maybe that without the matching changes in the the proxy is causing an issue

EnochPrimeOP•3y ago

I updated my proxy to include those changes and there was no change to this issue.

jrasm91•3y ago

What changes did you make?

EnochPrimeOP•3y ago

Before: https://pastebin.com/kW7w5hfE After: https://pastebin.com/WB7RUXbd

Pastebin

Version 2023/01/04server { listen 443 ssl; listen [::]:443...

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

Pastebin

Version 2023/01/04map $http_x_forwarded_proto $forwarded_protoco...

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

jrasm91•3y ago

The only change in the web container between 1.51.1 and 1.51.2 is exporting that env variable... hmm Everything works fine when the web container uses 1.51.1?

EnochPrimeOP•3y ago

I haven't extensively tested it. Right now it's a clean slate with no images uploaded. But I can at least get to the webpage, login, and navigate around to all the pages with 1.51.1.

jrasm91•3y ago

Could you manually ssh into 1.51.2 (web) and remove the new export env then restart the web to verify if that is an issue or not?

EnochPrimeOP•3y ago

What is the command to restart?

jrasm91•3y ago

docker restart immich_web or container id or name

EnochPrimeOP•3y ago

Wouldn't restarting the container reapply the env?

Alex Tran•3y ago

I think what Jason mean was to add the same line in the entrypoint.sh on 1.51.1

jrasm91•3y ago

*remove I was saying, run 1.51.2 but remove the new line and just make sure that is what is causing the problem

EnochPrimeOP•3y ago

So... It wasn't as easy as restarting the container, but I overrode the entrypoint with a copy with the line in question commented out. And it was successful I was able to connect via the web and mobile with /api. *Without /api

jrasm91•3y ago

Yeah, maybe @MicheI knows why that would break your setup. I sure have no idea. Web stops working when the new env variable is added.

MicheI•3y ago

It could be because Svelte uses that header for things I didn't anticipate. Based on your nginx config X-Forwarded-Proto will always be https, so it might try to make requests to the server over https. Don't know for certain at all and will look into this tomorrow. We could in the meantime remove export PROTOCOL_HEADER=X-Forwarded-Proto if it's causing issues.

rosso•3y ago

@EnochPrime I hit this exact same issue on upgrade and worked around it by pointing my proxy to immich-proxy (so double proxied) instead of pointing to immich-web and immich-server directly.

MicheI•3y ago

Can't actually reproduce the issue. I suspect you're somehow hitting this https://github.com/sveltejs/kit/blob/becf2b0e630ea5d23070873dfd30cecc07d5e498/packages/adapter-node/src/handler.js#L88-L92 error condition, which could be related to setting PROTOCOL_HEADER. Guess we can just remove it for now, since it's not being used yet.

EnochPrimeOP•3y ago

This seems related as well. https://github.com/sveltejs/kit/issues/5370

GitHub

Invalid request body · Issue #5370 · sveltejs/kit

Describe the bug I upgraded svelte-kit to its latest version today. I configured my project to use the new vite.config.js setup, however now "Invalid request body" is the only response I ...

EnochPrimeOP•3y ago

One commenter said the issue resolved with node 16.15.1 which just happens to be exactly what the web image uses with node:16-apline3.14.

Gaming

Programming

Invalid request body

Did you find this page helpful?