Railway•6mo ago

Reverse proxy backend is unresponsive

I have a revers proxy through Caddy, that is connecting my frontend and backend. It seems to be working fine for my frontend. But is failing on the backend. The backend is a django service. The proxy is defined by

# Proxy file for connecting services through railway.app
{
    debug
    admin off # theres no need for the admin api in railway's environment
    persist_config off # storage isn't persistent anyway
    auto_https off # railway handles https for us, this could in some cases cause issues if left enabled
    log {
        format json # set runtime log format to json mode
    }
    # server options
    servers {
        trusted_proxies static private_ranges # trust railway's proxy
    }
}

:{$PORT} {
    log {
        format json # set access log format to json mode
    }

    encode zstd gzip

    handle_path /ingest* {
        rewrite * {path}
        reverse_proxy https://us.i.posthog.com:443 {
          header_up Host us.i.posthog.com
          header_down -Access-Control-Allow-Origin
        }
    }

    @django path /admin* /api* /accounts* /static/*
    handle @django {
        reverse_proxy {$BACKEND_HOST}
    }


    handle {
    reverse_proxy {$FRONTEND_HOST}
    }
}

# Proxy file for connecting services through railway.app
{
    debug
    admin off # theres no need for the admin api in railway's environment
    persist_config off # storage isn't persistent anyway
    auto_https off # railway handles https for us, this could in some cases cause issues if left enabled
    log {
        format json # set runtime log format to json mode
    }
    # server options
    servers {
        trusted_proxies static private_ranges # trust railway's proxy
    }
}

:{$PORT} {
    log {
        format json # set access log format to json mode
    }

    encode zstd gzip

    handle_path /ingest* {
        rewrite * {path}
        reverse_proxy https://us.i.posthog.com:443 {
          header_up Host us.i.posthog.com
          header_down -Access-Control-Allow-Origin
        }
    }

    @django path /admin* /api* /accounts* /static/*
    handle @django {
        reverse_proxy {$BACKEND_HOST}
    }


    handle {
    reverse_proxy {$FRONTEND_HOST}
    }
}

I am not getting any responses from django. But I am hitting it. I see some network activity. But no response I put the service in debug mode, but yet, it still is not responding with anything. I am getting 502 errors. The django service is being ran with python manage.py runserver 0.0.0.0:${PORT:-8000}. Unsure what to do / change to get this running

43 Replies

Percy•6mo ago

Project ID: 40943c2e-d9a8-4f3c-b414-73ea73eccd8f

JoshieOP•6mo ago

40943c2e-d9a8-4f3c-b414-73ea73eccd8f

JoshieOP•6mo ago

There are no real logs from django (normally it would respond with what endpoints are hit in debug mode)

JoshieOP•6mo ago

Requests to the backend, end up with an error

JoshieOP•6mo ago

And frontend requests make it through no problem

JoshieOP•6mo ago

My env for the reverse proxy:

BACKEND_HOST=${{Django.RAILWAY_PRIVATE_DOMAIN}}:${{Django.PORT}}
FRONTEND_HOST=${{Frontend.RAILWAY_PRIVATE_DOMAIN}}:${{Frontend.PORT}}
PORT=3000

BACKEND_HOST=${{Django.RAILWAY_PRIVATE_DOMAIN}}:${{Django.PORT}}
FRONTEND_HOST=${{Frontend.RAILWAY_PRIVATE_DOMAIN}}:${{Frontend.PORT}}
PORT=3000

Brody•6mo ago

runserver starts a development server, even if you are only testing you still dont want to be using it on Railway, instead use gunicorn like this gunicorn mysite.wsgi -b [::]:$PORT (replacing mysite with the name of the folder that contains wsgi.py)

JoshieOP•6mo ago

I was originally doing gunicorn BBWare.wsgi -b [::]:${PORT:-8000} but was still getting the same issue. But yea, let me try with that but debug still on (I have my deployment select between the two based on if debug is on)

Brody•6mo ago

little context, the private network is ipv6 only so your app would need to listen on ipv6

JoshieOP•6mo ago

For the run server, yea, I guess it wouldn't be. But the gunicorn should be correct for doing that? Unless there is an extra setting I need to change in the django settings?

Brody•6mo ago

what you have should have worked, maybe you are overwriting the start command somewhere else

JoshieOP•6mo ago

Also, can confirm, just having the gunicorn config file in existence does make it get used. Even if the cli isn't pointing to it. 😄 Going to see though real quick if the gunicorn + debug produces anything

Brody•6mo ago

just set the workers to a fixed number

JoshieOP•6mo ago

No yea, for sure. Was just circling back on that earlier topic Yep, didn't change anything

Brody•6mo ago

show me the depoloy logs for your django app please

JoshieOP•6mo ago

Still get this when I go directly to a backend url

JoshieOP•6mo ago

You reached the start of the range → Jun 14, 2024 1:45 PM
Starting Container
+ python manage.py migrate --noinput
  Apply all migrations: account, admin, auth, authtoken, breaker, contenttypes, permissions, purchase, sale, sessions, sites, socialaccount, stock
+ 
python manage.py collectstatic --noinput
 
+ gunicorn BBWare.wsgi -b [::]:3000
[2024-06-14 17:46:54 +0000] [30] [INFO] Starting gunicorn 22.0.0
[2024-06-14 17:46:54 +0000] [30] [INFO] Listening at: http://[::]:3000 (30)
[2024-06-14 17:46:54 +0000] [30] [INFO] Using worker: sync
[2024-06-14 17:46:54 +0000] [31] [INFO] Booting worker with pid: 31

You reached the start of the range → Jun 14, 2024 1:45 PM
Starting Container
+ python manage.py migrate --noinput
  Apply all migrations: account, admin, auth, authtoken, breaker, contenttypes, permissions, purchase, sale, sessions, sites, socialaccount, stock
+ 
python manage.py collectstatic --noinput
 
+ gunicorn BBWare.wsgi -b [::]:3000
[2024-06-14 17:46:54 +0000] [30] [INFO] Starting gunicorn 22.0.0
[2024-06-14 17:46:54 +0000] [30] [INFO] Listening at: http://[::]:3000 (30)
[2024-06-14 17:46:54 +0000] [30] [INFO] Using worker: sync
[2024-06-14 17:46:54 +0000] [31] [INFO] Booting worker with pid: 31

Brody•6mo ago

can you send the link to the proxy

JoshieOP•6mo ago

https://website-reverse-proxy-production.up.railway.app

Bleacher Breaks Inventory

Inventory Management for Bleacker Breaks.

JoshieOP•6mo ago

Oh lol, I have a typo in the desc :p

Brody•6mo ago

looks like it does make it to the django backend

JoshieOP•6mo ago

You can see the pure frontend working if you go to https://website-reverse-proxy-production.up.railway.app/sign-in

JoshieOP•6mo ago

Works for me. 500s becuase can't hit the auth endpoint becuase backend isn't working.

Brody•6mo ago

can you add --forwarded-allow-ips="*" to your gunicorn start command

JoshieOP•6mo ago

Well the "good" news is that the requests are now responding with 301 and not 502. But not actually sure if that is better. It is redirecting to nothing? Not sure.

JoshieOP•6mo ago

Same result

JoshieOP•6mo ago

Brody•6mo ago

django and or gunicorn is attempting to redirect for some reason, do you have any idea why?

JoshieOP•6mo ago

The only thing I could think of is that I force things to be HTTPS:

SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
SECURE_SSL_REDIRECT = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True

SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
SECURE_SSL_REDIRECT = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True

This would normally work no issue. At least it has with Caddy. But maybe becuase these Caddyfiles have https off, it is messing with things? Idk how railway is taking care of https?

Brody•6mo ago

yeah dont do that railway does https on its own proxy for you, you should run your app in http with no notion of https

JoshieOP•6mo ago

What part should I be getting rid of? But also, there really should be a way to make this work without removing that stuff

Brody•6mo ago

well railway sets a X-Forwarded-Proto header, so you need to have your app accept that value

JoshieOP•6mo ago

It feels very wrong to remove those protections on the django level

Brody•6mo ago

your app on railway can only ever be accessed over https so theres no issues removing them

JoshieOP•6mo ago

I mean sure. But it is more like "extra precaution".

Brody•6mo ago

i promise you its not needed, there is no way to access a railway hosted app over http

JoshieOP•6mo ago

This is going to make for quite the blog post. Eventually. Long list of backlog things to get to before I touch this.

Brody•6mo ago

well you have two options - railway sets a X-Forwarded-Proto header, have your app accept that value - remove anything that tries to redirect to https

JoshieOP•6mo ago

:party_cat: It is working I officially can transfer everything to railway. .... That is going to be a bit of a pain. But it being possible is the first step. Now I just need to also do all the other silly things like add monitoring and add db backup cron stuff. That will be fun

Brody•6mo ago

use the railway template for db backups!

JoshieOP•6mo ago

Yea, I will. Thanks

Brody•6mo ago

no problem!

Gaming

Programming

Reverse proxy backend is unresponsive