Coder.com•2y ago

Coder behind Reverse Proxy and Port Forwarding

Good evening everyone, just a quick question since I'm kinda Stuck: Does Coder Server need any open Ports besides 443 by default? 80 should only be used for Let's Encrypt but since I'm using DNS Challenge it shouldn't matter. Or am I missing something? To clarify, my current setup is as follows: Router with Port Forwarding -> Reverse Proxy -> Coder Server I'm forwarding port 443 currently but I either get certificate errors or bad Gateway. According to my browser the certificate is issued by localhost in case of a certificate error. Caddy Logs show it has succesfully generated new certificates for the domain and wildcard subdomain. Bad Gateway usually happens when I try to generate the certificate on my reverse proxy already, while a straight passthrough gives me the cert error. Probably because it's forwarding using http instead? Hope somebody has an Idea.

68 Replies

Phorcys•2y ago

what port is coder listening to ? I think you should just proxy coder via http and let caddy handle the certs

HannsrOP•2y ago

Coder is listening to 3000 iirc, from the coder docs, then caddy maps it to 443. I did try to proxy via http only, but then I get an invalid Certificate, despite Caddy logs saying it got the requested certificates.

Phorcys•2y ago

can you show me your caddy conf http or https ?

HannsrOP•2y ago

This is the caddy file:

my.domain.dev, *.my.domain.dev {
        # Set this path to your site's directory.
        root * /usr/share/caddy

        # Enable the static file server.
        file_server

        # Another common task is to set up a reverse proxy:
        reverse_proxy localhost:3000

        # Or serve a PHP site through php-fpm:
        # php_fastcgi localhost:9000

  tls {
     dns cloudflare my_cloudflare_token
   }
}
# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

my.domain.dev, *.my.domain.dev {
        # Set this path to your site's directory.
        root * /usr/share/caddy

        # Enable the static file server.
        file_server

        # Another common task is to set up a reverse proxy:
        reverse_proxy localhost:3000

        # Or serve a PHP site through php-fpm:
        # php_fastcgi localhost:9000

  tls {
     dns cloudflare my_cloudflare_token
   }
}
# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

` Just removed the token and domains Now that's a good question, I'd have to check but iirc it is https. In my first test server I blocked all traffic despite 22 and 443 and it worked just fine. So I'm kind of at a loss here

Phorcys•2y ago

here's your issue then this config scares me I don't think it can work

HannsrOP•2y ago

It's with caddy DNS Challenge, it works just like that on my other dev system

Phorcys•2y ago

my.domain.dev, *.my.domain.dev {
  # remember to always specify scheme
  reverse_proxy http://localhost:3000

  tls {
     dns cloudflare my_cloudflare_token
   }
}

my.domain.dev, *.my.domain.dev {
  # remember to always specify scheme
  reverse_proxy http://localhost:3000

  tls {
     dns cloudflare my_cloudflare_token
   }
}

this would work but the file_server is likely what's breaking it all

HannsrOP•2y ago

hm, that's from the coder docs and what is also working on the other system. But I'll try commenting that out

Phorcys•2y ago

can you link me to that docs page ? cause I don't see a way this is working in that case I think the file_server directive takes over your reverse_proxy one

HannsrOP•2y ago

It's linked here: https://coder.com/docs/coder-oss/latest/admin/configure#system-packages to: https://github.com/coder/coder/tree/main/examples/web-server/caddy

GitHub

coder/examples/web-server/caddy at main · coder/coder

A tool that provisions remote development environments via Terraform - coder/examples/web-server/caddy at main · coder/coder

Configuration - Coder OSS main docs

Learn how to configure Coder

Phorcys•2y ago

😶

HannsrOP•2y ago

oh no wait, it's not, lol

Phorcys•2y ago

i think you just pasted it into your example caddy host

HannsrOP•2y ago

Guess it's residue from the base caddyfile then 🫠

Phorcys•2y ago

for the record here's mine

https://*.code.mydomain.tld, https://code.mydomain.tld {
        tls {
                resolvers 1.1.1.1
                dns cloudflare <hello>
        }

        reverse_proxy http://172.23.0.2:7080
}

https://*.code.mydomain.tld, https://code.mydomain.tld {
        tls {
                resolvers 1.1.1.1
                dns cloudflare <hello>
        }

        reverse_proxy http://172.23.0.2:7080
}

btw, if you're using a fresh domain and it won't fetch the certs then resolvers 1.1.1.1 is a nice tip they update almost instantly

HannsrOP•2y ago

that's good to know, thanks. Now let's see if it works now.. Mh, still certificate error

Phorcys•2y ago

can you show me ?

HannsrOP•2y ago

While caddy showing

Dez 22 08:58:20 coder caddy[1226]: {"level":"info","ts":1671695900.1699286,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"my.domain.dev"}

Dez 22 08:58:20 coder caddy[1226]: {"level":"info","ts":1671695900.1699286,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"my.domain.dev"}

HannsrOP•2y ago

Just says

NET::ERR_CERT_AUTHORITY_INVALID

NET::ERR_CERT_AUTHORITY_INVALID

And this is the "certificate":

HannsrOP•2y ago

So it def. doesn't see the proper cert caddy just optained

Phorcys•2y ago

what

HannsrOP•2y ago

yeah, that's what I thought as well 😄

Phorcys•2y ago

what URL are you using to visit your coder deployment ? https://my.domain.dev ?

HannsrOP•2y ago

yeah, but ofc replaces with the proper domain

Phorcys•2y ago

yeah what's your conf now ? like, the whole file

HannsrOP•2y ago

you mean coder or caddy? Either way. Coder:

# Coder must be reachable from an external URL for users and workspaces to connect.
# e.g. https://coder.example.com
#CODER_TUNNEL=true
CODER_ACCESS_URL=https://my.domain.dev
CODER_WILDCARD_ACCESS_URL=*.my.domain.dev
CODER_ADDRESS=127.0.0.1:3000
CODER_PG_CONNECTION_URL=
CODER_TLS_CERT_FILE=
CODER_TLS_ENABLE=
CODER_TLS_KEY_FILE=

# Run "coder server --help" for flag information.

# Coder must be reachable from an external URL for users and workspaces to connect.
# e.g. https://coder.example.com
#CODER_TUNNEL=true
CODER_ACCESS_URL=https://my.domain.dev
CODER_WILDCARD_ACCESS_URL=*.my.domain.dev
CODER_ADDRESS=127.0.0.1:3000
CODER_PG_CONNECTION_URL=
CODER_TLS_CERT_FILE=
CODER_TLS_ENABLE=
CODER_TLS_KEY_FILE=

# Run "coder server --help" for flag information.

Caddy:

# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.

my.domain.dev, *.my.domain.dev {
        # Set this path to your site's directory.
#        root * /usr/share/caddy

        # Enable the static file server.
#        file_server

        # Another common task is to set up a reverse proxy:
        reverse_proxy localhost:3000

        # Or serve a PHP site through php-fpm:
        # php_fastcgi localhost:9000

  tls {
     dns cloudflare my-dns-token
   }
}
# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.

my.domain.dev, *.my.domain.dev {
        # Set this path to your site's directory.
#        root * /usr/share/caddy

        # Enable the static file server.
#        file_server

        # Another common task is to set up a reverse proxy:
        reverse_proxy localhost:3000

        # Or serve a PHP site through php-fpm:
        # php_fastcgi localhost:9000

  tls {
     dns cloudflare my-dns-token
   }
}
# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

` (I'm lazy so all the comments are still in there ^^

Phorcys•2y ago

oh OOOOOOOOOH nervermind i got too hyped

HannsrOP•2y ago

😄

Phorcys•2y ago

try setting only this

CODER_ACCESS_URL=https://my.domain.dev
CODER_WILDCARD_ACCESS_URL=*.my.domain.dev
CODER_ADDRESS=127.0.0.1:3000

CODER_ACCESS_URL=https://my.domain.dev
CODER_WILDCARD_ACCESS_URL=*.my.domain.dev
CODER_ADDRESS=127.0.0.1:3000

my idea here was that you didn't add https at the beginning of the host

https://my.domain.dev, https://*.my.domain.dev {
  reverse_proxy http://localhost:3000

  tls {
     dns cloudflare my-dns-token
   }
}

https://my.domain.dev, https://*.my.domain.dev {
  reverse_proxy http://localhost:3000

  tls {
     dns cloudflare my-dns-token
   }
}

but I think caddy defaults to that anyways I'm thinking maybe since you have env variables related to TLS, coder thinks it should run in HTTP/s

HannsrOP•2y ago

Hm, unchanged. The SSL/Cert error relates to HSTS (full message is in german, so probably no use for you ^^), so I guess it's maybe down to caddy not trusting my proxy? But I'm not sure.

Phorcys•2y ago

nah it only means that the given cert doesn't match your HSTS policy are you sure caddy is the server on 443 ?

HannsrOP•2y ago

waait a second...

Phorcys•2y ago

either that or you're getting firewalled

HannsrOP•2y ago

All I see is

caddy    1352 caddy    3u  IPv4  16934      0t0  TCP 127.0.0.1:2019 (LISTEN)
caddy    1352 caddy    7u  IPv6  16938      0t0  TCP *:443 (LISTEN)
caddy    1352 caddy    9u  IPv6  16940      0t0  TCP *:80 (LISTEN)

caddy    1352 caddy    3u  IPv4  16934      0t0  TCP 127.0.0.1:2019 (LISTEN)
caddy    1352 caddy    7u  IPv6  16938      0t0  TCP *:443 (LISTEN)
caddy    1352 caddy    9u  IPv6  16940      0t0  TCP *:80 (LISTEN)

` No IPv4 listening...

Phorcys•2y ago

same here though

COMMAND     PID  USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
caddy   3377485 caddy    9u  IPv6 469830924      0t0  TCP *:http (LISTEN)
caddy   3377485 caddy   22u  IPv4 720526378      0t0  TCP 172.20.0.1:46138->172.20.0.2:http (ESTABLISHED)
caddy   3377485 caddy   58u  IPv4 720513421      0t0  TCP 172.20.0.1:46140->172.20.0.2:http (ESTABLISHED)
caddy   3377485 caddy   75u  IPv4 723012853      0t0  TCP 172.20.0.1:46270->172.20.0.2:http (ESTABLISHED)
caddy   3377485 caddy   77u  IPv4 723086708      0t0  TCP 172.20.0.1:46296->172.20.0.2:http (ESTABLISHED)

COMMAND     PID  USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
caddy   3377485 caddy    9u  IPv6 469830924      0t0  TCP *:http (LISTEN)
caddy   3377485 caddy   22u  IPv4 720526378      0t0  TCP 172.20.0.1:46138->172.20.0.2:http (ESTABLISHED)
caddy   3377485 caddy   58u  IPv4 720513421      0t0  TCP 172.20.0.1:46140->172.20.0.2:http (ESTABLISHED)
caddy   3377485 caddy   75u  IPv4 723012853      0t0  TCP 172.20.0.1:46270->172.20.0.2:http (ESTABLISHED)
caddy   3377485 caddy   77u  IPv4 723086708      0t0  TCP 172.20.0.1:46296->172.20.0.2:http (ESTABLISHED)

HannsrOP•2y ago

hm yeah it's the same on the other system as well

Phorcys•2y ago

I think even if it's listening to [::]:80 that's fine can you try creating a syntax error in your caddy conf and restart caddy ? i'm not even sure caddy is loading your config at this point

HannsrOP•2y ago

Nope, instant error when trying to restart with a syntax error:

Job for caddy.service failed because the control process exited with error code.

Job for caddy.service failed because the control process exited with error code.

It does list my domains in the log as well as it get's the CF certificates according to the logs.

Phorcys•2y ago

idk wtf is going on then what's the full caddy logs (with no syntax error) ?

HannsrOP•2y ago

This is after a restart

Dez 22 09:39:56 coder caddy[1438]: {"level":"info","ts":1671698396.98281,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Dez 22 09:39:56 coder caddy[1438]: {"level":"info","ts":1671698396.9828575,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
Dez 22 09:39:56 coder caddy[1438]: {"level":"info","ts":1671698396.9828773,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Dez 22 09:39:56 coder caddy[1438]: {"level":"info","ts":1671698396.9829097,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Dez 22 09:39:56 coder caddy[1438]: {"level":"info","ts":1671698396.9829347,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Dez 22 09:39:56 coder caddy[1438]: {"level":"info","ts":1671698396.9829416,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["my.domain.dev","*.my.domain.dev"]}
Dez 22 09:39:56 coder caddy[1438]: {"level":"info","ts":1671698396.9833262,"logger":"tls","msg":"finished cleaning storage units"}
Dez 22 09:39:56 coder caddy[1438]: {"level":"info","ts":1671698396.9838738,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Dez 22 09:39:56 coder caddy[1438]: {"level":"info","ts":1671698396.9838982,"msg":"serving initial configuration"}
Dez 22 09:39:56 coder systemd[1]: Started Caddy.

Dez 22 09:39:56 coder caddy[1438]: {"level":"info","ts":1671698396.98281,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Dez 22 09:39:56 coder caddy[1438]: {"level":"info","ts":1671698396.9828575,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
Dez 22 09:39:56 coder caddy[1438]: {"level":"info","ts":1671698396.9828773,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Dez 22 09:39:56 coder caddy[1438]: {"level":"info","ts":1671698396.9829097,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Dez 22 09:39:56 coder caddy[1438]: {"level":"info","ts":1671698396.9829347,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Dez 22 09:39:56 coder caddy[1438]: {"level":"info","ts":1671698396.9829416,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["my.domain.dev","*.my.domain.dev"]}
Dez 22 09:39:56 coder caddy[1438]: {"level":"info","ts":1671698396.9833262,"logger":"tls","msg":"finished cleaning storage units"}
Dez 22 09:39:56 coder caddy[1438]: {"level":"info","ts":1671698396.9838738,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Dez 22 09:39:56 coder caddy[1438]: {"level":"info","ts":1671698396.9838982,"msg":"serving initial configuration"}
Dez 22 09:39:56 coder systemd[1]: Started Caddy.

Phorcys•2y ago

what's in /var/lib/caddy/.config/caddy/autosave.json?

HannsrOP•2y ago

cat /var/lib/caddy/.config/caddy/autosave.json
{"apps":{"http":{"servers":{"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"localhost:3000"}]}]}]}],"match":[{"host":["my.domain.dev","*.my.domain.dev"]}],"terminal":true}]}}},"tls":{"automation":{"policies":[{"issuers":[{"challenges":{"dns":{"provider":{"api_token":"my_cf_token","name":"cloudflare"}}},"module":"acme"},{"challenges":{"dns":{"provider":{"api_token":"my_CF_token","name":"cloudflare"}}},"module":"zerossl"}],"subjects":["*.my.domain.dev","my.domain.dev"]}]}}}}

cat /var/lib/caddy/.config/caddy/autosave.json
{"apps":{"http":{"servers":{"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"localhost:3000"}]}]}]}],"match":[{"host":["my.domain.dev","*.my.domain.dev"]}],"terminal":true}]}}},"tls":{"automation":{"policies":[{"issuers":[{"challenges":{"dns":{"provider":{"api_token":"my_cf_token","name":"cloudflare"}}},"module":"acme"},{"challenges":{"dns":{"provider":{"api_token":"my_CF_token","name":"cloudflare"}}},"module":"zerossl"}],"subjects":["*.my.domain.dev","my.domain.dev"]}]}}}}

Phorcys•2y ago

how i don't get it

HannsrOP•2y ago

Me neither... it's just so weird

Phorcys•2y ago

are your DNS records pointing to the right server ? actually, you can check that easily by checking if the error still occurs when you stop caddy make sure to ctrl+shift+r

HannsrOP•2y ago

Yeah I doublechecked Cloudflare. But wait a minute, let me clear the CF cache, I did not try that yet. who knows

Phorcys•2y ago

are you proxying via CF ? meh, doesn't matter

HannsrOP•2y ago

nah, not yet. Well, same error with caddy stopped. So it has to be nginx... at least in my mind

Phorcys•2y ago

wait hold on are you reverse proxying a reverse proxy

HannsrOP•2y ago

It is behind a nginx reverse proxy, yes. It's in the title even 😄

Phorcys•2y ago

well no cause I thought the "reverse proxy" was caddy port forwarding is just letting some port open so I assumed there was only one web server

HannsrOP•2y ago

oh well ok, I see how that can be misunderstood

Phorcys•2y ago

and you never specified nginx so i didn't get it okay so show me your nginx conf then well yeah no that defo can't work you have to let nginx handle the certs

HannsrOP•2y ago

So basically point nginx to :3000 on coder and let it get the certs

Phorcys•2y ago

yes or use caddy instead of nginx because that's less of a hassle I've already tried forwarding https-enabled hosts (while letting their cert through) and to my knowledge it isn't possible

HannsrOP•2y ago

Haven't done much with caddy tbh, this is the first time, usually I just use nginx.

Phorcys•2y ago

you'd need to have nginx act as a TCP proxy while still being able to see the host, notice the issue here ? how would it know what server to ship the request to without being able to read the request

HannsrOP•2y ago

yeah, true

Phorcys•2y ago

well, you actually made it clearer for me too as to why I couldn't do that last time idk caddy is less hassle on the eyes imo

HannsrOP•2y ago

I'm back to bad gateway now btw 😄 Pointing nginx to :3000, no matter if http or https

Phorcys•2y ago

but you can't have global configs, so it might be annoying you can't just define headers for all sites neither can you use include hmm can you show me your nginx conf then

HannsrOP•2y ago

oh wait a sec, I have it only listen on localhost:3000 Scratch that, I don't ^^ Oh F me, I got it It was really listening to itself on :3000. So I changed it to

CODER_ADDRESS=0.0.0.0:3000

CODER_ADDRESS=0.0.0.0:3000

in the coder.env file and now it works

Phorcys•2y ago

yay!!!

HannsrOP•2y ago

Now don't forget to disable caddy 😄 Thank you for the time. This gave me a headache for far to long ^^

Codercord•2y ago

Marked the thread as resolved. Marked the thread as unresolved.

Phorcys•2y ago

rly

HannsrOP•2y ago

I guess this was resolved too early …🫠
I can login just fine and create a user, basically do everything in the WebUI, yer when I run

coder login my.domain.dev

coder login my.domain.dev

I get

Failed to check server "https://my.domain.dev" for first user, is the URL correct and is coder accessible from your browser? Error - has initial user: do: Get "https://my.domain.dev/api/v2/users/first": x509: certificate is valid for OPNsense.localdomain, not my.domain.dev

Failed to check server "https://my.domain.dev" for first user, is the URL correct and is coder accessible from your browser? Error - has initial user: do: Get "https://my.domain.dev/api/v2/users/first": x509: certificate is valid for OPNsense.localdomain, not my.domain.dev

So it hit's my firewall somehow? It shouldn't do that, unless I missed something... Opening https://my.domain.dev/api/v2/buildinfo opens the correct page I assume. At least the output is

{"external_url":"https://github.com/coder/coder/commit/86257ce7fcc2b695b6700b5983eb40d0ef0ab193","version":"v0.13.5+86257ce"}

{"external_url":"https://github.com/coder/coder/commit/86257ce7fcc2b695b6700b5983eb40d0ef0ab193","version":"v0.13.5+86257ce"}

Same goes for /api/v2/users/first which returns

{"message":"The initial user has already been created!"}

{"message":"The initial user has already been created!"}

Ok wait, I think I know what this is. I can login from my local system so probably down to hairpin nat or something like that. Edit: Yuuup, Hairpin NAT issues. I even answered a related question in this discord just recently. 🤦‍♂️

Codercord•2y ago

Marked the thread as resolved.

Gaming

Programming

Coder behind Reverse Proxy and Port Forwarding