Container can't download install.sh
My setup includes an AWS instance which runs coderd/provisionerd behind a proxy. A desktop on a separate network, which I have physical access to, is running Docker. All three are on a VPN network using tailscale, and I have the AWS instance set as a tailscale 'exit node' for the desktop.
I configured two example templates, the docker, and the docker-code-server examples.
In this configuration, I can access the coder instance from the browser at my public domain. I am able to log in and spin up the templates. I can log into the terminal on them. On the code-server example I can load the remote vs code instance in the browser. All of this works great. However, I noticed the docker example never stars the code-server instance.
Digging a little deeper, the problem is, the docker container that runs on the desktop can't download files over https specifically from
objects.githubusercontent.com. I can't even pull a certificate from it using openssl s_client. (I can ping it though). So the step to download install.sh is failing on the docker example. This only happens from the container on the desktop provisioned by terraform. I can access that site on the machine that runs the docker instances. I can also access it from the same docker container running on my desktop at home. Its only the container running on the desktop that can't. I also can pull a cert and download files from other https sites that I've tried. This behavior is consistent on both templates.
I could just use a docker container with the code-server preinstalled, but this does not work for me since so much of the other tooling required is hosted on github.
Any thoughts or debug tips that could help here?
16 Replies
I would've told you to pre-install it in the image but since that's not an option I'm kind of out of ideas.
I've never used this kind of setup, do you have a clue on what could be blocking the traffic ?
also, does http traffic to http://objects.githubusercontent.com go through ?
Not entirely sure the right way to try it, but doing
wget http://objects.githubusercontent.com/ correctly pulls in the 405 response, so I assume yes
to compare, doing the same thing on the https just hangs (but does correctly resolve the IP)
Actually it even says its connected, like its able to communicate out, but not get a response back in.
Though it never says 'HTTP request sent' so I guess its likely hanging while trying to establish a secure connection? This part is not my strong suit.
Slightly related observation, running the same container on the same server with podman instead of docker works correctly.can you show me your terraform config ?
Which makes me think it would have to be something with docker's networking, and potentially how it was configured with terraform.
yes exactly what i'm thinking
i can't seem to think of a reason why though
So the first time I ran it, I used exactly the default template running
coder templates createI ran the docker template first, which failed to start code-server on the first run. Then I ran the docker-code-server example. Finally, this is my setup that I've run later, which starts from the code-server template. https://github.com/FRC3005/infra-coder-templates/tree/main/frc-javaare you using windows to upload the templates?
Nope, I upload from the aws instance that is running coder.com infra, which is running the AWS Linux AMI
alright
seems fine to me
do some other websites work with https ?
Yes, every one I tried, even https://github.com
could you try using
docker run with the same image and doing the wget inside there ?
since you said it works with podman it might aswell work with default docker settings
it looks like there's proxying of some sort being done thoughSo it fails even with the most basic test, trying it with
docker run -it --rm ubuntu:latest
Alright, I think I figured it out (at least got past one hurdle)!
if I set mtu in docker to some smaller value, it gets further but still not where it needs to be. e.g. setting sudo vim /etc/docker/daemon.json
wget now works, but running openssl s_client hangs half way, so curl also still hangs.
Now, looking at the mtu for various network adapters, I see the physical ethernet ports are 1500, but the tailscale mtu is set to 1280. So if I set the docker mtu to 1280 to match, it now works. No idea why this works on some sites and not others.
For completeness, my ip a output
is this resolved ?
Yes it is. Thanks for the help!
great find, what a weird issue
you can mark the issue as resolved by running
/resolveyeah weird issue for sure, marking complete