C
Coder.comā€¢3y ago
Turtle Kayak

Failed to accept WebSocket connection WebSocket protocol violation

I am overhauling our Coder server deployment. It is in a private AWS subnet with an AWS client VPN endpoint allowing my laptop to connect to the subnet. I can use the HTTP API (via simple commands like workspaces list in the CLI) and access the web dashboard. However when I run the template create command I get a weird networking error. This new deployment of Coder is much more locked down and only allows traffic on the port in the access URL. Additionally now the Coder server is behind an Nginx reverse proxy. Here is the full terminal output: 
$ coder template create aws-linux --directory ./aws-linux/terraform --verbose
> Create and upload "aws-linux/terraform"? (yes/no) yes
ā§—  Queued 
WebSocket protocol violation: handshake request must be at least HTTP/1.1: "HTTP/1.0"                                                                                              
{"message":"Failed to accept websocket.","detail":"failed to accept WebSocket connection: WebSocket protocol violation: handshake request must be at least HTTP/1.1: \"HTTP/1.0\""}
.                                                                                                                                                                                  
logs: unexpected status code 426: WebSocket protocol violation: handshake request must be at least HTTP/1.1: "HTTP/1.0"                                                            
{"message":"Failed to accept websocket.","detail":"failed to accept WebSocket connection: WebSocket protocol violation: handshake request must be at least HTTP/1.1: \"HTTP/1.0\""}
                                                                                                                                                                                   
Run 'coder templates create --help' for usage.
$ coder template create aws-linux --directory ./aws-linux/terraform --verbose
> Create and upload "aws-linux/terraform"? (yes/no) yes
ā§—  Queued 
WebSocket protocol violation: handshake request must be at least HTTP/1.1: "HTTP/1.0"                                                                                              
{"message":"Failed to accept websocket.","detail":"failed to accept WebSocket connection: WebSocket protocol violation: handshake request must be at least HTTP/1.1: \"HTTP/1.0\""}
.                                                                                                                                                                                  
logs: unexpected status code 426: WebSocket protocol violation: handshake request must be at least HTTP/1.1: "HTTP/1.0"                                                            
{"message":"Failed to accept websocket.","detail":"failed to accept WebSocket connection: WebSocket protocol violation: handshake request must be at least HTTP/1.1: \"HTTP/1.0\""}
                                                                                                                                                                                   
Run 'coder templates create --help' for usage.
22 Replies
Turtle Kayak
Turtle KayakOPā€¢3y ago
I'm suspecting one of two causes: - I need to allow network traffic through on another port besides the one in the access URL - I need to configure Nginx to forward specific headers through to the Coder server
thy
thyā€¢3y ago
Turtle Kayak
Turtle KayakOPā€¢3y ago
Ayo I bet so Unfortunately that was not it Here's my Nginx conf 
server {
    listen 80;
    server_name coder-server.dev.mainstay;

    location / {
        proxy_pass http://localhost:3000;

        # Required to make server communication work correctly: https://discord.com/channels/747933592273027093/1007075267119947807/1007314775493070978
        proxy_pass_request_headers on;
        
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Accept-Encoding gzip;
    }
}
server {
    listen 80;
    server_name coder-server.dev.mainstay;

    location / {
        proxy_pass http://localhost:3000;

        # Required to make server communication work correctly: https://discord.com/channels/747933592273027093/1007075267119947807/1007314775493070978
        proxy_pass_request_headers on;
        
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Accept-Encoding gzip;
    }
}
Here's the Nginx access log: 
10.1.1.42 - - [19/Nov/2022:00:48:40 +0000] "GET /api/v2/users/me/organizations HTTP/1.1" 502 166 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:48:40 +0000] "GET /api/v2/organizations/00000000-0000-0000-0000-000000000000/templates/aws-linux HTTP/1.1" 502 166 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:48:42 +0000] "POST /api/v2/files HTTP/1.1" 502 166 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:49:08 +0000] "GET /api/v2/buildinfo HTTP/1.1" 200 125 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:49:08 +0000] "GET /api/v2/users/me/organizations HTTP/1.1" 200 148 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:49:08 +0000] "GET /api/v2/organizations/0e6409db-4460-4af1-b2d7-5fe49aa1e054/templates/aws-linux HTTP/1.1" 404 76 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:49:09 +0000] "POST /api/v2/files HTTP/1.1" 200 76 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:49:09 +0000] "POST /api/v2/organizations/0e6409db-4460-4af1-b2d7-5fe49aa1e054/templateversions HTTP/1.1" 201 559 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:49:09 +0000] "GET /api/v2/templateversions/1e66ab1f-85d6-4790-a095-cb41c179721d HTTP/1.1" 200 559 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:49:10 +0000] "GET /api/v2/templateversions/1e66ab1f-85d6-4790-a095-cb41c179721d/logs?follow&after=1668818949777 HTTP/1.1" 426 266 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:50:11 +0000] "GET /api/v2/buildinfo HTTP/1.1" 200 125 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:50:11 +0000] "GET /api/v2/users/me/organizations HTTP/1.1" 200 148 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:50:11 +0000] "GET /api/v2/organizations/0e6409db-4460-4af1-b2d7-5fe49aa1e054/templates/aws-linux HTTP/1.1" 404 76 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:50:12 +0000] "POST /api/v2/files HTTP/1.1" 200 76 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:50:12 +0000] "POST /api/v2/organizations/0e6409db-4460-4af1-b2d7-5fe49aa1e054/templateversions HTTP/1.1" 201 556 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:50:12 +0000] "GET /api/v2/templateversions/d3395370-b8d8-4a41-97de-7b5a9b21270d HTTP/1.1" 200 556 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:50:12 +0000] "GET /api/v2/templateversions/d3395370-b8d8-4a41-97de-7b5a9b21270d/logs?follow&after=1668819012307 HTTP/1.1" 426 266 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:48:40 +0000] "GET /api/v2/users/me/organizations HTTP/1.1" 502 166 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:48:40 +0000] "GET /api/v2/organizations/00000000-0000-0000-0000-000000000000/templates/aws-linux HTTP/1.1" 502 166 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:48:42 +0000] "POST /api/v2/files HTTP/1.1" 502 166 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:49:08 +0000] "GET /api/v2/buildinfo HTTP/1.1" 200 125 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:49:08 +0000] "GET /api/v2/users/me/organizations HTTP/1.1" 200 148 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:49:08 +0000] "GET /api/v2/organizations/0e6409db-4460-4af1-b2d7-5fe49aa1e054/templates/aws-linux HTTP/1.1" 404 76 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:49:09 +0000] "POST /api/v2/files HTTP/1.1" 200 76 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:49:09 +0000] "POST /api/v2/organizations/0e6409db-4460-4af1-b2d7-5fe49aa1e054/templateversions HTTP/1.1" 201 559 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:49:09 +0000] "GET /api/v2/templateversions/1e66ab1f-85d6-4790-a095-cb41c179721d HTTP/1.1" 200 559 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:49:10 +0000] "GET /api/v2/templateversions/1e66ab1f-85d6-4790-a095-cb41c179721d/logs?follow&after=1668818949777 HTTP/1.1" 426 266 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:50:11 +0000] "GET /api/v2/buildinfo HTTP/1.1" 200 125 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:50:11 +0000] "GET /api/v2/users/me/organizations HTTP/1.1" 200 148 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:50:11 +0000] "GET /api/v2/organizations/0e6409db-4460-4af1-b2d7-5fe49aa1e054/templates/aws-linux HTTP/1.1" 404 76 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:50:12 +0000] "POST /api/v2/files HTTP/1.1" 200 76 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:50:12 +0000] "POST /api/v2/organizations/0e6409db-4460-4af1-b2d7-5fe49aa1e054/templateversions HTTP/1.1" 201 556 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:50:12 +0000] "GET /api/v2/templateversions/d3395370-b8d8-4a41-97de-7b5a9b21270d HTTP/1.1" 200 556 "-" "Go-http-client/1.1"
10.1.1.42 - - [19/Nov/2022:00:50:12 +0000] "GET /api/v2/templateversions/d3395370-b8d8-4a41-97de-7b5a9b21270d/logs?follow&after=1668819012307 HTTP/1.1" 426 266 "-" "Go-http-client/1.1"
Here are the logs from coder 
Nov 19 00:51:17 ip-10-1-1-211 coder-aws-start.sh[1491]: 2022-11-19 00:51:17.003 [INFO]        <./provisionerd/provisionerd.go:219>        (*Server).acquireJob        acquired job        {"initiator_username": "noah", "provisioner": "terraform", "job_id": "1ef72c51-0762-441a-9cba-69774a386b3b"}
Nov 19 00:51:17 ip-10-1-1-211 coder-aws-start.sh[1491]: 2022-11-19 00:51:17.060 [INFO]        <./provisionerd/runner/runner.go:298>        (*Runner).do        unpacking template source archive        {"job_id": "1ef72c51-0762-441a-9cba-69774a386b3b", "size_bytes": 9728}
Nov 19 00:51:17 ip-10-1-1-211 coder-aws-start.sh[1491]: 2022-11-19 00:51:17.185 [INFO]        <./provisionerd/runner/runner.go:590>        (*Runner).runTemplateImportParse        parse complete        {"parameter_schemas": [{"name": "region", "description": "What region should your workspace live in?", "default_source": {"value": "us-east-2"}, "allow_override_source": true, "default_destination": {"scheme": 1}, "redisplay_value": true, "validation_type_system": 1, "validation_error": "Invalid region!", "validation_condition": "contains([\n      \"ap-northeast-1\",\n      \"ap-northeast-2\",\n      \"ap-northeast-3\",\n      \"ap-south-1\",\n      \"ap-southeast-1\",\n      \"ap-southeast-2\",\n      \"ca-central-1\",\n      \"eu-central-1\",\n      \"eu-north-1\",\n      \"eu-west-1\",\n      \"eu-west-2\",\n      \"eu-west-3\",\n      \"sa-east-1\",\n      \"us-east-1\",\n      \"us-east-2\",\n      \"us-west-1\",\n      \"us-west-2\"\n    ], var.region)"}]}
Nov 19 00:51:23 ip-10-1-1-211 coder-aws-start.sh[1491]: 2022-11-19 00:51:23.197 [INFO]        <./provisionerd/runner/runner.go:135>        (*Runner).Run        sent FailedJob        {"job_id": "1ef72c51-0762-441a-9cba-69774a386b3b"}
Nov 19 00:51:17 ip-10-1-1-211 coder-aws-start.sh[1491]: 2022-11-19 00:51:17.003 [INFO]        <./provisionerd/provisionerd.go:219>        (*Server).acquireJob        acquired job        {"initiator_username": "noah", "provisioner": "terraform", "job_id": "1ef72c51-0762-441a-9cba-69774a386b3b"}
Nov 19 00:51:17 ip-10-1-1-211 coder-aws-start.sh[1491]: 2022-11-19 00:51:17.060 [INFO]        <./provisionerd/runner/runner.go:298>        (*Runner).do        unpacking template source archive        {"job_id": "1ef72c51-0762-441a-9cba-69774a386b3b", "size_bytes": 9728}
Nov 19 00:51:17 ip-10-1-1-211 coder-aws-start.sh[1491]: 2022-11-19 00:51:17.185 [INFO]        <./provisionerd/runner/runner.go:590>        (*Runner).runTemplateImportParse        parse complete        {"parameter_schemas": [{"name": "region", "description": "What region should your workspace live in?", "default_source": {"value": "us-east-2"}, "allow_override_source": true, "default_destination": {"scheme": 1}, "redisplay_value": true, "validation_type_system": 1, "validation_error": "Invalid region!", "validation_condition": "contains([\n      \"ap-northeast-1\",\n      \"ap-northeast-2\",\n      \"ap-northeast-3\",\n      \"ap-south-1\",\n      \"ap-southeast-1\",\n      \"ap-southeast-2\",\n      \"ca-central-1\",\n      \"eu-central-1\",\n      \"eu-north-1\",\n      \"eu-west-1\",\n      \"eu-west-2\",\n      \"eu-west-3\",\n      \"sa-east-1\",\n      \"us-east-1\",\n      \"us-east-2\",\n      \"us-west-1\",\n      \"us-west-2\"\n    ], var.region)"}]}
Nov 19 00:51:23 ip-10-1-1-211 coder-aws-start.sh[1491]: 2022-11-19 00:51:23.197 [INFO]        <./provisionerd/runner/runner.go:135>        (*Runner).Run        sent FailedJob        {"job_id": "1ef72c51-0762-441a-9cba-69774a386b3b"}
Oh I might be missing this from the top of my config: 
map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
}
map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
}
Still no dice Is there a specific problem I'm trying to address that I can google about? Is this trying to get a websocket connection through nginx?
Phorcys
Phorcysā€¢3y ago
can you show me your full config? this is a websocket reverse-proxy issue well you did send it in parts but I wanna make sure that you placed it at the right spot something else, did you restart nginx? just making sure and could you show the output of coder templates create aws-linux after doing those
Turtle Kayak
Turtle KayakOPā€¢3y ago
https://gist.github.com/Mainstay-Noah-Huppert/2e6d46b10d2ccf158bc590ee55fa0c6d Here you go. There is the top level /ect/nginx/nginx.conf which loads files in /etc/nginx/conf.d/ and then I have a Nginx conf for coder in that directory. The script to launch the Coder server service actually dynamically creates the Nginx config and then restarts Nginx (bc some parameters about which ports everything should be running on couldn't be known at the time I was building the Coder server AMI so instead I grab those ports from a key value store at runtime and make the config, that's why you see the Nginx config in the Coder service logs). The only unusual thing about my setup is I'm using the AWS managed VPN service to connect into a private subnet.
The attachment creates an elastic network interface (ENI) in the subnet. All of the network traffic from the client subnet is NATed (Network Address Translated) to the ENI IP address.
https://aws.amazon.com/es/blogs/networking-and-content-delivery/using-aws-client-vpn-to-scale-your-work-from-home-capacity/ And I remember something about not being able to use NATs in the server docs. But this way of making private subnets and having a VPN is probably the most common enterprise patterns ever so I assume it work w Coder bc otherwise that would be bonkers
Turtle Kayak
Turtle KayakOPā€¢3y ago
https://github.com/aws/amazon-chime-sdk-js/issues/1985
GitHub
Reconnection timeout on VPN Ā· Issue #1985 Ā· aws/amazon-chime-sdk-js
What happened and what did you expect to happen? The Chime SDK for JavaScript reconnection timed out when turning WiFI on/off on the VPN connection. Have you reviewed our existing documentation? Gi...
Turtle Kayak
Turtle KayakOPā€¢3y ago
It looks like AWS client VPN does work w WebSockets so I think I'm configuring something wrong or idk but it should be possible
Phorcys
Phorcysā€¢3y ago
I'd like to see the one in conf.d
Turtle Kayak
Turtle KayakOPā€¢3y ago
It's all in there
Phorcys
Phorcysā€¢3y ago
oh you did add it
Turtle Kayak
Turtle KayakOPā€¢3y ago
Phorcys
Phorcysā€¢3y ago
yeah honestly this looks good to me I don't really know you should try installing the CLI on the machine where nginx is installed nevermind that wouldn't change anything also if you can use http then you can also use websockets wait you can visit the home page right?
Turtle Kayak
Turtle KayakOPā€¢3y ago
Yea And it seems like some normal cmds like listing workspaces and listing templates works I saw in the help thread linked above that there were two different versions of the Nginx config. You version and then someone else posted a screenshot of theirs. In theirs they set the Connection header to lowercase upgrade while yours had uppercase. They also set the http version manually. Could that make a difference? I mean good to at least see if the server is functioning. I logged in w the CLI on the server, but not through the reverse proxy and directly through the local port. The template create cmd worked
Phorcys
Phorcysā€¢3y ago
you can try forcing the http ver, headers bring uppercase or lowercase rarely matters since most servers interpret them in all lowercase and the convention is to use Upgrade
Turtle Kayak
Turtle KayakOPā€¢3y ago
.... Um Setting the http version did it :poggers: :poggers: :poggers: :poggers: Thank you so much @Phorcys and @colin (rust enthusiast) !
Codercord
Codercordā€¢3y ago
Marked the thread as resolved.
Turtle Kayak
Turtle KayakOPā€¢3y ago
Whoa did a slash cmd just work in mobile šŸ‘€šŸ‘€šŸ‘€ Been a while since that happened šŸ˜… So I guess it's good to know that setting the http version is also a key to the puzzle
Phorcys
Phorcysā€¢3y ago
yeah thanks ! what did you set it to ?
Turtle Kayak
Turtle KayakOPā€¢3y ago
1.1
Phorcys
Phorcysā€¢3y ago
alright, you should probably use 2.0 it's way faster
Turtle Kayak
Turtle KayakOPā€¢3y ago
Here is the full working Nginx conf for reference 
map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
}

server {
    listen 80;
    server_name coder-server.dev.mainstay;

    location / {
        proxy_pass http://localhost:3000;

        # Required to make server communication work correctly: https://discord.com/channels/747933592273027093/1007075267119947807/1007314775493070978
        proxy_pass_request_headers on;
        
        proxy_http_version 1.1;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Accept-Encoding gzip;
    }
}
map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
}

server {
    listen 80;
    server_name coder-server.dev.mainstay;

    location / {
        proxy_pass http://localhost:3000;

        # Required to make server communication work correctly: https://discord.com/channels/747933592273027093/1007075267119947807/1007314775493070978
        proxy_pass_request_headers on;
        
        proxy_http_version 1.1;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Accept-Encoding gzip;
    }
}
I'll try 2.0 and edit the above config if it works Looks like only 1.0 and 1.1 are supported values for that option http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_http_version
Phorcys
Phorcysā€¢3y ago
welp
Want results from more Discord servers?
Add your server