Coder.com•3y ago

Running workspace inside k8s pods

General question, what backends are most people running? I'm all for the k8s approach as its the easiest way for me (on top of other benefits) - but I'm not sure if this is where Coders' focus is at. For example, I can't seem to "port forward" or use coder config-ssh command. Are these supported when coder runs inside of a pod? For example, while wildcard domains is configured, I get

Port forward
Your deployment does not have web port forwarding enabled. See the docs for more details.

You can use the Coder CLI to forward ports from your workspace to your local machine, as shown below.

Port forward
Your deployment does not have web port forwarding enabled. See the docs for more details.

You can use the Coder CLI to forward ports from your workspace to your local machine, as shown below.

Running coder config-ssh returns

coder config-ssh 
fetch workspace configs failed: json: cannot unmarshal array into Go value of type codersdk.WorkspacesResponse
Run 'coder config-ssh --help' for usage.

coder config-ssh 
fetch workspace configs failed: json: cannot unmarshal array into Go value of type codersdk.WorkspacesResponse
Run 'coder config-ssh --help' for usage.

Any tips appreciated.

35 Replies

Phorcys•3y ago

I'm running in Docker but both those features should still work in any env as long as the coder agent is running and can communicate with the server

tfishlerOP•3y ago

So I'm missing the token thing. the agent fails to register with the server.. @Phorcys - thanks for being so attentive!

Phorcys•3y ago

yeah so that's defo the issue can you show me your CODER_WILDCARD_ACCESS_URL env var ? also for the token part I really don't know I think @Ben might be able to help ? you should post your error logs to make the help faster when the team has time to take a look

bpmct•3y ago

Hey! 1) For port forward you'll need to set a wildcard access url: https://coder.com/docs/coder-oss/latest/admin/configure#wildcard-access-url 2) Try updating your local client to the latest Coder version to config-ssh. We changed the API recently and didn't display a proper error

Configuration - Coder OSS main docs

Learn how to configure Coder

Phorcys•3y ago

he did set an access URL but I suspect he might've left the scheme in

tfishlerOP•3y ago

You're right.

bpmct•3y ago

Ah awesome

tfishlerOP•3y ago

my bad. I did leave the scema. Still.. the agent cannot register.. I get a 401. with the default k8s pod resource taking the token from the coder_agent.main.token resource.

bpmct•3y ago

Hmm what are the logs of the pod? kubectl logs <workspace-pod-name>

tfishlerOP•3y ago

message.txt

bpmct•3y ago

Can you share your pod spec in the Terraform template?

tfishlerOP•3y ago

resource "kubernetes_pod" "main" {
  count = data.coder_workspace.me.start_count
  metadata {
    name      = "coder-${lower(data.coder_workspace.me.owner)}-${lower(data.coder_workspace.me.name)}"
    namespace = var.namespace
  }
  spec {
    security_context {
      run_as_user = "1000"
      fs_group    = "1000"
    }
    container {
      name    = "dev"
      image = "codercom/enterprise-base:ubuntu"

      command = ["sh", "-c", coder_agent.main.init_script]
      security_context {
        run_as_user = "1000"
      }
      env {
        name  = "CODER_AGENT_TOKEN"
        value = coder_agent.main.token
      }
      volume_mount {
        mount_path = "/home/coder"
        name       = "home"
        read_only  = false
      }
    }

    volume {
      name = "home"
      persistent_volume_claim {
        claim_name = kubernetes_persistent_volume_claim.home.metadata.0.name
        read_only  = false
      }
    }
  }
}

resource "kubernetes_pod" "main" {
  count = data.coder_workspace.me.start_count
  metadata {
    name      = "coder-${lower(data.coder_workspace.me.owner)}-${lower(data.coder_workspace.me.name)}"
    namespace = var.namespace
  }
  spec {
    security_context {
      run_as_user = "1000"
      fs_group    = "1000"
    }
    container {
      name    = "dev"
      image = "codercom/enterprise-base:ubuntu"

      command = ["sh", "-c", coder_agent.main.init_script]
      security_context {
        run_as_user = "1000"
      }
      env {
        name  = "CODER_AGENT_TOKEN"
        value = coder_agent.main.token
      }
      volume_mount {
        mount_path = "/home/coder"
        name       = "home"
        read_only  = false
      }
    }

    volume {
      name = "home"
      persistent_volume_claim {
        claim_name = kubernetes_persistent_volume_claim.home.metadata.0.name
        read_only  = false
      }
    }
  }
}

Just trying to get a feeling of how it should work before diving in.

bpmct•3y ago

Hmm... that should work. When you start/stop the workspace, does anything change? Or check if it is properly set in the pod

kubectl exec --stdin --tty <pod-name> -- /bin/bash
echo $CODER_AGENT_TOKEN

kubectl exec --stdin --tty <pod-name> -- /bin/bash
echo $CODER_AGENT_TOKEN

With your other screenshot (the port-forward button) it seemed like everything connected OK

tfishlerOP•3y ago

Yep. this contains a uuid.

$ echo $CODER_AGENT_TOKEN
6744db36-98fb-4d7d-a13c-cc614311167e

$ echo $CODER_AGENT_TOKEN
6744db36-98fb-4d7d-a13c-cc614311167e

bpmct•3y ago

What version of Coder are you using? (shows on the bottom of the /workspaces page)

tfishlerOP•3y ago

/tmp/coder.7BJuF7$ ./coder version
Coder v0.12.5+165b6fb Mon Nov  7 23:35:01 UTC 2022
https://github.com/coder/coder/commit/165b6fbc6a23fd5f58ed459ea50f42e642c2f696

/tmp/coder.7BJuF7$ ./coder version
Coder v0.12.5+165b6fb Mon Nov  7 23:35:01 UTC 2022
https://github.com/coder/coder/commit/165b6fbc6a23fd5f58ed459ea50f42e642c2f696

tfishlerOP•3y ago

funny enough, stopping and starting recreated the pod and no 401 errors. config-ssh still does not work. Let me try to create a new workspace.

bpmct•3y ago

Oh nice! So your CLI is the same version as the server?

tfishlerOP•3y ago

The cli looks newer.

coder version
Coder v0.12.7+49b340e Mon Nov 14 20:11:50 UTC 2022

coder version
Coder v0.12.7+49b340e Mon Nov 14 20:11:50 UTC 2022

bpmct•3y ago

Ahh ok. Yeah that mismatch is why you're running into the issue with config-ssh. I'd update the server 🙂

tfishlerOP•3y ago

A new workspace still gets 401 😦 Something with the order of things? can it be that the code agent starts before the token is being exported?

bpmct•3y ago

Can you send your full template? Is it anything out of the ordinary compared to our example

tfishlerOP•3y ago

Nothing I took your vanilla example just Yesterday and wanted to run it first. The only thing I did, is to create a new docker image from your base in order to add a custom CA cert so I could download code-server (curl fails for self signed certs) - see my other thread from today. upgrading the server worked for ssh!

bpmct•3y ago

It's still super odd that the first build leads to a 401 error :/

tfishlerOP•3y ago

So now I'm a bit worried about backward compatibility for my users.. Lets recreate another one.

bpmct•3y ago

Yeah we don't plan on introducing breaking changes like that, especially over patch versions. At the very least, we should document it and display a message in the CLI for developers to upgrade. It was a bug on our end

tfishlerOP•3y ago

ok. this is weird. got a 401 but ssh connects. I'll get you the entire log

bpmct•3y ago

Oh and in the dashboard it shows up fine?

tfishlerOP•3y ago

yes.

bpmct•3y ago

Yeah it's possible there was just an order issue. If the dashboard shows the agent is connected, you can disregard that. We can file a bug report though so that there is not log spam

tfishlerOP•3y ago

test3.log

tfishlerOP•3y ago

Alright then... will continue my journey here 🙂 Thanks a lot @Ben and @Phorcys for your support! I'll probably bug you again later. @Ben mind filing the bug report?

bpmct•3y ago

Yeah i'm on it!

Codercord•3y ago

Marked the thread as resolved.

Gaming

Programming

Running workspace inside k8s pods