Coder.com•3y ago

PVC Creating forever...

I'm trying to deploy Coder.OSS on K8S... during the workspace provisioning, TF reports volume.claim creating until time exceeded. On K8S, the PVC shows as pending. Any clue?

32 Replies

kyle•3y ago

Seems like your configuration doesn't have anything mounting the PVC... I see you have a pod there. Is it using the PVC?

f0ssel•3y ago

Could you describe the storage class that is being used by the PV?

hutgerOP•3y ago

Storage class is the K3S Local-path... Indeed, I have two pods within the same namespace (coder), but they are coderd and coder-db... no workspace related pod though.

hutgerOP•3y ago

it seems that pvc is waiting for a pod to bind it but no pod is being started.

EvokeMe•3y ago

Same.

f0ssel•3y ago

What templates are you running into this with?

EvokeMe•3y ago

kubernetes @f0ssel

Phorcys•3y ago

What template specifically ?

EvokeMe•3y ago

I dont understand. Looks like already answered:

coder templates init
> Choose kubernetes
cd ./kubernetes && coder templates create

coder templates init
> Choose kubernetes
cd ./kubernetes && coder templates create

EvokeMe•3y ago

Phorcys•3y ago

oh, sorry, I thought there was multiple kubernetes templates

EvokeMe•3y ago

Nope, just trying from scratch in k3s cluster in hetzner single master.

EvokeMe•3y ago

Postgres claim bound succesfully.

EvokeMe•3y ago

But workspace always fail. First problem was that default workspace in tf was not connected correctly. For some reason coder-workspaces does not work, fixed it in tf template (role succefully created, extended to * resources, and binded)

EvokeMe•3y ago

After that claim was created without storage class. Fixed like this:

EvokeMe•3y ago

But still claim stuck:

EvokeMe•3y ago

Looks like local-path-provisioner waits pod as first consumer like in this example: https://github.com/rancher/local-path-provisioner

EvokeMe•3y ago

But logic of workspace creation is different. It waits pvc claim bound and after that creating pod.

bpmct•3y ago

Does it work if you disable wait_until_bound? https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/persistent_volume_claim#wait_until_bound

Terraform Registry

EvokeMe•3y ago

@Ben creation of workspace we passed with wait_until_boun flag. Right now code-server unhealthy.

bpmct•3y ago

Oh that's odd. What are the logs inside the workspace of /tmp/coder-startup-script.log? I imagine the Terminal and SSH works to connect?

EvokeMe•3y ago

cp: cannot create regular file '/home/coder/.profile': Permission denied
cp: cannot create regular file '/home/coder/.bashrc': Permission denied
tee: code-server-install.log: Permission denied
Ubuntu 20.04.5 LTS
Installing v4.7.1 of the amd64 deb package from GitHub.

+ mkdir -p ~/.cache/code-server
mkdir: cannot create directory '/home/coder/.cache': Permission denied
/bin/bash: line 12: code-server: command not found
tee: code-server-install.log: Permission denied

cp: cannot create regular file '/home/coder/.profile': Permission denied
cp: cannot create regular file '/home/coder/.bashrc': Permission denied
tee: code-server-install.log: Permission denied
Ubuntu 20.04.5 LTS
Installing v4.7.1 of the amd64 deb package from GitHub.

+ mkdir -p ~/.cache/code-server
mkdir: cannot create directory '/home/coder/.cache': Permission denied
/bin/bash: line 12: code-server: command not found
tee: code-server-install.log: Permission denied

I imagine the Terminal and SSH works to connect?

Terminal works fine.

$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
overlay         78601312 37849864  37503240  51% /
tmpfs              65536        0     65536   0% /dev
tmpfs            1963764        0   1963764   0% /sys/fs/cgroup
/dev/sda1       78601312 37849864  37503240  51% /home/coder
shm                65536        0     65536   0% /dev/shm
tmpfs            3927532       12   3927520   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs            1963764        0   1963764   0% /proc/acpi
tmpfs            1963764        0   1963764   0% /proc/scsi
tmpfs            1963764        0   1963764   0% /sys/firmware

$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
overlay         78601312 37849864  37503240  51% /
tmpfs              65536        0     65536   0% /dev
tmpfs            1963764        0   1963764   0% /sys/fs/cgroup
/dev/sda1       78601312 37849864  37503240  51% /home/coder
shm                65536        0     65536   0% /dev/shm
tmpfs            3927532       12   3927520   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs            1963764        0   1963764   0% /proc/acpi
tmpfs            1963764        0   1963764   0% /proc/scsi
tmpfs            1963764        0   1963764   0% /sys/firmware

Looks like /home/coder was initialized with root:root permission only. After sudo chown -R 1000:1000 /home/coder vscode server started.

bpmct•3y ago

Ah nice. I'm surprised that happened in the first place but running that in the startup script should do the trick.

EvokeMe•3y ago

Yep, right now last small improvement with wildcard certificate and we have done. Dont have questions. Thanks a lot! @Ben

bpmct•3y ago

Nice! So the wildcard is working too?

EvokeMe•3y ago

I did not start implementing it! 🙂 On this week will try.

bpmct•3y ago

Ahh got it!

Gaming

Programming

PVC Creating forever...