Workspace build failed for Kubernetes (Devcontainers)
I've mounted kubeconfig secret onto my Coder deployment and when I deploy Kubernetes (Deployment), it reads the secret correctly and deploys it into the Kubernetes Cluster.
However, for Devcontainers (Kubernetes), it seems like it doesn't read the kubeconfig and workspace build failed with "persistentvolumeclaims is forbidden ... cannot create resource "persistentvolumeclaims"
For both, the provider.kubernetes.configpath is exactly the same.
30 Replies
<#1325636629326725163>
Category
Help needed
Product
Coder (v2)
Platform
Linux
Logs
Please post any relevant logs/error messages.
I'm using "Devcontainers (Kubernetes)" under starter templates
Here's the full error log:
Error: persistentvolumeclaims is forbidden: User "system:serviceaccount:coder-p8wjr:coder" cannot create resource "persistentvolumeclaims" in API group "" in the namespace "coder"
on main.tf line 182, in resource "kubernetes_persistent_volume_claim" "home":
182: resource "kubernetes_persistent_volume_claim" "home" {
facing the same permission issues for coder-logstream-kube as well, logs weren't fetched
I'm deploying the kubernetes pod into another namespace, but it seems like these serviceaccounts do not have permissions for it.
Yes, you'll need to expand the coder service account's permissions to go outside its own namespace.
Can this be done via values.yaml? Or I have to create myself and link it via kubectl
But I still find it a bit weird that Devcontainers (Kubernetes) doesn't read the kubeconfig..? I've tried on my VM deployment, and it reads this kubeconfig perfectly fine hmm
Hmm.. let me back up and ask. Are you firing up workspaces in an external k8s cluster to where you installed Coder?
same k8s cluster, different namespace
In the future, i intend the fire workspaces in an external k8s cluster, that's why i'm testing out with the kubeconfig
Ok. So AFAIK, the kubeconfigs are for external clusters only. If you want the coderd (its service account) to have access to other namespaces inside the k8s cluster it is installed in, you have to give it the proper permissions in that cluster. The kubeconfig won't be used. For my single cluster, I just gave the coder service account full cluster admin permissions.
yes you can.
if you choose to use host kubeconfig in ~/.kube/config, you have to mount into coder agent and make sure kubeconfig is correct.
If you choose to use serviceaccount on Pod running Coder (
use_config variable set to False in this case), serviceaccount should have correct permission. This can be done via values.yaml
https://github.com/coder/coder/blob/b98a1add98f9e0b2f0c11d081a5e1ab73f0928e0/helm/coder/values.yaml#L99-L110thank you! let me try again tomorrow
thank you!
@joelynnnnn @Scott @Codercord this is the error that is coming while creating the workspace with kubernetes devecontainer as a starter template
what is your coder deployed on? vm / docker / kubernetes?
docker
where is your kubeconfig in your docker?
i have install kubenetes with minkube only
how can i verify it like the location of the kubeconfig file
this is all i have
the kubeconfig file that connects to your minikube needs to be present in the docker
how can i access docker files can u suggest
i think it is already presnet in the docker As image is running
u can see minikube is also added
what we use docker to install kubernetes other than minikube
KinD is built to use k8s inside of Docker. I'd look into that. But, I'd also suggest this is a backwards way to work with k8s, unless you are actually developing apps to work with k8s.
so what would you suggest me now like how should i run my workspace related with kubernetes devcontainer template
how can i do this
What is it your devs need to do? Or what is your intentions with Coder to begin with? Why is k8s needed, when you obviously have only a small bit of experience with k8s? It seems you are on a path to a solution, which might not even be good for what your intentions are. I see this all the time. So, a little more background into what it is you expect to achieve and do can help us help you.
i am created a workspace for my project under a docker container template but after researching that which is better docker or kubernetes in terms of loadbalancing stuffs then i found kubernete much better so like what i want is just to creatte my workspace in kubernetes container template so that my project works much better with better efficiency that's all i want
in terms of auto scaling and stuffs
What is your project? Is it a web application? A k8s specific service? k8s does handle loadbalancing and scaling, but are you running your app now in k8s?
no i m not running it in k8
i running it in in docker compose
Docker compose is only an orchestration process for Docker. And Docker isn't made for running production applications. It's made to containerize apps and theoretically develop on them. But, if your application needs k8s to run, you should be developing in k8s. And that is what Coder can do. But, the clusters should be "up and running" and not generated on the fly.
I'd also highly suggest you set up a k8s cluster and work with it for some time. A k8s cluster should be a long running set of processes. It's definitely not something you would just fire up on occasion to work with, despite all the solutions given out there to do so.
Once you have a production k8s cluster, it runs for life!(you can quote me on that) When you have a production cluster running, then you can use smaller clusters as staging setups for development and whatever else you need to work on your apps/ microservices (and that is where Coder can help). k8s is a huge realm of knowledge to gather and a lot of cognitive load, especially just to get "auto-scaling and stuffs". I don't want to be disrespectful, but I believe you are going into this rabbit hole rather naively. There is also the added realm of CI/CD for microservices/ apps in k8s. It's a huge realm on its own. Imagine you have an app and are developing it in a staging cluster. How do you get it into the production cluster? This is a HUGE process with a good number of working parts and possibilities. Once you get into it, it seems like a much bigger problem than just writing your code. 🙂 Believe me, I know. It's something I've been working to learn for about 3 years now (as a hobby) and I'm still learning. Even if you have both prod and staging in the same cluster, which is possible, you'd still need to get the code into the production path. And it aint easy at all. So, I guess what I'm trying to say is, you might want to rethink k8s, unless you absolutely are certain your app needs "auto-scaling and stuffs".
ok got the concept lemme just discuss it with my team
@joelynnnnn closed the thread.