Coder.com•2w ago

Is there a way to have a number of coder_agents defined from a single resource block?

I'm looking to have something like the following so that I can define an arbitrary number of nodes and have all of them report their stats in the Coder page. When I do this, I get errors related to duplicate resource definitions. One such example is error: template import provision for start: duplicate metadata resource: libvirt_domain.node Full template's source can be found here. Is something like this possible, or am I going to have to figure out a different way to go about this?

resource "coder_agent" "main" {
  count = data.coder_workspace.me.start_count == 0 ? 0 : data.coder_parameter.node_count
  ...
}

resource "libvirt_domain" "node" {
  count = data.coder_workspace.me.start_count == 0 ? 0 : data.coder_parameter.node_count
  ...
  provisioner "remote-exec" {
    inline = [ "echo ${coder_agent.main[count.index].token} > ~/.config/coder/token" ]
  }
}

resource "coder_metadata" "libvirt_domain_node" {
  count = data.coder_workspace.me.start_count == 0 ? 0 : data.coder_parameter.node_count
  resource_id = libvirt_domain.node[count.index].id
  hide = true
}

resource "coder_agent" "main" {
  count = data.coder_workspace.me.start_count == 0 ? 0 : data.coder_parameter.node_count
  ...
}

resource "libvirt_domain" "node" {
  count = data.coder_workspace.me.start_count == 0 ? 0 : data.coder_parameter.node_count
  ...
  provisioner "remote-exec" {
    inline = [ "echo ${coder_agent.main[count.index].token} > ~/.config/coder/token" ]
  }
}

resource "coder_metadata" "libvirt_domain_node" {
  count = data.coder_workspace.me.start_count == 0 ? 0 : data.coder_parameter.node_count
  resource_id = libvirt_domain.node[count.index].id
  hide = true
}

24 Replies

Codercord•2w ago

Category: Help needed Product: Coder OSS (v2) What platform are you hosting Coder OSS (v2) on?

Phorcys•2w ago

@bamhm182 (BytePen) try using for_each

Phorcys•2w ago

https://developer.hashicorp.com/terraform/language/meta-arguments/for_each

The for_each Meta-Argument - Configuration Language | Terraform | HashiCorp Developer

The for_each Meta-Argument - Configuration Language | Terraform | H...

The for_each meta-argument allows you to manage similar infrastructure resources without writing a separate block for each one.

Phorcys•2w ago

also I'd recommend defining a "node" module that you import multiple times should make it more manageable

bamhm182 (BytePen)OP•2w ago

Thanks! Somehow I had thought that for_each would have the same outcome as trying to use count. Will give it a shot. @Phorcys , looks like that does have the same outcome. Will look into modules.

Phorcys•2w ago

does it work if you get rid of the coder_metadata resource? yeah sorry i thought somehow you could define the resource name using foreach

bamhm182 (BytePen)OP•2w ago

It's more so the coder_agent that I'm needing to have duplicates of, but they both fail in the same way.

Phorcys•2w ago

i know, but the error screams about the metadata

bamhm182 (BytePen)OP•2w ago

Only because I only had a single coder_agent. If I try to do any sort of loop with coder_agent, it also yells. I think that the problem is that I need to have different names for each coder_agent, and I can't use for_each or count with them. I think modules are going to have the same issues, but as I was watching some videos on modules I had an idea that might be worth pursuing. It seems as though you can programmatically create *.tf.json files, so I'm thinking that maybe I can make a module that creates a "node-x.tf.json" file for each instance, and then have terraform use those... I have to dig more into it...

Phorcys•2w ago

i'm not sure you can generate these files at runtime and have terraform take them into account i'm not sure how to achieve what you want to do but it should be possible i'll look into it

bamhm182 (BytePen)OP•7d ago

Yeah. I think you're right... I thought that's what I was seeing, but I misunderstood. The only way I can see to do it is to create a bunch of resource blocks, and then have logic which excludes them if node_count is below a certain number...

resource "coder_agent" "node-0" {
  count = var.node_count >= 1 ? coder_workspace.me.start_count : 0
}

resource "coder_agent" "node-1" {
  count = var.node_count >= 2 ? coder_workspace.me.start_count : 0
  ...
}

resource "coder_agent" "node-0" {
  count = var.node_count >= 1 ? coder_workspace.me.start_count : 0
}

resource "coder_agent" "node-1" {
  count = var.node_count >= 2 ? coder_workspace.me.start_count : 0
  ...
}

I hate that, but I could at least create a python script that would create all of the required files based on a max number of nodes... Ran into a flaw with that plan... Coder does type checking against tf.json files and is mad when count isn't a number

Phorcys•7d ago

not great but I'm not sure there's anything better

bamhm182 (BytePen)OP•7d ago

Yeah. I think I'm just going to hard code it at 3 nodes.

Phorcys•6d ago

@bamhm182 (BytePen) thanks for the issue, I totally forgot to ask you to create one so I'm happy you did! how are your experiments with Windows on QEMU going? i'll give it a stab again soon

bamhm182 (BytePen)OP•4d ago

No problem! It sounds like it probably won't get addressed any time soon. May do some digging to see how huge of a lift it would be. It got moved over to the terraform-provider-coder repo, which is a lot less intimidating than the main Coder repo. As for Windows, I got a few different versions of Windows to boot with crappy hardcoded passwords and it met my goals, so I haven't bothered going back to make it better. That said, your recommendation to use an unattended.xml to bootstrap a sysprep'd image was spot on and if/when I get around to trying again, that's the route I will be taking. In the meantime, I have still been using Coder constantly for my NixOS images and couldn't be happier. @Phorcys I think I didn't fully understand the problem when I had written off modules initially. I was thinking about it more today and it felt like modules may actually solve the problem, so I tried and am currently getting a different error I am having difficult deciphering. Can you take a look and see if I'm missing something obvious? https://github.com/bamhm182/Coder-Templates/blob/c8505f6a56bccb7c96980f789a714c981c316152/libvirt-k3s/libvirt.tf This is the error I am currently getting:

Encountered an error running "coder templates push", see "coder templates push --help" for more information
error: template import provision for start: couldn't find node on graph: "module.node0"

Encountered an error running "coder templates push", see "coder templates push --help" for more information
error: template import provision for start: couldn't find node on graph: "module.node0"

I tried local modules as well as the module hosted on github as seen in this commit. Both returned the same message. I thought it may have been because I didn't know how to use modules, so I added in code-server to make sure and was able to get that one working both as expected to be used, and as a local module. I tried adding a coder_agent in the root module and passing through agent_id to the child module. No luck there either. Of course I just needed to ask you in order to get it figured out... I deleted the counts and it worked... Then I added a second node and it failed due to duplicate resource names.... Womp womp

Phorcys•3d ago

yeah i don't think it's possible i have an idea for a workaround though, it's hacky but should do the job i'll get back to you soon

bamhm182 (BytePen)OP•3d ago

I wound up getting it working ALMOST as intended here. https://github.com/bamhm182/Coder-Templates/tree/fe3f4e6161ca0c3ead1a7b26fad49bb85844315d/libvirt-k3s I put the coder_agents in the root module and created differently named "node" modules. The problem I am running into now, though, is that USUALLY only one coder_agent will be visible at a time. One time, I got them all to show up, but I think it was when they had no coder_app's As I sent that, if occurred to me that maybe it is quietly mad in the background because the coder_app's all have the same name and those need to be moved into the root module too. Will give that a shot when I get home. No dice...

Phorcys•3d ago

yeah so i'm looking into it but it seems pretty hard because you would need macros to do it properly, which aren't a thing in terraform where are your coder_app definitions?

bamhm182 (BytePen)OP•2d ago

https://github.com/bamhm182/Coder-Templates/blob/fe3f4e6161ca0c3ead1a7b26fad49bb85844315d/libvirt-k3s/modules/node/guacamole.tf

GitHub

Coder-Templates/libvirt-k3s/modules/node/guacamole.tf at fe3f4e6161...

Contribute to bamhm182/Coder-Templates development by creating an account on GitHub.

bamhm182 (BytePen)OP•2d ago

Yeah... I think I'm just going to succumb to manually defining each and every resource on its own. I am probably going to wind up with node0.tf, node1.tf, etc. So that I can just make changes to node0, blast it over the rest, then use sed to replace the instances of node0

Phorcys•2d ago

yeah i don't think there's a better way

bamhm182 (BytePen)OP•this hour

Yeah. Unfortunate, but far from the end of the world. Just hate unnecessary duplicated code. Alternatively... Somewhere along the line, I had clean-ish code, and just had all the various coder_apps shoved under one agent. I don't actually need the different agent for each VM

Phorcys•23h ago

yeah alternatively you could have a single agent and just put the other nodes in a virtual net

bamhm182 (BytePen)OP•22h ago

Yeah. The plan was for them to be in a virtual net anyway, I just wanted to have their stats visible in Coder, but it isn't a big deal.

Gaming

Programming

Is there a way to have a number of coder_agents defined from a single resource block?