Should terraform init run on every deploy?
I've done an init locally to generate a terraform.lock.hcl and pushed that, but I can see that on every deploy Coder is running an init that's taking ~4-5 seconds, at least 3 of which is installing providers, which seems excessive if the providers had been cached locally from a previous run on a machine with an SSD.
I'm fairly new to terraform but I thought that if these were in place it should be able to skip that and go straight to plan/apply?
My current deployment is running off the coder:v2.18.2 docker image.
17 Replies
Looking back at this, I'm not sure why it's saying there's changes between the registry and my lock file when I made it within a few hours of this.
Are terraform providers platform specific? Maybe I need to generate this in wsl rather than windows?
Terraform providers are os-arch specific binaries, but FWICT,
terraform init writes hashes for all platforms into .terraform.lock.hcl
, Terraform will pre-populate the hashes value with any checksums that are covered by the provider developer's cryptographic signature, which usually covers all of the available packages for that provider version across all supported platforms.I'm not sure why it would be saying there's changes, presumably immediately after generating the lock file 🤔 Plus, you're already using the latest version of both providers Just as an experiment, you could try generating the lockfile on a machine with the same os-arch as your coder server / provisioner? Maybe there's some subtle difference in the way windows generates the lockfile
I tried generating the lock file on WSL rather than windows proper and the init stage is using the cache and the time is down to about 1s now, but for some reason my code-server isn't starting healthily 😅
maybe that was just a fluke, have done a few more tests and can get my workspaces up and connected in browser in under 25s now
Can see that there's something differeing in the h1 hash in the diff, will keep this in mind in the future, think I can call this question answered
Though I am still a little curious as to whether an init is still necessary to run after the first deployment of a particular version? if the files can't change between published versions of a template can you get away with only running an init to cache the modules/providers the very first time?
h1: Hash scheme 1 is also a SHA256 hash, but is one computed from the contents of the provider distribution package, rather than of the .zip archive it's contained within. This scheme therefore has the advantage that it can be calculated for an official .zip file, an unpacked directory with the same contents, or a recompressed .zip file which contains the same files but potentially different metadata or compression schemes.
The lock file is in the current version. I've had a quick look around the provisioner code, and I could be in the wrong place or misunderstanding things but it looks like there's a call to init in the Plan method, which is called everytime regardless of whether the providers/modules have been installed before?
GitHub
coder/provisioner/terraform/provision.go at cbe300448c15570753eec64...
Provision remote development environments via Terraform - coder/coder
Oh I see what you mean
Yeah the init uses the lockfile to see if it has the providers cached, so that should be correct
I suppose this is the safe option, but seems like there could be some time saved by only doing this once? I'm probably in a very small minority of people for whom startup time is important, maybe when I get time I could look at trying this in a fork
How long does the init take with the caching working?
currently 1s out of ~20-25s for our deployment to kubernetes. We're currently trying to get that under 20, part of which will be reducing out startup script time and pod scheduling
but this also seemed like a possible vector for improvement. Assumed that a terraform init would occur when a version was published if there was a lock file, then that would be skipped straight to plan on requesting a workspace
there's very likely a good reason why coder does things the way it does that I'm not seeing right now but if our particular use case is somewhat narrow compared to most users I might be able to get some optimisations in, or contribute back if it's actually a meaningful improvement for everyone
Sounds good, let me know how you go!
maybe the line endings?
We have a feature called "workspace prebuilds" in the works to speed up workspace startup time, you might be interested in taking a look -> https://github.com/coder/coder/issues/5325
GitHub
Workspace pre-builds · Issue #5325 · coder/coder
Some of our users have templates with a long time to provision (e.g. Kubernetes clusters, OpenStack VMs, etc.) or large startup scripts / git clones that take 30+ minutes. Overview For users with t...
yeah that would definitely be of interest. The improved profiling has already been of help in figuring out where our time is being spent.