Coder.com•9mo ago

Envbuilder in Coder not using caches at all despite configuring CACHE_REPO

Hello, I am using coder with envbuilder but I seem to be getting 0 cache hits even when restarting the same workspace with 0 changes. I have configured CACHE_REPO to use an empty AWS ECR image repository and I see evbuilder successfully checking for caches and, supposedly,

#1: 📦 Cloned repository! [46.039909689s]
#2: Deleting filesystem...
#2: 🏗️ Building image...
#2: Using dockerignore file: /workspaces/project/.dockerignore
#2: Retrieving image manifest python:3.9-slim
#2: Retrieving image python:3.9-slim from registry index.docker.io
#2: Built cross stage deps: map[]
#2: Retrieving image manifest python:3.9-slim
#2: Returning cached image manifest
#2: Executing 0 build triggers
#2: Building stage 'python:3.9-slim' [idx: '0', base-idx: '-1']
#2: Checking for cached layer MYREMOTERPEO/image:d00475f8d556bc445027bebba8d27c1a4efb356acab9e5ecc8aa4b18719d9f6f...
...
#2: Pushing layer MYREMOTERPEO/image:d00475f8d556bc445027bebba8d27c1a4efb356acab9e5ecc8aa4b18719d9f6f to cache now
...
#2: Taking snapshot of files...
#2: Pushing layer MYREMOTERPEO/image:d00475f8d556bc445027bebba8d27c1a4efb356acab9e5ecc8aa4b18719d9f6f to cache now
...

#1: 📦 Cloned repository! [46.039909689s]
#2: Deleting filesystem...
#2: 🏗️ Building image...
#2: Using dockerignore file: /workspaces/project/.dockerignore
#2: Retrieving image manifest python:3.9-slim
#2: Retrieving image python:3.9-slim from registry index.docker.io
#2: Built cross stage deps: map[]
#2: Retrieving image manifest python:3.9-slim
#2: Returning cached image manifest
#2: Executing 0 build triggers
#2: Building stage 'python:3.9-slim' [idx: '0', base-idx: '-1']
#2: Checking for cached layer MYREMOTERPEO/image:d00475f8d556bc445027bebba8d27c1a4efb356acab9e5ecc8aa4b18719d9f6f...
...
#2: Pushing layer MYREMOTERPEO/image:d00475f8d556bc445027bebba8d27c1a4efb356acab9e5ecc8aa4b18719d9f6f to cache now
...
#2: Taking snapshot of files...
#2: Pushing layer MYREMOTERPEO/image:d00475f8d556bc445027bebba8d27c1a4efb356acab9e5ecc8aa4b18719d9f6f to cache now
...

Every single Dockerfile step is ran even though I seem to be pushing layers to the cache repo every image build. Am I missing anything?

5 Replies

Codercord•9mo ago

<#1218291841145765949>

Category

Help needed

Product

Coder OSS (v2)

Platform

Linux

Logs

Please post any relevant logs/error messages.

Atif•9mo ago

cc: @kyle

kyle•9mo ago

Can you give me the full output of two runs? If possible, the image as well.

tedOP•9mo ago

Yep, the image is ghcr.io/coder/envbuilder:0.2.6 Shared the full logs with @kyle via dm. If needed for others can also go that route or create a smaller reproduceable example to share publicly This is probably what we're all expecting but I also tested in envbuilder 0.2.9 with the same results. My uneducated guess would be that the tagging is not behaving as intended for whatever reason. I would expect that these cache pull/push lines should all have the same image tag. But they do seem to match between run 1 and 2 so it's curious that I seem to get 0 cache hits. 🤔 Think I might've figured this out. The tags which were reported in the logs as having been successfully pushed to the ECR repo were not actually being pushed. I added additional permissions to the ECR repo and now I can see them. I'm about to retry to see how many cache hits I get. It was:

          "ecr:BatchCheckLayerAvailability",
          "ecr:BatchGetImage",
          "ecr:GetDownloadUrlForLayer"

          "ecr:BatchCheckLayerAvailability",
          "ecr:BatchGetImage",
          "ecr:GetDownloadUrlForLayer"

It is now:

                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:DescribeImages",
                "ecr:BatchGetImage",
                "ecr:InitiateLayerUpload",
                "ecr:UploadLayerPart",
                "ecr:CompleteLayerUpload",
                "ecr:PutImage"

                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:DescribeImages",
                "ecr:BatchGetImage",
                "ecr:InitiateLayerUpload",
                "ecr:UploadLayerPart",
                "ecr:CompleteLayerUpload",
                "ecr:PutImage"

I'm not sure why I wasn't getting clearer errors from Kaniko previously when layer uploadds were presumeably failing 🤔 Yep, I am getting at least some cache hits now: #2: Found cached layer, extracting to filesystem 2m0.765828453s vs 5m13.460330402s 👍 Still getting this at the end which I'm not really sure about:

`#2: Error uploading layer to cache: open : no such file or directory
#2: :construction_site: Built image! [1m58.428319491s]

That last error is no longer happening on new workspaces 🤷‍♂️ I think I can consider this issue solved for my own case. Do you think it would be worth updating the docs with the ECR setup steps somewhere? If so I can create a PR and get more input there. Not sure how much kaniko setup you wanna have in the envbuilder repo

kyle•9mo ago

I think that'd be awesome. It's weird that the layers appeared to get pushed, but weren't actually.

Gaming

Programming

Envbuilder in Coder not using caches at all despite configuring CACHE_REPO