Pulling from the wrong cache when multiple Dockerfiles in same GitHub repo

I seem to be having an issue where the wrong cache is being pulled by a worker when I have specified the Dockerfile in GitHub integration. Any help would be appreciated!
15 Replies
nerdylive
nerdylive2w ago
Wrong cache? Like What, Outdated cache?
JohnDoe
JohnDoeOP2w ago
The cache of another Dockerfile within that repo Have I misunderstood that you can have multiple Docker images of the same repo?
nerdylive
nerdylive2w ago
Yeah I think you can, but I never tried running both dockerfile in my repo on runpod Only one of them, what does it look like? I mean when the layer from dockerfile isn't the same it won't grab the cache right Or you could report this to runpod and they can check ur dockerfile
JohnDoe
JohnDoeOP2w ago
I've reported to Runpod to see if there's a solution. My Dockerfiles are different for each tool but sit in the same repo It looks like it's pulling another Dockerfile from that repo weirdly
nerdylive
nerdylive2w ago
Did you specify the name correctly, also selected the right branch?
JohnDoe
JohnDoeOP2w ago
Yep specified the Dockerfile location and the main branch which seems to be correct
nerdylive
nerdylive2w ago
I see, yeah that's weird
JohnDoe
JohnDoeOP2w ago
It's weird as when the endpoint first initiates it uses the correct image but after that it then pulls the other image Hopefully just a fixable bug on Runpod's end
flash-singh
flash-singh2w ago
will have eng look at this
PRB
PRB2w ago
@JohnDoe i am looking at this now. Is this what is happening: - when you first create the endpoint, it builds using the dockerfile you specified but after a push to that branch, it is built using a different dockerfile? While we are at it, do you mind sharing the buildId/buildIds where this problem happened?
JohnDoe
JohnDoeOP2w ago
One of the Build IDs is this e78ce3ae-7f8e-4333-92aa-d8c90d44a026 What happens is I create the endpoint, the image builds and it seems to be building the correct image. However when it pulls the image it pulls another image from the same repo What I think is happening is that the newest build for a given repo is overwriting any existing images/this is being pulled through from the container registry. This is the case even if the Dockerfiles are different
flash-singh
flash-singh2w ago
thats possible @PRB
Eren
Eren2w ago
Hey @flash-singh let me also add one of my similar issue: 1 Repo, 2 branch, 2 deployment Branches are mostly identical but each branch's Dockerfile has an ENV variable model name different - And on docker build, downloads the model with this ENV variable & uses that ENV variable on runtime to use that model On my concurrent deployment for these 2 branches After deployment, on deployment b - branch b: It tried to find correct model name (ENV on runtime was ok) but couldn't find it's model I doubt that it used the cache layer from branch-a's deployment of download_models.py, code is 100% same, file reads model_type from ENV Re-deploy branch-b fixed issue
JohnDoe
JohnDoeOP2w ago
@flash-singh @PRB Whatever you have done it seems to have fixed the issue. Thank you
PRB
PRB2w ago
Happy to help

Did you find this page helpful?