Pulling from the wrong cache when multiple Dockerfiles in same GitHub repo
I seem to be having an issue where the wrong cache is being pulled by a worker when I have specified the Dockerfile in GitHub integration. Any help would be appreciated!
15 Replies
Wrong cache? Like What, Outdated cache?
The cache of another Dockerfile within that repo
Have I misunderstood that you can have multiple Docker images of the same repo?
Yeah I think you can, but I never tried running both dockerfile in my repo on runpod
Only one of them, what does it look like? I mean when the layer from dockerfile isn't the same it won't grab the cache right
Or you could report this to runpod and they can check ur dockerfile
I've reported to Runpod to see if there's a solution. My Dockerfiles are different for each tool but sit in the same repo
It looks like it's pulling another Dockerfile from that repo weirdly
Did you specify the name correctly, also selected the right branch?
Yep specified the Dockerfile location and the main branch which seems to be correct
I see, yeah that's weird
It's weird as when the endpoint first initiates it uses the correct image but after that it then pulls the other image
Hopefully just a fixable bug on Runpod's end
will have eng look at this
@JohnDoe i am looking at this now. Is this what is happening:
- when you first create the endpoint, it builds using the dockerfile you specified but after a push to that branch, it is built using a different dockerfile?
While we are at it, do you mind sharing the buildId/buildIds where this problem happened?
One of the Build IDs is this e78ce3ae-7f8e-4333-92aa-d8c90d44a026
What happens is I create the endpoint, the image builds and it seems to be building the correct image. However when it pulls the image it pulls another image from the same repo
What I think is happening is that the newest build for a given repo is overwriting any existing images/this is being pulled through from the container registry. This is the case even if the Dockerfiles are different
thats possible @PRB
Hey @flash-singh let me also add one of my similar issue:
1 Repo, 2 branch, 2 deployment
Branches are mostly identical but each branch's Dockerfile has an ENV variable model name different
- And on docker build, downloads the model with this ENV variable & uses that ENV variable on runtime to use that model
On my concurrent deployment for these 2 branches
After deployment, on deployment b - branch b:
It tried to find correct model name (ENV on runtime was ok) but couldn't find it's model
I doubt that it used the cache layer from branch-a's deployment of download_models.py, code is 100% same, file reads model_type from ENV
Re-deploy branch-b fixed issue
@flash-singh @PRB Whatever you have done it seems to have fixed the issue. Thank you
Happy to help