Do you cache docker layers to avoid repulling ?
I have a question, does the current serverless system caches the docker layers to avoid repulling ?
Let's assume I make a docker image with the first layers being the AI models, if I upload this, the dokcer repo will only get the new ones and won't update layers it already knows
If I pull the same will happen
Does the Serverless docker pulling also does the same? so that if I update my image, it will avoid repulling the layers it already knows ?
6 Replies
https://canary.discord.com/channels/912829806415085598/1244895085859180544
yeah i think it does pull the changed only
but if you want it to be sure, i guess try to use the github builds, it surely caches the layers on build
As far as I know, some data centres do, and some don't. For example, on EU-RO I had to redownload the whole image every time. On EU-CZ it's pulling only the difference.
@flash-singh
EU-RO DC is bigger and has lot of machines so there is higher chance you end up on new machine that has no image cached
- all data centers support local layer caching
- some data centers with network storage also offer caching in network storage for serverless
- local cache is always used if the changes you make to endpoint don't change gpu, cuda version or some other attributes which require changing the server, otherwise we keep the worker and have it update in-place
I would check EU-RO then. I've never experienced layer caching there. I didn't even know it's supported on RunPod until I was able to use EU-CZ and was pleasantly surprised it's there. This was happening even on a tiny few kb rewrites of the handler file. While cached just fine by the docker hub on upload, the data centre always pulled the whole huge image again. It's wasting your bandwidth there for sure.