Pushing a new release to my existing endpoint takes too long
I've pushed a new release to my endpoint with a new Docker tag image. This tag only modifies some app code, so all the heavy docker layers should already be there.
The system logs show "Pulling fs layer" for most of the layers except the first 5. Isn't RunPod caching the layers somewhere or does it have to pull ALL layers everytime I push a new release even though only the last layer has changed...?
6 Replies
Ya i think so
they doesn't cache the layers, so it repulls all
Well, it's not working at all. It's been 30 minutes since I created the new release and nothing happened. The workers are still on the previous version. I needed to terminate all workers to get update ones..
Ok
set max workers to 0
and back to 3
This is supposed to work seamlessly and without down-time. Setting max workers to 0 would incur in downtime.
i think it works if the worker is more than 3
@Tim aka NERDDISCO can confirm why the new release doesn't work?
you could try asking too about your endpoint in contact from the website
@EMPZ could you please open a new serverless issue via https://contact.runpod.io/hc/en-us/requests/new?ticket_form_id=28101597978771 for this? Because the "new release" should totally work like you would expect it, so this sounds like a bug to me right now. But lets see what our team can do here to help.