What happens during cold start time?
Hi! I am new to docker and as well as to serverless. I would like to know what happens during cold start, does my image gets pulled again each time? Or it's just the time required for provising resources and loading models in vram etc?
22 Replies
Yes, the second one
Image pulling(exc from cold start) in serverless wont be billed, cold start will
Awesome. I have another question. I have model weights of 30gbs. Do you recommend adding them in docker image or I should download them using a script to mounted volume?
download them into a mounted volume
What's the path of mounted drive?
usually on serverless its /runpod-volume
but if on pods its usually /workspace
Cool. Thanks for answers
I think flash recommended to add your model a part of docker image whenever possible. It probably better than network volume for performanceš
I am super confused now. Lol
What's up
How to decide when to add model as part of docker image or download it on mounted drive
I mean if you want you can test them to decide
Which suits better
And time your tests
Lmk if you did that hahah
Your model is quite big right
30gbs of disk space
Yeah
Try them both, and run tests
Image is being pulled on each cold start
No its just the loading of your model, applications
No pulling again
Check the logs
If you think about it, when your Docker image contains everything necessary, the container is ready to go as soon as it starts, with all data stored on the host disk for fast access. In contrast, if you store the model on a network volume, you would need to mount it and connect through the data centerās Ethernet to network storage, which is likely to be slower than accessing the local disk.
I checked on my docker hub, I can see how many times my image was being pulled
Okay the logs also say when it's pulled
When deploying your serverless function for the first time requires pulling the image, which can be slow. If you keep sending requests, the container remains active, that avoid cold starts. However, if thereās a pause in requests, Runpod stops the container. When requests resume, a cold start occurs, but it will be faster than the initial image pull.
Yeah cold starts aren't loading image for the first time, cold starts only for when you rarely hit requests to your endpoint
@nerdylive How long before the endpoint goes ācoldā? It doesn't seem to be a constant time, if so do you know what it is?
yeah no specific amount