zeeb0t
RRunPod
•Created by yasek on 11/16/2024 in #⚡｜serverless
What methods can I use to reduce cold start times and decrease latency for serverless functions
would love to pay for that kind of storage
26 replies
RRunPod
•Created by yasek on 11/16/2024 in #⚡｜serverless
What methods can I use to reduce cold start times and decrease latency for serverless functions
i just hope permanent docker stores become a thing
26 replies
RRunPod
•Created by yasek on 11/16/2024 in #⚡｜serverless
What methods can I use to reduce cold start times and decrease latency for serverless functions
less reliant on flashboot, i should say
26 replies
RRunPod
•Created by yasek on 11/16/2024 in #⚡｜serverless
What methods can I use to reduce cold start times and decrease latency for serverless functions
once they implement a permanent store for docker images to boot from, similar to network volumes, i think it’ll make a great solve for the cold start scenario and make us less reliant on runtime network volume and less concerned about flashboot
26 replies
RRunPod
•Created by yasek on 11/16/2024 in #⚡｜serverless
What methods can I use to reduce cold start times and decrease latency for serverless functions
cold start when the files are in the network volume instead of in the docker image will be faster, UNLESS the files are single files that need to be read in full as part of the runtime process to service a request. it’ll then be a delaying factor - and one that you are paying gpu time for while things load. for instance if an ai model is being read from a network storage and is 20gb, and nothing works until that model is read into vram in every non-flashboot scenario, it’s then going to feel slow. however my experience has been that the docker container, once loaded, tends to be more reliably cached than flashboot can boot in a flash. plus the docker image is part of the initialising phase and i don’t think you are billed for that? so it may be better financially, too
26 replies
RRunPod
•Created by yasek on 11/16/2024 in #⚡｜serverless
What methods can I use to reduce cold start times and decrease latency for serverless functions
Your cold start will suck, but while the worker has a cache copy, it'll be pretty fine after that.
26 replies
RRunPod
•Created by zeeb0t on 11/12/2024 in #⚡｜serverless
Mounting network storage at runtime - serverless
It is something I'd happily pay to have
11 replies
RRunPod
•Created by zeeb0t on 11/12/2024 in #⚡｜serverless
Mounting network storage at runtime - serverless
Is there any plan to allow network storage to host our docker images? Or a persistant cache otherwise?
11 replies
RRunPod
•Created by yasek on 11/16/2024 in #⚡｜serverless
What methods can I use to reduce cold start times and decrease latency for serverless functions
You can put the large model files in either the container directly or on the network storage to be read from in runtime. I prefer to put in container and have raised a similar topic: https://discordapp.com/channels/912829806415085598/1305822723917873152
26 replies
RRunPod
•Created by yasek on 11/16/2024 in #⚡｜serverless
What methods can I use to reduce cold start times and decrease latency for serverless functions
@pg2571 I just meant using the runpod handler function
26 replies
RRunPod
•Created by yasek on 11/16/2024 in #⚡｜serverless
What methods can I use to reduce cold start times and decrease latency for serverless functions
create a docker container that has everything needed to get the job done, and registers with the worker (using the runpod python module for instance). sometimes the container gets cached by the host and can save on cold start times. and if you have large files you need available, you can use the network storage
26 replies
RRunPod
•Created by zeeb0t on 11/12/2024 in #⚡｜serverless
Mounting network storage at runtime - serverless
works if the model is small. otherwise it takes an age to download the image and rarely is it cached
11 replies