pg2571
RRunPod
•Created by yasek on 11/16/2024 in #⚡｜serverless
What methods can I use to reduce cold start times and decrease latency for serverless functions
thanks that makes sense! whats a rough estimate of "use your endpoint for a very long time"? Is it in the range of hours or days?
26 replies
RRunPod
•Created by yasek on 11/16/2024 in #⚡｜serverless
What methods can I use to reduce cold start times and decrease latency for serverless functions
for context mine is a comfyui workflow with ~30GB of models (but each individiual model is 1-6GB)
26 replies
RRunPod
•Created by yasek on 11/16/2024 in #⚡｜serverless
What methods can I use to reduce cold start times and decrease latency for serverless functions
ofc. but when a worker gets its first request, i assume it initializes from the dockerfile which it has to download right?
or is the ops is such that the worker is already on a pod with the dockerfile downloaded, and its able to load it from the local pod when a request comes in
26 replies
RRunPod
•Created by yasek on 11/16/2024 in #⚡｜serverless
What methods can I use to reduce cold start times and decrease latency for serverless functions
maybe i don't fully understand how cold starts work. but isn't it faster to get a 30GB file from a local file server than downloading it from the internet (i.e. when downloading the dockerfile with everything baked in)?
26 replies
RRunPod
•Created by pg2571 on 11/16/2024 in #⚡｜serverless
Not able to connect to the local test API server
Sorry! this is solved. Just had to update the runpod lib.
4 replies
RRunPod
•Created by yasek on 11/16/2024 in #⚡｜serverless
What methods can I use to reduce cold start times and decrease latency for serverless functions
because my traffic is very very spikey
26 replies
RRunPod
•Created by yasek on 11/16/2024 in #⚡｜serverless
What methods can I use to reduce cold start times and decrease latency for serverless functions
how bad is cold start for a 20gb docker image?
and is cold start faster on a network volume?
26 replies
RRunPod
•Created by yasek on 11/16/2024 in #⚡｜serverless
What methods can I use to reduce cold start times and decrease latency for serverless functions
gotcha thanks, lemme try putting it in the container and see perf
26 replies
RRunPod
•Created by yasek on 11/16/2024 in #⚡｜serverless
What methods can I use to reduce cold start times and decrease latency for serverless functions
@zeeb0t i had a similar question (https://discord.com/channels/912829806415085598/1307189362806624378/1307189362806624378)
"docker container that has everything to get the job done" vs "large files you need available, ... network storage"
so where do large models go? eg. SDXL checkpoints
what do you mean by "registers with the workers"
26 replies