Emad
RRunPod
•Created by Emad on 8/19/2024 in #⚡｜serverless
LLAMA 3.1 8B Model Cold Start and Delay time very long
I thought flashboot was for the first request as well
30 replies
RRunPod
•Created by Emad on 8/19/2024 in #⚡｜serverless
LLAMA 3.1 8B Model Cold Start and Delay time very long
but for a request after a while it takes over a minute
30 replies
RRunPod
•Created by Emad on 8/19/2024 in #⚡｜serverless
LLAMA 3.1 8B Model Cold Start and Delay time very long
Yes next time it is faster
30 replies
RRunPod
•Created by Emad on 8/19/2024 in #⚡｜serverless
LLAMA 3.1 8B Model Cold Start and Delay time very long
30 replies
RRunPod
•Created by Emad on 8/19/2024 in #⚡｜serverless
LLAMA 3.1 8B Model Cold Start and Delay time very long
both give same result
30 replies
RRunPod
•Created by Emad on 8/19/2024 in #⚡｜serverless
LLAMA 3.1 8B Model Cold Start and Delay time very long
I tried through network volume and normally too
30 replies
RRunPod
•Created by Emad on 8/19/2024 in #⚡｜serverless
LLAMA 3.1 8B Model Cold Start and Delay time very long
according to the blog posts
30 replies
RRunPod
•Created by Emad on 8/19/2024 in #⚡｜serverless
LLAMA 3.1 8B Model Cold Start and Delay time very long
But i thought for LLMs the cold start time was in seconds
30 replies
RRunPod
•Created by Emad on 8/19/2024 in #⚡｜serverless
LLAMA 3.1 8B Model Cold Start and Delay time very long
The reason runpod was pushed by our team was because we say it gave record cold start times.
30 replies
RRunPod
•Created by Emad on 8/19/2024 in #⚡｜serverless
LLAMA 3.1 8B Model Cold Start and Delay time very long
It's usually not used every minute. At night our user count is less so it is not used as frequently.
30 replies
RRunPod
•Created by Emad on 8/19/2024 in #⚡｜serverless
LLAMA 3.1 8B Model Cold Start and Delay time very long
to control costs as well
30 replies
RRunPod
•Created by Emad on 8/19/2024 in #⚡｜serverless
LLAMA 3.1 8B Model Cold Start and Delay time very long
Is there no other solution?
30 replies
RRunPod
•Created by NERDDISCO on 8/9/2024 in #⚡｜serverless
Slow network volume
When I used 70B I gave the network volume 150gb
55 replies
RRunPod
•Created by NERDDISCO on 8/9/2024 in #⚡｜serverless
Slow network volume
But again I am not trying with a 70b model like I was before, just with a 8b model
55 replies
RRunPod
•Created by NERDDISCO on 8/9/2024 in #⚡｜serverless
Slow network volume
Not getting stuck in queue
55 replies
RRunPod
•Created by Emad on 8/8/2024 in #⚡｜serverless
Can't run a 70B Llama 3.1 model on 2 A100 80 gb GPUs.
Didn't calculate
67 replies
RRunPod
•Created by NERDDISCO on 8/9/2024 in #⚡｜serverless
Slow network volume
and can i test on any region?
55 replies
RRunPod
•Created by NERDDISCO on 8/9/2024 in #⚡｜serverless
Slow network volume
should i try again with the 8b model with a network volume?
55 replies