thisisfine
RRunPod
•Created by thisisfine on 6/19/2024 in #⚡｜serverless
Loading models from network volume cache is taking too long.
Nope this is not our inference time. It's only for model loading. Our inference time has been consistently fast. It's just that the model loading latency has been unpredictable.
23 replies
RRunPod
•Created by thisisfine on 6/19/2024 in #⚡｜serverless
Loading models from network volume cache is taking too long.
Yeah with an active instance, it's much faster taking only 3~7 secs. So I think it's a mix of cold start + establishing a new connection with network volume + etc?
23 replies
RRunPod
•Created by thisisfine on 6/19/2024 in #⚡｜serverless
Loading models from network volume cache is taking too long.
Yeah. That could be an option too. But if anyone knows how to fundamentally resolve this issue, pls lmk!
23 replies
RRunPod
•Created by thisisfine on 6/19/2024 in #⚡｜serverless
Loading models from network volume cache is taking too long.
Assume that these are all cold starts. I'm still seeing different latency performance (from 3~40 secs) and I think it's a network volume issue.
23 replies
RRunPod
•Created by thisisfine on 6/19/2024 in #⚡｜serverless
Loading models from network volume cache is taking too long.
I'm seeing a lot of 30~40 secs latency recently. Please let me know if there is a way to optimize this!
23 replies
RRunPod
•Created by thisisfine on 6/19/2024 in #⚡｜serverless
Loading models from network volume cache is taking too long.
They are small (2GB, 1GB)
I have the flashboot on. Does it mean that all my workers should be flashbooted when we are cold starting them?
And I only log time that it takes for loading the models. I'm not sure cold start or flashboot impacts it.
23 replies
RRunPod
•Created by thisisfine on 6/11/2024 in #⚡｜serverless
Uploading a file to network volume takes forever and fails after a few mins
I just got it from our s3 via boto3. I don't recommend using the upload UI in Jupyter.
5 replies
RRunPod
•Created by papanton on 2/25/2024 in #⚡｜serverless
Two Network Volumes
Is this updated now?
23 replies