kip
RRunPod
•Created by kip on 8/24/2024 in #⚡｜serverless
Execution time discrepancy
Is there a plan to fix this, or not yet?
75 replies
RRunPod
•Created by kip on 8/24/2024 in #⚡｜serverless
Execution time discrepancy
I had it concurrent first and then moved to synchronous to try and debug, I’d prefer not to share the code for that one though
75 replies
RRunPod
•Created by kip on 8/24/2024 in #⚡｜serverless
Execution time discrepancy
Yeah, I had issues with this too, which I think is also tied to the billing
75 replies
RRunPod
•Created by kip on 8/24/2024 in #⚡｜serverless
Execution time discrepancy
I’m surprised you haven’t run into that before though
75 replies
RRunPod
•Created by kip on 8/24/2024 in #⚡｜serverless
Execution time discrepancy
Ahh, I kind of suspected that this might be the case, the usage was always suspiciously just over a second
75 replies
RRunPod
•Created by kip on 8/24/2024 in #⚡｜serverless
Execution time discrepancy
Had a chance to have a look yet?
75 replies
RRunPod
•Created by kip on 8/24/2024 in #⚡｜serverless
Execution time discrepancy
Thanks for taking a look 🙂
75 replies
RRunPod
•Created by kip on 8/24/2024 in #⚡｜serverless
Execution time discrepancy
I tried keeping 1 worker active, didn't make a difference
75 replies
RRunPod
•Created by kip on 8/24/2024 in #⚡｜serverless
Execution time discrepancy
just not sure what to do to decrease this hidden delay, the worker should be able to handle 20ish requests a second, but at the moment isn't able to handle 1 request per second
75 replies
RRunPod
•Created by kip on 8/24/2024 in #⚡｜serverless
Execution time discrepancy
I don't really mind about how long it takes the request to reach the runpod scheduler, this isn't blocking
the delay has to be happening sometime after the it hits the scheduler
75 replies
RRunPod
•Created by kip on 8/24/2024 in #⚡｜serverless
Execution time discrepancy
Yeah i don't think it accounts for the 1 second delay, maybe the proxy polls to see when the job is done or something and this adds some delays though
75 replies
RRunPod
•Created by kip on 8/24/2024 in #⚡｜serverless
Execution time discrepancy
Correct, yeah
75 replies
RRunPod
•Created by kip on 8/24/2024 in #⚡｜serverless
Execution time discrepancy
I've tested running the handler locally with
--rp_serve_api and the time to perform the forward pass is something like 20 -> 50ms75 replies
RRunPod
•Created by kip on 8/24/2024 in #⚡｜serverless
Execution time discrepancy
How are you handing your models?I've given an example in the github repo i sent, but generally i just load them at the top of the handler file
Are they stored on Network Storage, baked into the image, or loaded at run time?The models are baked into the image, i think i saw that this was the recommended way of doing it on Runpod
What region are your workers in?Global, i'm not sure how to see the region of a specific pod though. also i don't see how this can really impact execution time
What is your active and max workers set to?One for now, just to test
75 replies
RRunPod
•Created by kip on 8/24/2024 in #⚡｜serverless
Execution time discrepancy
Each request is being proxied. There is going to be some overhead in that. 1 second? That is likely just overhead in proxying your request.I assume you mean by whatever scheduler Runpod uses? I would expect that to be included in the delay time? Also if the request is being proxied that should be blocking any new requests getting to the pod, but for me it seems to be blocking. And one full second is a lot of time
75 replies
RRunPod
•Created by kip on 8/24/2024 in #⚡｜serverless
Execution time discrepancy
The response from the worker is the embedding as well as the time it takes to perform the forward pass, and the time it takes to execute is always significantly higher than the time it takes to perform the forward pass
75 replies
RRunPod
•Created by kip on 8/24/2024 in #⚡｜serverless
Execution time discrepancy
@nerdylive @yhlong00000 I put together a small example that has the issue https://github.com/kipgparker/clip-runpod
75 replies
RRunPod
•Created by kip on 8/24/2024 in #⚡｜serverless
Execution time discrepancy
I'll do this tomorrow anyway though, I'll ping you with the code then
75 replies
RRunPod
•Created by kip on 8/24/2024 in #⚡｜serverless
Execution time discrepancy
maybe, one of the faster trasnformers for sure
75 replies