tolga
RRunPod
•Created by tolga on 9/9/2024 in #⚡｜serverless
Getting slow workers randomly
yes, for sure but the thing is i've had this issue many times for the past 2-3 months. What's the best way for me to gather large data for you to easily check and spot the issue? This screenshot is a rare scenario where i have the request logs for both slow and fast workers that popped up. AFAIK there's no way for users to gather logs from runpod serverless endpoints so this is all i could gather as evidence
7 replies
RRunPod
•Created by tolga on 9/9/2024 in #⚡｜serverless
Getting slow workers randomly
I'm not sure how vCPUs work but my guess is that these slow workers are getting bottlenecked by cpus somehow since comfy's custom node imports are taking way longer than usual in those
7 replies
RRunPod
•Created by tolga on 9/9/2024 in #⚡｜serverless
Getting slow workers randomly
@yhlong00000 thank you for the response! that makes it more weird though 😅 comfy is initialized when worker boots up, and the prompt that's sent to both of these workers are exact same. You can see the slow worker's cold start execution time was 134 seconds and warm requests was taking around 12 seconds, whereas the other takes 41 seconds for cold start and 3 seconds warm execution time
7 replies