jim
RRunPod
•Created by jim on 2/19/2025 in #⚡｜serverless
Feb 20 - Serverless Issues Mega-Thread
I see. An ETA on the rollout would help in our planning if you can get one
56 replies
RRunPod
•Created by jim on 2/19/2025 in #⚡｜serverless
Feb 20 - Serverless Issues Mega-Thread
All good 🙂 thanks. Any update to share?
56 replies
RRunPod
•Created by jim on 2/19/2025 in #⚡｜serverless
Feb 20 - Serverless Issues Mega-Thread
@Dj You mentioned the patch should be ready within a day, but mentioned Friday. I assume you mean Thursday (today is the 20th)?
56 replies
RRunPod
•Created by jim on 2/19/2025 in #⚡｜serverless
Feb 20 - Serverless Issues Mega-Thread
Confirming that SDK 1.7.7 does not solve this @Dj. US-GA-2 has fewer issues than CA
56 replies
RRunPod
•Created by blue whale on 2/11/2025 in #⚡｜serverless
Job stuck in queue and workers are sitting idle
What is runpod doing???
36 replies
RRunPod
•Created by jim on 2/19/2025 in #⚡｜serverless
Feb 20 - Serverless Issues Mega-Thread
@Dj Any updates?
56 replies
RRunPod
•Created by jim on 2/19/2025 in #⚡｜serverless
Feb 20 - Serverless Issues Mega-Thread
@Felipe Fontana US-GA-2 has the same issue for me unfortunately
56 replies
RRunPod
•Created by jim on 2/19/2025 in #⚡｜serverless
Feb 20 - Serverless Issues Mega-Thread
I see. Has there been a significant increase in the % of cold boots in the recent few days (esp. for the endpoints that have flash boot enabled) due to increasingly scarce capacity?
56 replies
RRunPod
•Created by jim on 2/19/2025 in #⚡｜serverless
Feb 20 - Serverless Issues Mega-Thread
All the other zones don't have a lot of H100 availability tho, but let's see
56 replies
RRunPod
•Created by jim on 2/19/2025 in #⚡｜serverless
Feb 20 - Serverless Issues Mega-Thread
Ah yeah, all of mine are CA
56 replies
RRunPod
•Created by jim on 2/19/2025 in #⚡｜serverless
Feb 20 - Serverless Issues Mega-Thread
The fact that a lot of my workers were not flash booted, despite having flash boot enabled seems a little sus too
56 replies
RRunPod
•Created by jim on 2/19/2025 in #⚡｜serverless
Feb 20 - Serverless Issues Mega-Thread
@Dj Does this explain why workers are "running" for over 10 minutes, while not picking up any queue'd up requests?
56 replies
RRunPod
•Created by blue whale on 2/11/2025 in #⚡｜serverless
Job stuck in queue and workers are sitting idle
Also tracking here: https://discord.com/channels/912829806415085598/1341839170787741786
36 replies
RRunPod
•Created by blue whale on 2/11/2025 in #⚡｜serverless
Job stuck in queue and workers are sitting idle
Yes zvhg9gcnqmkugx
36 replies
RRunPod
•Created by jim on 2/19/2025 in #⚡｜serverless
Feb 20 - Serverless Issues Mega-Thread
56 replies
RRunPod
•Created by jim on 2/19/2025 in #⚡｜serverless
Feb 20 - Serverless Issues Mega-Thread
Sounds like that may be part of the cause: very slow boots that while being booted are in "running" state
56 replies
RRunPod
•Created by jim on 2/19/2025 in #⚡｜serverless
Feb 20 - Serverless Issues Mega-Thread
Very interesting that it's not flash booting, despite it being enabled
56 replies
RRunPod
•Created by jim on 2/19/2025 in #⚡｜serverless
Feb 20 - Serverless Issues Mega-Thread
56 replies
RRunPod
•Created by jim on 2/19/2025 in #⚡｜serverless
Feb 20 - Serverless Issues Mega-Thread
Ah maybe that's a hint at the cause of the issue?
56 replies
RRunPod
•Created by jim on 2/19/2025 in #⚡｜serverless
Feb 20 - Serverless Issues Mega-Thread
In the meantime, I'll upgrade to latest SDK version to see if it helps
56 replies