Is there something wrong in US-OR-1?
There seemed to be issues on Thu 10/31 making my ComfyUI pod unusable (Comfy taking 20+ minutes to start and be available, throwing errors when it ran, web terminal challenging for auth and continuously rejecting valid creds). Are these issues ongoing? I went to start a pod today (11/1) and it seemed to exhibit the same issues so I backed away before I burned more credits.
15 Replies
What issue, is there an incident telling that makes data gone from network storage
@K-Woww
Escalated To Zendesk
The thread has been escalated to Zendesk!
No recent incidents listed in #🚨｜incidents that say anything about network storage data loss unless you go back to July. I'm asking if there is an issue going on right now. I described my symptoms and none of them area data loss, though I haven't looked for data loss.
US-OR-1 is bad. We described the same problem directly to the RunPod team and also supplemented the report with speedtest. It is usually 3x slower than US-TX-3 with x6 ping
Ugh. I'm new to US-OR-1 and it sounds like that's just a crappy DC, which sort of tracks with me moving from EU-RO-1 and feeling that US-OR-1 is a little better but not a lot better than EU-RO-1. So US-TX-3 sounds good, maybe I'll move my network storage there.
@K-Woww are you still experience issue?
I'm starting a pod to look.
Thank you!
Short answer is yes, the issue is still happening. The symptoms appear unchanged, i.e. pod starts, CPU indicates that it's working to start up services for ~30s and then subsides. Under normal conditions, when CPU goes to zero, services including ComfyUI and Web Terminal are connectible. Now, even after the CPU utilization goes to zero, ComfyUI and Web Terminal take a long time (~5m+) to start and become connectible. After the both Comfy and Terminal claim to be connectible, and I click to connect, ComfyUI spins forever (10+m) until I stop the pod to stop burning credits. Web Terminal works though as far as I can see.
could you send me your pod id?
I DM'd you my podid but I deployed a new pod using the "valyriantech/comfyui-with-flux:latest" template and had the same issues.
i mean the web terminal is automatically auth'ed after you click connect isn't it? hows its looking when it rejects valid auth
and i guess comfyui's errors when it ran should be because of data corruption or some bad extension (probably)
Usually that’s the case, that web terminal opens without needing to explicitly type in username/password. When it was having issues, it popped up a username and password dialog box, like a Chrome/javascript type box, and when I’d enter the provided credentials and submitted it, the box would disappear for like 10ms and reappear empty. Endless cycle and no way to access web terminal.
Ahh I see
Maybe it has something to do with the template
Let staffs check first
There is an incident Friday afternoon in US-OR-1 for poor network performance and it was dog slow even at after midnight.