Jobs suddenly queuing up: only 1 worker active, 9 jobs queued
Endpoint: vieo12phdoc8kh
Hi, are there any known issues at the moment with 4090s? Our processing has suddenly started backing up with no known changes on our side. Only 1 of our 8 workers seems to be active, with the others sitting "ready" according to the logs. This issue appears to have started about 90 minutes ago.
We have failed over to another provider for now.
4 Replies
Things seem to be running smoothly again now but it does look like there was an issue. It would be awesome to know if there's any way I could get more info on what happened here.
are you using network volume? if so what region?
No network volume
we had a huge spike in 4090s usage, thats more stable now, we plan to add 100s more by next week