Occasional freeze with many docker workspaces on 1 big machine
Ok so I think I got myself into trouble with https://pad.ws
Here's my very naïve setup:
I run Coder on a single machine with docker compose, 125GiB and 32cpu as you can see in the screenshot
At the time i'm writing this, there is 160+ workspaces (75 currently running)
And it's growing by the minute
I have to admit I'm super impressed by coder because everything runs super smoothly
People are not heavily using their workspaces so I hoped I could get away with just chucking them all on one powerful instance
But a few hours ago ~ when I crossed the 150 workspaces mark, I started noticing a regular spike in CPU usage that blocks and makes the workspace connections unresponsive for 1 to 2 second, it happens randomly every 5 to 30 seconds
It feels like garbage collection or something that didn't scale well with my reckless deployment
I'm starting my own research but I figured I'd post here because you might see the obvious bottleneck
thanks a lot (I'm having so much fun 😊 )

4 Replies
<#1362705301534539918>
Category
Help needed
Product
Coder (v2)
Platform
Linux
Logs
Please post any relevant logs/error messages.
update: now we're at 400 workspaces lol
It's ALMOST perfectly responsive, I suspect sysbox-fs (which I use as a container runtime) to be causing the lag spikes due to 1200 threads being open

Update: everything went down then back up
I came to the conclusion that I should just move to Kubernetes very soon!
@Yasbaltrine closed the thread.