Occasional freeze with many docker workspaces on 1 big machine

Ok so I think I got myself into trouble with https://pad.ws Here's my very naïve setup: I run Coder on a single machine with docker compose, 125GiB and 32cpu as you can see in the screenshot At the time i'm writing this, there is 160+ workspaces (75 currently running) And it's growing by the minute I have to admit I'm super impressed by coder because everything runs super smoothly People are not heavily using their workspaces so I hoped I could get away with just chucking them all on one powerful instance But a few hours ago ~ when I crossed the 150 workspaces mark, I started noticing a regular spike in CPU usage that blocks and makes the workspace connections unresponsive for 1 to 2 second, it happens randomly every 5 to 30 seconds It feels like garbage collection or something that didn't scale well with my reckless deployment I'm starting my own research but I figured I'd post here because you might see the obvious bottleneck thanks a lot (I'm having so much fun 😊 )
No description
4 Replies
Codercord
Codercord3d ago
<#1362705301534539918>
Category
Help needed
Product
Coder (v2)
Platform
Linux
Logs
Please post any relevant logs/error messages.
Yasbaltrine
YasbaltrineOP3d ago
update: now we're at 400 workspaces lol It's ALMOST perfectly responsive, I suspect sysbox-fs (which I use as a container runtime) to be causing the lag spikes due to 1200 threads being open
No description
Yasbaltrine
YasbaltrineOP2d ago
Update: everything went down then back up I came to the conclusion that I should just move to Kubernetes very soon!
Codercord
Codercord2d ago
@Yasbaltrine closed the thread.

Did you find this page helpful?