Single Coder server instance crashing with 20ish or so workspaces
We have been giving Coder a trial and have seen really great improvements to DX. But we did run into an issue with 20 or so workspaces running seemingly crashed the server instance.
@Théo asked to upload some info we found regarding the issue.
From working with our Platform Engineer, here's what we can share:
Workspaces would be temporarily accessible when the Coder Embedded Relay (CER) was green, but if CER went red and then back to green shortly after, all workspaces became inaccessible again until each of the individual workspaces were restarted. This became a cycle about halfway through debugging; CER green, I restart my WS and access it, CER goes red, CER goes green, I restart my WS to access it...
Unfortunately those logs would have hit the lifecycle policy on the 4th.
During the incident, the Coder Embedded Relay / DERP portions of Coder's health page were rapidly flipping between green and red. See screenshot.
4 Replies
<#1307071966766694461>
Category
Help needed
Product
Coder OSS (v2)
Platform
Linux
Logs
Please post any relevant logs/error messages.
@Fattyacid i see
has this happened multiple times?
I can check but believe it was isolated to that specific event
okay, this really shouldn't happen but without logs or a way to reproduce i'm afraid we can't really identify the root cause