DB Latency
Hi everyone. On our Coder instance, we've been experiencing Coder-DB latency that triggers disconnects on workspaces. I believe we have an overload of database operations that are flooding postgres. The usual log we see is this:
2025-01-23 18:47:21.790 [info] coderd: disconnected possibly outdated agent workspace_id=xxxx agent_id=xxxx request_id=xxxx ...
error= fetch object:
github.com/coder/coder/v2/coderd/database/dbauthz.(*querier).GetWorkspaceByID.fetch[...].func1
/home/runner/work/coder/coder/coderd/database/dbauthz/dbauthz.go:522
- context canceled
I have a couple of questions here:
1. How do you usually measure DB workload? With Prometheus I can see active connections and max transaction duration, but they don't seem that high.
2. Is this issue somehow related to this behavior? https://github.com/coder/coder/issues/15082 @DanielleGitHub
Reduce DB load of autobuild · Issue #15082 · coder/coder
Motivation The autobuild/ package periodically queries for workspaces that are eligible for a state transition via GetWorkspacesEligibleForTransition. This is run every CODER_AUTOBUILD_POLL_INTERVA...
7 Replies
<#1332092301056413737>
Category
Help needed
Product
Coder (v2)
Platform
Linux
Logs
Please post any relevant logs/error messages.
hey @vlvrd, which version of Coder are you running?
we're using 2.18.3, I'm about to upgrade to 2.18.4
hey @vlvrd, sorry for the delay, any luck?
hi! Not really, the issue appears less frequently after I buffed the DB resources (we're using our own postgresql deployment, we have about 80 users and usually ~40 WS are active at the same time) but it still looms over us. I'm going to create a new Grafana dashboard measuring Postgresql and workspaces IOPS to see if we have insights there, and as soon as I have some time I want to go over how I can monitor the Coder agents activity in case I'm missing something there.
please let us know how it goes!
see https://coder.com/docs/admin/integrations/prometheus for metrics
I'm closing this so we can focus on the ongoing issues but feel free to send a message here, we'll see it!
@Phorcys closed the thread.