Regional routing issue
Are there any plans for supporting custom connection limits per region? Right now we seem to be hitting that limit and losing performance by being routed to a nearby region.
12 Replies
Hi! Let me take a quick look at your traffic
For now I'm running some load tests in our staging environment to see if we can ditch our own pgBouncer
When we started this project I tried hyperdrive but it didn't perform as we were expecting
This looks much better already btw, so great job!
Love to hear it!
I do see what I assume are your load tests. What sort of traffic volume are you expecting to see, ballpark?
Yup those are the load tests. This is a couple of times higher than what we would see at peak traffic as we need to have some room to grow comfortably
To answer the original question, we do intend to support custom connection limits, now that trying to set them per-region isn't going to needlessly complicate things. We probably won't be shipping that very soon, so in the meantime we have been setting higher limits on a case by case basis for users who expect heavier traffic.
What I noticed is that we had 100 connections open to our DB, but if we pass the 20 connections limit we start to see significant increases in latency
Let me check in with product, and see about getting you sorted out for the next release.
In terms of expected latency/throughput, how far off of "comfortable" would you say your current tests are?
That 100 is the new limit (going down to 65 as a default so we don't blow up free-tier hosting on most platforms)
I'm curious what the breakpoint you're seeing around 20 actually is. Any monitoring on your DB for latency/contention there?
Right now I'm just testing the path of placing orders and some events related to it, so I would need to change our tests to have a better idea of how many connections we would be using at peak traffic as we don't track this currently
I'm monitoring k6 request latency and using pgAdmin to see the active connections
Got it. K6 will only give you the e2e latency, if I remember right. I'm trying to get my head around where the time is going. Usually it's very little on the actual DB itself, but that's worth ruling out before I dig elsewhere, which is why I asked.
Yeah k6 will only give e2e latency
I remember seeing that query latency would stay similar, but I'll double check it as i'm not completely sure
I meant, given that I see almost zero errors, the issue is latency degradation, I assume? Latencies degrade below what you need them to be at some level of traffic. Is that level 50% of your full test load? 20%? etc.
Following up, the reason I ask is it'll help me assess if just adding some more connections might be enough to help get you where you need to be, or if there may be other perf issues we should look at.