When are connections to the cluster available during the moving of a branch?
For several minutes the branches are moving to a newly provisioned cluster. When are new connections to the branch made to the cluster?
30 Replies
hi, looking into it now
I see the branch move error out, I'm going to cancel it for now as it might be a transient issue
could it be a version mismatch?
i accidentally provisioned the cluster 1 minor version below what the shared cluster has
for now the error is a connectivity issue, like at networking level. I'm checking if it's maybe something transient or a mis-configuraion on our side
The goal is to scale more since we went live and ran into scalability issues due to max concurrent connections.
aha, let me then see if I can increase that temporarily for you on the shared cluster, while we figure this out
one sec
thank you!
the affected branch is called "main"
and dbname is partner-tool, right?
Yes, that's correct.
do you have a rough idea of how many connection you'd need? Also double checking: you are hitting an issue with the connections over the wire protocol, not the concurrency limit over the API, correct?
the problem is, with vercel we have no idea how many serverless functions are actually created.
yes, we are directly connecting over tcp.
ok, I've increased it to 50, let me know if that seems to help
from vercel it might be better to use our SQL over HTTP proxy for this reason
we have also reduced the pg pooling "max" setting from 5 to 3.
the problem is, we are frequently using transactions where we run code in between (drizzle-orm)
I see, gotcha
this is something we cannot currently do over http afaik
yes, that's correct
ok, then let's see if 50 helps, in the meantime we'll see what went wrong with the branch move
let me know what you see on your side
Ok, we will. Thank you.
Quick question, I thought the pro plan's limit is 60. Is this not the case? Or is there also a limit per branch?
ah, that could be
Also, is there a way to check how many connections are currently open? Either through the dashboard or e.g. DBeaver
@tsg We are running again into the same issue.
sorry, was in a call, coming back to it
sorry, indeed, I had it in my mind that the Pro connection limit is 20, but you are right it's 60.
I have increased it to 200 now, hopefully that works
once you move to a dedicate cluster, you will get a view on this, something like:
perfect, thanks
looking at the shared cluster that you are on, I see a peak around 14:30 CET. Is that when you were seeing most errors?
the first burst we had, was at ~<t:1725029400:t>
the branches are still marked as moving, is this expected atm?
I just canceled it now
at least the one for
main, are others in progress?
I see the preview branch is also marked as moving, I'm going to cancel that one as well
ok, done (the "move" wasn't really doing anything at this point, because it failed early on)Thank you so much!
The increase seems to have helped and there shouldn't be any more bursts today.
Is there anything we can do differently regarding cluster provisioning? Feel free to provision a new one with the same scalability settings and/or a different version etc. if necessary.
ok, great, I'll keep you up to date
the connectivity issue that caused the branch move to fail should be solved now.
Let me know if you'd like us to retry the branch move over the weekend, or you prefer to wait until Monday.
Thank you.
It would be great to have it moved before Monday 9AM CEST. Preferably, to a cluster with the same version as the shared cluster is on right now.
I can do this over the weekend myself, I assume, correct? If yes, I will provision a new cluster (as we were in hurry, we didn't think about the other cluster settings) and terminate the current one.
I must once more thank you for the fast response time! I appreciate it! 🙏
We can't wait to scale with Xata 🚀
yes, you can provision a new cluster and terminate the old one. Perhaps give me a ping when you are about to start the branch move, but I'd expect it to work fine
Hi @tsg
Thank you for the update. I've just started provisioning a new cluster. As soon that's done, I will make a backup and start the move of the main branch again. I will ping you if the move is taking more than 30 minutes - no need to actively monitor from the start. I'll post an update here if it's successful as well.
It seems to have worked 🥳
At least the branch management mentions the cluster assignment 👍
Nice!