Vinnymac
PPrisma
•Created by Vinnymac on 2/26/2025 in #help-and-questions
Queries hanging indefinitely
https://github.com/prisma/prisma/issues/26537
just to close the loop here.
20 replies
PPrisma
•Created by Vinnymac on 2/26/2025 in #help-and-questions
Queries hanging indefinitely
The next step here may just be to open a GitHub issue on Prisma, although I imagine the only way to reproduce this properly outside my project would be to have an actual Supabase instance with a live Supavisor pooler running, so that may require some work to configure and setup with a decent dump of data to work with.
20 replies
PPrisma
•Created by Vinnymac on 2/26/2025 in #help-and-questions
Queries hanging indefinitely
Yes, the problem is solved by commenting out new PrismaInstrumentation(). It must be modifying the query request in such a way that’s incompatible with the supabase supervisor.
Which makes sense given some of the other reported issues, like the prepared statement leaking issue I linked above. If Prisma is modifying the query in some way, whatever those modifications are, would most likely explain this issue. Which is why capturing the query would be incredibly useful to debug this issue further.
20 replies
PPrisma
•Created by Vinnymac on 2/26/2025 in #help-and-questions
Queries hanging indefinitely
Is it possible to log queries in Prisma before they are sent, or to get any other diagnostics? The current debug logs aren’t so useful for this sort of problem, and I’m not seeing anything reported on the backend which makes me think the issue is in Prisma alone. Since that would be very odd, I’d expect the issue to have something to do with a malformed request that is sent to the pooler, but for whatever reason isn’t reported in my pooler logs. I don’t have much else to look into here aside from reading all the supabase pooler and Prisma source code myself.
My current plan is to migrate any problematic queries from using Prisma to using Supabase for now to unblock the upgrade of Prisma. And hope this issue gets resolved at some point in the future.
20 replies
PPrisma
•Created by Vinnymac on 2/26/2025 in #help-and-questions
Queries hanging indefinitely
20 replies
PPrisma
•Created by Vinnymac on 2/26/2025 in #help-and-questions
Queries hanging indefinitely
Possibly related to the following issue, although it says it is for MySQL, I imagine if instrumentation effects MySQL prepared statements it most assuredly also effect Postgres prepared statements.
https://github.com/prisma/prisma/issues/18975
20 replies
PPrisma
•Created by Vinnymac on 2/26/2025 in #help-and-questions
Queries hanging indefinitely
What’s weird is that disabling the telemetry successfully makes the deallocate all call and it works with the pgbouncer pooler connection.
So I think we’ve found that something with connection pooling and telemetry is broken, but it’s not clear what that is exactly just yet.
20 replies
PPrisma
•Created by Vinnymac on 2/26/2025 in #help-and-questions
Queries hanging indefinitely
We now know three things
1) disabling telemetry fixes the issue
2) simplifying the query fixes the issue, maybe it doesn’t treat it as a prepared statement?
3) using a direct connection fixes the issue, implying Supabase supervisor is problematic in some way.
20 replies
PPrisma
•Created by Vinnymac on 2/26/2025 in #help-and-questions
Queries hanging indefinitely
This wouldn’t be the first time deallocate all calls have caused that type of hanging issue, and a quick search online shows both queries hanging with deallocate calls in past Postgres versions as well as Prisma related issues with PGBouncer. I think we may have found the main culprit here, an incompatibility between telemetry changing the underlying query and the connection pooler.
20 replies
PPrisma
•Created by Vinnymac on 2/26/2025 in #help-and-questions
Queries hanging indefinitely
Okay, I tried something new just now, and this is a big lead.
I found that using a direct connection instead of the database pooler pgbouncer=true fixes the issue as well.
20 replies
PPrisma
•Created by Vinnymac on 2/26/2025 in #help-and-questions
Queries hanging indefinitely
I was able to backup 100% of the data, took a while as I had to wait for the full dump, but then I seeded my local database, and ran the test.
It ran successfully (and failed to reproduce the bug).
So either something about the database is different or my environment is different.
If I use one of the live databases, the issue occurs, even without changing a single line of code. The entire reproduction is containerized and uses the same image as all of the other tests I’ve done. If it’s not the environment it must have something to do with the db. There are plenty of things that are different in a local database vs the live ones, even the hardware configuration. Perhaps postgres extensions, or connection pooling, or one of the billion other things.
I suppose next I can stop using a simple postgres database, and try an actual supabase local instance with all the extensions installed and see if that leads me anywhere. Nonetheless this is a really strange issue.
20 replies
PPrisma
•Created by Vinnymac on 2/26/2025 in #help-and-questions
Queries hanging indefinitely
I was able to reproduce the issue with very little code / dependencies involved on my local.
The only thing I haven't been able to do yet is reproduce the issue with a local database. Unfortunately the issue doesn't appear to occur if no data exists in the database tables which is peculiar because its not even running a db query, so why in the world would data existing or not have any impact.
I populated about half the tables, and still it didn't break, but I didn't populate the main FROM table yet. So I am going to try to take a backup of the tables where I am seeing this issue, and then seed my local db with the data to see if that reproduces the issue fully locally.
If the data is required to simulate the issue, unfortunately I won't be able to give you the data because this is proprietary information, so I am not sure what we would do next, or if Prisma would be open to getting on a call or something at that point.
20 replies
PPrisma
•Created by Vinnymac on 2/26/2025 in #help-and-questions
Queries hanging indefinitely
Yea, everything in production is running on 6.0.1, because I can’t upgrade past it without a handful of specific queries hanging indefinitely when instrumentation is enabled.
All of the queries that hang are using $queryRaw, and if I simplify the queries to basically not do anything or if I disable instrumentation by commenting out new PrismaInstrumentation() the issue disappears.
Reproducing it outside of my project isn’t the easiest, but I’ll keep working on it when I find time and see if I can get to the bottom of it. Also I am running on Supabase with connection pooling for the database. And I can reproduce the issue outside production in canary test environments, but haven’t had luck making it happen outside my project.
20 replies