Cannot connect to supabase db from railway
I just ported my project from a planetscale db to a supabase postgresdb. Everything works fine locally and within Dockerfile using the new connection string. Added connection string to railway and suddenly my service no longer works and crash loops due to prisma disconnecting. I've only seen the prisma log once however.
I triple checked that I am using the right DATABASE_URL in my railway service definition.
Project ID ef7fcff6-4369-4ace-9d06-f74fd0f47a4e
65 Replies
Project ID:
ef7fcff6-4369-4ace-9d06-f74fd0f47a4e
You might find these helpful:
- Cannot connect to the database.
- is it possible to connect to a railway database from local project?
- Unable to connect to PostgresSQL
⚠️ experimental feature
send a screenshot of your service variables and send your dockerfile please
I changed some src code between this change as well, but like I said it builds just fine locally using this docker image and the new env var
cleared all caches too
you dont need to define ARGs in dockerfile if you arent using the args in the dockerfile
you also dont need an EXPOSE
you also shouldn't need a dockerfile_path variable
I have a nested turborepo
with multiple dockerfiles
I added the expose to debug some discord websocket stuff
add logging for the environment variables to try to see whats going wrong
theres better ways to define it, but ig a variable will work too
yeah, my main concern is that there is some kind of networking issue preventing connections to supabase or perhaps some caching going on causing my old prisma client to be reused
The dockerfile is unchanged, new database is synced with prisma schema and works with a local bot
I can try deleting database_url and re-adding maybe
logging!!!
hrmmm
it's crashing super fast before my logging goes off
it is logging the correct connection url too
seems like a code issue tbh
that only occurs in railway?
What could work in docker but not work when shipped up to railway is what confuses me
i know, sounds odd but i see this all the time
I can imagine
I will poke at it more tomorrow, thanks for the quick check in
I've spun up a new postgres database from railway, changed my env vars to that, and now my app works again
so there is some kind of connection issue with the string supabase gave me and the container running in railway
Because the same connection string works when injected with .env and run with yarn and with ARGS when run in docker image
I pretty much just went with supabase for my db so that I can take advantage of the vector extension. If I can add that to the railway db I could co-locate but I'm very confused by this issue overall
was your connection string unquoted?
I've seen someone have that issue before
railways environment key value parser might not be the most robust
hmm it does not have quotes around it
I think I tried adding them but let me try again
fwiw I set up an identical deployment in render this morning to try and rule out railway and it happened there as well
so something about this connection string just does not work in the cloud 🤪
I have seen other peoples app on railway use supabase, so I assure you railway isn't blocking a connection or anything of that nature
I finally found some relevant logs on supabase
C-0xaaaabe0cb810: pgbouncer/[email protected]:47242 pooler error: SASL authentication failed
So it sounds to me like my docker container does not have the correct certificates configured perhaps. Prisma was not telling me anything about auth errors, just crash looping my appwell you know what that means!! time to move to a Dockerfile!
I guess I could deploy a postgres instance inside of a dockerfile and serve it on railway but I would lose the nice table viewer and stuff
you actually couldn't do that, there's no persistent storage
man
I am so perplexed, I've exhausted most of my options at this point and confirmed that it's not a railway issue
I'm talking about a dockerfile for your app not the database
I do that
That's why I've been so confused
The dockerfile works locally but not when deployed to render or railway
and only with regards to connecting to the supabase database via prisma
since you build with a dockerfile, railway now doesn't have anything to do with any certificate stuff
right
but what in my local env could make certificate stuff work when given the same dockerfile is my confusion
makes me want to test a supabase connection for myself
I would be extremely grateful. All I did was
- Make a new supabase project
- Get the connection string / password
- Add it as DATABASE_URL var
- Try to run a prisma project with it
I can't easily try to run my app "baremetal" in railway to eliminate docker as a suspect because my node app requires a few python deps for audio processing
so just a question, let's ignore prisma for now, you can't even connect to supabase, correct?
Not when deployed, but I can from my laptop running like schema syncs and stuff
Let me try with a postgres client
I can connect via postico and I get this warning but I can ignore it
maybe this is not ignorable in production environments somehow?
AHA
I added ?sslmode=disable to the end of my connection string and it works
so it was definitely certs
I just have no idea how my local docker image was okay with bad certs but whatever
Now I just need to find a guide on fixing cert issues / adding certs to a docker container 😭
these kinds of things always intrigue me, I'm gonna try to write an app that will just test a connection to supabase
it works
https://github.com/brody192/supabase-test
So it sounds like a prisma ssl problem? It cannot gracefully degrade to non tls connection while in production so it dies
Thanks for doing that btw
can you connect to the database with Pg like I did?
I'm sure I could, I bet it properly handles invalid certs, but then I'm not using prisma :p
So ultimately, I blame Prisma
Bad prisma
using an old prisma version?
Nope, most up to date version
now what was that certificate supabase has for?
I saw some certificate in the dashboard or something
Yeah, I saw that too. I tried adding it to my dockerfile by hand and couldn't get it to be read
Planetscale requires ssl auth too for their connection string but I made that work by just installing the cert package from apt in my container
I don't know why prisma wouldn't be able to validate certs
I assume a configuration problem on my end. Still, my only outstanding question left is why did prisma only crash when deployed in a container in the cloud and not in a local container
I wonder if docker-compose does some magic there for me that I'm not aware of
I guess I better add a prisma test to my repo lol
I was going to PR that after work today, don't want you to have all the fun :p
I won't be on my computer for a while, so you're welcome to pr it since I've never used prisma before
Alright, built a replication. It works locally via npm and docker compose, but does not work when deployed to railway, just like my app
Railway Project ID: 761aff30-4067-4517-97e0-d02aea31fef1
GitHub
GitHub - cephalization/supabase-prisma-railway
Contribute to cephalization/supabase-prisma-railway development by creating an account on GitHub.
will take a look later
no worries if you have higher priority things to do
just some stuff around the house
I don't wanna be responsible for giving you a distraction from the dishes
real
node:18-slim - doesn’t work
node:18 - doesn’t work
node:16 - works
node:18.16.0-alpine3.17 - works
testing your repo with different node images, i dont know why some of those work and not others, but hopefully that can help you
I should have thought of that, thanks! I should try to get my app running in alpine anyway
Giving this more thought, specifying just something like “node:18-slim” could very easily lead inconsistency.
I imagine platforms could cache a different version of that image than what would you pull locally
yep railway does cache images, I wish I had a reason as to why some of those image don't work
Maybe they have older certificates baked into the OS?
gotta be something along those lines. but look on the bright side, it won't be that hard to switch your dockerfile you initially sent to use an alpine based image!
Yeah definitely. I’m satisfied with these findings, thanks for helping with research!
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
It was a prisma bug the whole time https://github.com/prisma/prisma/issues/10649
GitHub
Segmentation fault crash when using prisma client when using Postgr...
Bug description When trying to use findMany() (or any other method) on a model, prisma client crashes and the following line appears in my log: Segmentation fault (core dumped). This only happens i...
Solved in todays release
woohoo let's celebrate