Topics

Railway•2y ago

Cannot connect to supabase db from railway

I just ported my project from a planetscale db to a supabase postgresdb. Everything works fine locally and within Dockerfile using the new connection string. Added connection string to railway and suddenly my service no longer works and crash loops due to prisma disconnecting. I've only seen the prisma log once however. I triple checked that I am using the right DATABASE_URL in my railway service definition. Project ID ef7fcff6-4369-4ace-9d06-f74fd0f47a4e

65 Replies

Percy•2y ago

Project ID: ef7fcff6-4369-4ace-9d06-f74fd0f47a4e

Percy•2y ago

You might find these helpful: - Cannot connect to the database. - is it possible to connect to a railway database from local project? - Unable to connect to PostgresSQL

⚠️ experimental feature

Brody•2y ago

send a screenshot of your service variables and send your dockerfile please

cephalizationOP•2y ago

FROM node:18-slim AS builder
# Set working directory
WORKDIR /app
RUN yarn global add turbo
COPY . .
RUN turbo prune --scope=bot --docker

FROM node:18-slim AS installer

RUN apt-get update
RUN apt-get install -y ffmpeg python3 make g++ openssl libssl-dev ca-certificates
RUN update-ca-certificates
WORKDIR /app

# First install the dependencies (as they change less often)
COPY .gitignore .gitignore
COPY --from=builder /app/out/json/ .
COPY --from=builder /app/out/yarn.lock ./yarn.lock
RUN yarn install

# Build the project
COPY --from=builder /app/out/full/ .
COPY turbo.json turbo.json
RUN yarn turbo run build --force --filter=bot...

FROM node:18-slim as runner
ARG DATABASE_URL
ARG REDIS_URL
ARG OPENAI_API_KEY
ARG ELEVENLABS_API_KEY
ARG DEEPGRAM
ARG TOKEN
EXPOSE 45000-60000

WORKDIR /app
COPY --from=installer /app/apps/bot/dist/* .
COPY --from=installer /app/apps/bot/package.json .
COPY --from=installer /app/node_modules/ /app/node_modules/
COPY --from=installer /app/packages/ /app/packages/
COPY --from=installer /etc/ssl/certs /etc/ssl/certs

CMD ["node", "bot.js"]

FROM node:18-slim AS builder
# Set working directory
WORKDIR /app
RUN yarn global add turbo
COPY . .
RUN turbo prune --scope=bot --docker

FROM node:18-slim AS installer

RUN apt-get update
RUN apt-get install -y ffmpeg python3 make g++ openssl libssl-dev ca-certificates
RUN update-ca-certificates
WORKDIR /app

# First install the dependencies (as they change less often)
COPY .gitignore .gitignore
COPY --from=builder /app/out/json/ .
COPY --from=builder /app/out/yarn.lock ./yarn.lock
RUN yarn install

# Build the project
COPY --from=builder /app/out/full/ .
COPY turbo.json turbo.json
RUN yarn turbo run build --force --filter=bot...

FROM node:18-slim as runner
ARG DATABASE_URL
ARG REDIS_URL
ARG OPENAI_API_KEY
ARG ELEVENLABS_API_KEY
ARG DEEPGRAM
ARG TOKEN
EXPOSE 45000-60000

WORKDIR /app
COPY --from=installer /app/apps/bot/dist/* .
COPY --from=installer /app/apps/bot/package.json .
COPY --from=installer /app/node_modules/ /app/node_modules/
COPY --from=installer /app/packages/ /app/packages/
COPY --from=installer /etc/ssl/certs /etc/ssl/certs

CMD ["node", "bot.js"]

cephalizationOP•2y ago

cephalizationOP•2y ago

I changed some src code between this change as well, but like I said it builds just fine locally using this docker image and the new env var cleared all caches too

Brody•2y ago

you dont need to define ARGs in dockerfile if you arent using the args in the dockerfile you also dont need an EXPOSE you also shouldn't need a dockerfile_path variable

cephalizationOP•2y ago

I have a nested turborepo with multiple dockerfiles I added the expose to debug some discord websocket stuff

Brody•2y ago

add logging for the environment variables to try to see whats going wrong theres better ways to define it, but ig a variable will work too

cephalizationOP•2y ago

yeah, my main concern is that there is some kind of networking issue preventing connections to supabase or perhaps some caching going on causing my old prisma client to be reused The dockerfile is unchanged, new database is synced with prisma schema and works with a local bot I can try deleting database_url and re-adding maybe

Brody•2y ago

logging!!!

cephalizationOP•2y ago

hrmmm

cephalizationOP•2y ago

it's crashing super fast before my logging goes off it is logging the correct connection url too

Brody•2y ago

seems like a code issue tbh

cephalizationOP•2y ago

that only occurs in railway? What could work in docker but not work when shipped up to railway is what confuses me

Brody•2y ago

i know, sounds odd but i see this all the time

cephalizationOP•2y ago

I can imagine I will poke at it more tomorrow, thanks for the quick check in I've spun up a new postgres database from railway, changed my env vars to that, and now my app works again so there is some kind of connection issue with the string supabase gave me and the container running in railway Because the same connection string works when injected with .env and run with yarn and with ARGS when run in docker image I pretty much just went with supabase for my db so that I can take advantage of the vector extension. If I can add that to the railway db I could co-locate but I'm very confused by this issue overall

Brody•2y ago

was your connection string unquoted? I've seen someone have that issue before railways environment key value parser might not be the most robust

cephalizationOP•2y ago

hmm it does not have quotes around it I think I tried adding them but let me try again fwiw I set up an identical deployment in render this morning to try and rule out railway and it happened there as well so something about this connection string just does not work in the cloud 🤪

Brody•2y ago

I have seen other peoples app on railway use supabase, so I assure you railway isn't blocking a connection or anything of that nature

cephalizationOP•2y ago

I finally found some relevant logs on supabase C-0xaaaabe0cb810: pgbouncer/[email protected]:47242 pooler error: SASL authentication failed So it sounds to me like my docker container does not have the correct certificates configured perhaps. Prisma was not telling me anything about auth errors, just crash looping my app

Brody•2y ago

well you know what that means!! time to move to a Dockerfile!

cephalizationOP•2y ago

I guess I could deploy a postgres instance inside of a dockerfile and serve it on railway but I would lose the nice table viewer and stuff

Brody•2y ago

you actually couldn't do that, there's no persistent storage

cephalizationOP•2y ago

man I am so perplexed, I've exhausted most of my options at this point and confirmed that it's not a railway issue

Brody•2y ago

I'm talking about a dockerfile for your app not the database

cephalizationOP•2y ago

I do that That's why I've been so confused The dockerfile works locally but not when deployed to render or railway and only with regards to connecting to the supabase database via prisma

Brody•2y ago

since you build with a dockerfile, railway now doesn't have anything to do with any certificate stuff

cephalizationOP•2y ago

right but what in my local env could make certificate stuff work when given the same dockerfile is my confusion

Brody•2y ago

makes me want to test a supabase connection for myself

cephalizationOP•2y ago

I would be extremely grateful. All I did was - Make a new supabase project - Get the connection string / password - Add it as DATABASE_URL var - Try to run a prisma project with it I can't easily try to run my app "baremetal" in railway to eliminate docker as a suspect because my node app requires a few python deps for audio processing

Brody•2y ago

so just a question, let's ignore prisma for now, you can't even connect to supabase, correct?

cephalizationOP•2y ago

Not when deployed, but I can from my laptop running like schema syncs and stuff Let me try with a postgres client

cephalizationOP•2y ago

I can connect via postico and I get this warning but I can ignore it

cephalizationOP•2y ago

maybe this is not ignorable in production environments somehow? AHA I added ?sslmode=disable to the end of my connection string and it works so it was definitely certs I just have no idea how my local docker image was okay with bad certs but whatever Now I just need to find a guide on fixing cert issues / adding certs to a docker container 😭

Brody•2y ago

these kinds of things always intrigue me, I'm gonna try to write an app that will just test a connection to supabase it works https://github.com/brody192/supabase-test

cephalizationOP•2y ago

So it sounds like a prisma ssl problem? It cannot gracefully degrade to non tls connection while in production so it dies Thanks for doing that btw

Brody•2y ago

can you connect to the database with Pg like I did?

cephalizationOP•2y ago

I'm sure I could, I bet it properly handles invalid certs, but then I'm not using prisma :p So ultimately, I blame Prisma Bad prisma

Brody•2y ago

using an old prisma version?

cephalizationOP•2y ago

Nope, most up to date version

Brody•2y ago

now what was that certificate supabase has for? I saw some certificate in the dashboard or something

cephalizationOP•2y ago

Yeah, I saw that too. I tried adding it to my dockerfile by hand and couldn't get it to be read Planetscale requires ssl auth too for their connection string but I made that work by just installing the cert package from apt in my container

Brody•2y ago

I don't know why prisma wouldn't be able to validate certs

cephalizationOP•2y ago

I assume a configuration problem on my end. Still, my only outstanding question left is why did prisma only crash when deployed in a container in the cloud and not in a local container I wonder if docker-compose does some magic there for me that I'm not aware of

Brody•2y ago

I guess I better add a prisma test to my repo lol

cephalizationOP•2y ago

I was going to PR that after work today, don't want you to have all the fun :p

Brody•2y ago

I won't be on my computer for a while, so you're welcome to pr it since I've never used prisma before

cephalizationOP•2y ago

Alright, built a replication. It works locally via npm and docker compose, but does not work when deployed to railway, just like my app Railway Project ID: 761aff30-4067-4517-97e0-d02aea31fef1

cephalizationOP•2y ago

https://github.com/cephalization/supabase-prisma-railway

GitHub

GitHub - cephalization/supabase-prisma-railway

Contribute to cephalization/supabase-prisma-railway development by creating an account on GitHub.

Brody•2y ago

will take a look later

cephalizationOP•2y ago

no worries if you have higher priority things to do

Brody•2y ago

just some stuff around the house

cephalizationOP•2y ago

I don't wanna be responsible for giving you a distraction from the dishes WorrySalute

WorrySalute

Brody•2y ago

real node:18-slim - doesn’t work node:18 - doesn’t work node:16 - works node:18.16.0-alpine3.17 - works testing your repo with different node images, i dont know why some of those work and not others, but hopefully that can help you

cephalizationOP•2y ago

I should have thought of that, thanks! I should try to get my app running in alpine anyway Giving this more thought, specifying just something like “node:18-slim” could very easily lead inconsistency. I imagine platforms could cache a different version of that image than what would you pull locally

Brody•2y ago

yep railway does cache images, I wish I had a reason as to why some of those image don't work

cephalizationOP•2y ago

Maybe they have older certificates baked into the OS?

Brody•2y ago

gotta be something along those lines. but look on the bright side, it won't be that hard to switch your dockerfile you initially sent to use an alpine based image!

cephalizationOP•2y ago

Yeah definitely. I’m satisfied with these findings, thanks for helping with research!

Unknown User•2y ago

Message Not Public

Sign In & Join Server To View

cephalizationOP•2y ago

It was a prisma bug the whole time https://github.com/prisma/prisma/issues/10649

GitHub

Segmentation fault crash when using prisma client when using Postgr...

Bug description When trying to use findMany() (or any other method) on a model, prisma client crashes and the following line appears in my log: Segmentation fault (core dumped). This only happens i...

cephalizationOP•2y ago

Solved in todays release

Brody•2y ago

woohoo let's celebrate

cephalizationOP•2y ago

kekamid

kekamid

kekamid

Hang out with other likeminded developers & talk about all things https://railway.app on the Railway Community Server.

25KMembers

View on Discord

Want results from more Discord servers?

Add your server