Error connecting to database from deployment process
I'm getting an error connecting to a database instance from a different service during its deployment process. This process has worked pretty much flawlessly for two years, but now I'm getting this error when the deployment migration is performed from a Dockerfile:
Error: P1001: Can't reach database server at monorail.proxy.rlwy.net:10861
The existing service, still running, can connect fine. And I can connect to the database remotely as well.
project: 58cd7410-04d6-4c6e-a74d-fa7c78455231
service: a6f97426-e4a4-4286-a7cf-c259f98f5ff9Solution:Jump to solution
i can also summarize:
I was using the public network to run DB migrations during the build process. I started having connectivity issues when doing that. I made my service use the DB's private networking url and enabled the V2 runtime in settings and set the
RAILWAY_BETA_ENABLE_BUILD_V2 variable to 1 - this resolved the issue
(there was a bug in the health check, where it was connecting to the service in the ipv6 range, so i had to also enable that for my app - but that was a separate issue)...108 Replies
Project ID:
58cd7410-04d6-4c6e-a74d-fa7c78455231,a6f97426-e4a4-4286-a7cf-c259f98f5ff9to be clear, you are getting this error during build or during runtime?
during the build
i've tried re-deployment a couple of times.. it's possible there's an intermittent connection issue, i do not have adequate monitoring
Should I just try again?
I started another attempt
looked like it was able to connect for the 3rd attempt
try running your migrations during runtime with the private network
i'll work on that, thanks
so the deployment i did 10m ago was successful, I had not changed anything
I just changed the database variable to use the internal one, and i'm getting the same error I was getting earlier:
#12 1.987 Error: P1001: Can't reach database server at postgres.railway.internal:5432
well.. same, except the reference being to the internal one
is there some particular internal-networking thing I need to enable?change the runtime to v2 in the service settings
i'm not familiar with this setting, is there anything i should know?
yep you beat me to it
aight, deploying
same error
is your postgres database in the same project and environment?
yes
and are you now making the migration during runtime?
no, that's a bigger change, i need to work on that
okay then we can try something else for that
is the internal network not available during deployment?
you mean to ask if it is available during build, and the answer to that would be not by default, but lets try change that
add a service variable
RAILWAY_BETA_ENABLE_BUILD_V2=1
more info: https://railway.app/changelog/2024-05-24-builder-v2-beta#builder-v2-arrives-in-beta
for clarity the v2 runtime and v2 builder are separate systemssorry, yes, build
it does look like it got past the migration this time, so that appears to have worked
sounds like the BUILD_V2 thing will be a nice change for you guys
and me
the v2 builder and v2 runtime aim fix so many things and bring so many improvements
though.. it seems like it's failing the health check.. retrying 7 times now
what do your deploy logs look like?
deploy log looks like everything is working
service started successfully
what is the reason the health check fails?
Attempt #5 failed with service unavailable. Continuing to retry for 4m44s
Attempt #10 failed with service unavailable. Continuing to retry for 2m28s
etc
what kind of app is this?
no other log message, and nothing showing up in the deploy log
it's a node.js app
rest api
share your build and deploy logs please - https://bookmarklets.up.railway.app/log-downloader/
not really comfortable sharing the log in a public forum
is there sensitive information in your logs?
well.. it looks like i'm printing out the database url there.. so yeah
separately, something to fix
with the username and password?
yeah, the full thing
i mean you can edit the logs before you send them
do you have a PORT service variable set?
no
it defaults to 8080
can you show me your app.listen?
you mean the typescript code?
yes it would be code
why?
im just trying to work through some debug steps
which question could be answered by that?
this is probably what you are most likely interested in:
if you arent interested in debugging this ill have to ask you to revert back to the legacy builder and make your migrations during runtime before starting your app
this is parsing
process.env
i'm interested in debugging this, but I'm a bit confused how that could help.. the only thing i can think of is:
- either you think there's a bug in the code
- you are trying to figure out whether the app is really listening to port 8080
- you are wondering if the app is listening to only a particular host (it's listning to 0.0.0.0:8080)
and to those, i think it would be more efficient to ask me
it would perhaps be also easier to test out on the dev environment
since i could just push debug changes etcive been doing this for a while and screenshots of code are far more helpful than a written response
i'm sorry, i'm probably being rude, i'm sorry - in my defense it is 3 am here and i mostly just want to go to bed 🙂
thank you
now whats up with the build logs containing just the health check, and the deploy logs containing both the build and deploy logs?
yeah, that was weird to me.. because that's not what they look like in the UI
thats not an edit you did?
it's just how they came out using the bookmarklet
no
and i also had to click the bookmarklet twice.. first time it only downloaded that deploy log, second time it did this weird combo for the other file
i'm assuming it had something to do with the fact that i had the logs open at the time?
possibly i could try to look at the bookmarklet source and debug it
it calls railway's api, it doesnt scrape the web page
weird.. i can try clicking it again
but im not sure whats going on with the failed health check, can you try removing the health check for now
well.. now the bookmarklet says:
do you have a deployment open?
🤦♂️
same thing .. build_logs_xxx just has the output of the health check
interesting, ill have to look into this
yeah, if you open the deployment log.. it will give you the actual build log.. concatenated with the output of what should be the deployment log
i'm a bit worried that it will not work if i remove the health check
it might not
i'm fine with the currently running one just staying on for a while
until i can figure out this health check thing
i've got to go to bed - I should just start another thread tomorrow, ok?
this thread is good
ok, thanks a lot for your help ❤️
no problem!
fwiw I fixed the bookmarklet, you just have to delete your old one and drag over the new one -
https://bookmarklets.up.railway.app/log-downloader/
ah, nice
ok, so i've made the same changes in the dev environment, and it's failing the health check in the same way
but i can push code on to there to debug the issue
i'll start by removing the healthcheck to see if it all still works
--
Is anybody watching this thread?
Anyways, I'm still trying to debug this issue. I haven't made changes related to the active deployment, but it is now all of a sudden not able to connect to the database.
This is the same issue that's been happening on and off lately - and my initial suspicion for what was causing the failed migrations during deployment
Can't reach database server at monorail.proxy.rlwy.net:10861\n\nPlease make sure your database server is running at monorail.proxy.rlwy.net:10861.
This is a way bigger issue than anything related to the deploment process
I opened a new issue for this, since it's not really related to the build process
--
on to the previous issue, it looks like disabling the health check did not adversely affect things - and in the dev environment setting the database env var to the internal one + setting
- runtime: V2
- RAILWAY_BETA_ENABLE_BUILD_V2=1
with the disabled health check it really does seem like everything is working and the service does get deployed successfully
But i'd really like to have the health check there.. in cases where for some reason I make a mistake and the service doesn't actually work, even if the process / entry command gets called successfullyit was already 3:30am for me when I sent the message for the bookmarklet and I went to bed shortly after that
last night I went around enabling the v2 runtime on some services I had that were still using the legacy runtime, and my umami service started failing the health check with the same reason yours did, did you say your app was a nest app?
side note, I very much appreciate all the testing you've done!!
went to bed before i did more testing last night, but got back to it now and i found that having umami listen on
:: instead of 0.0.0.0 allowed the health check to pass and the service to come online without issue (when on v2 runtime)
cc @char8 runtime v2 health check odditieswow, dedication, i like it
did you say your app was a nest app?not a nest app, but a fastify app - some nest setups use fastify - not sure if that's related, but it would seem strange if what software was running could affect the health check
it bugged me that it didnt work
i feel you
ah, listening on the ipv6 address space instead
that would make sense
i mean it shouldn't make sense, the health check should work for either ipv4 or ipv6
the service is probably not listening to incoming connections on ipv6, let me try this
i'm using 0.0.0.0 which doesn't listen on ipv6 so that would do it
ideally you would be able to listen on either ipv4 or ipv6 and the health check would work regardless
yeah, setting to
:: should work, good call
i'm testing that right now
if this turns out to be the solution we are goldeni should have read through this: https://docs.railway.app/reference/private-networking 🙂 could have saved me some hours
no no, you should not need to listen on ipv6 for the health check, thats a bug
but thats the current work around
it literally says:
You will need to bind to a IPv6 port to receive traffic on the private network.ahh.. i see what you mean
right but the health check is not private networking
riight
so this line that says that private networking is not available during the build phase, that's what will no longer be the case for the new builder?
thats correct
the v2 builder is now a part of the private network
i remember fighing this a bit when i was setting things up initially.. and distinctly remember giving up on the private network for the db connection, probably because of this..
so i might be able to make my db private-network only?
thats the goal!
that's super nice
btw i really have been happy with railway in general and it has been really solid for me.. been using it now for two years - i think it's an excellent product and kind of overlooked in the dev community - fly.io etc grabbing all the attention
even though i dont work for railway, i appreciate that
you don't? you're just really helpful?
how do you know about the
RAILWAY_BETA_ENABLE_BUILD_V2 flag etc?
and the bookmarklet .. that's just your own thing to made to make it easier to help people?well i am a mod so i was told many weeks ago about that, but the info has since been made public
https://railway.app/changelog/2024-05-24-builder-v2-beta#builder-v2-arrives-in-beta
and yep the bookmarklet is my own thing, screenshots of code may be helpfull, but screenshots of build logs are rarely helpful
well, who do you work for then?
no one
you're at school?
nope
just doing this for the pleasure of helping people?
yeah!
🙇♂️
well, i bought you some trains - let me know if there's ever anything i can help you with
thank you so much!! i appreciate that!!
btw the service was deployed successfully after changing the listening address.. so all problems resolved
thats awsome!
are you now using the private network to run your migrations during build and for regular communication during runtime?
yes
or i'm deploying that to production atm
works on dev
sounds good!
btw the new build thing injects a bunch of extra lines into the logs:
do you know if there's a way to omit these?
i don't see this when building the docker image locally, so i'm not sure what that is
the v2 builder doesnt use buildx it uses something else entirely, and its a little more verbose, theres no way to omit that info
ok, thanks anyways
and thanks a lot for your help, i'm running on private networking both in the build and in the runtime 👍 both on dev & production
how do i mark this thread as resolved?
thats great, glad the health check issue was a simple fix
thats something a mod can only do, ill go find the best message to mark as the solution
to recap -
enable the v2 builder and the v2 runtime, if your health check fails, have your app listen on the
:: host
well the bot is offline so i will mark that as the answer when the bot comes back onlineSolution
i can also summarize:
I was using the public network to run DB migrations during the build process. I started having connectivity issues when doing that. I made my service use the DB's private networking url and enabled the V2 runtime in settings and set the
RAILWAY_BETA_ENABLE_BUILD_V2 variable to 1 - this resolved the issue
(there was a bug in the health check, where it was connecting to the service in the ipv6 range, so i had to also enable that for my app - but that was a separate issue)yes, or your recap, sorry, i started writing before i saw that
yours is better, ill use yours
sure 👍 either way