Is private networking down?
Hey is private networking down or something?
Because currently all my builds are crashing because they can't communicate through private networking, host names don't resolve
19 Replies
Project ID:
N/AAFAIK the private network isn't available during build
It's not during build, it's after the build, when the app is running
oh you said builds
what errors are you getting
Oh sorry I meant my services are crashing
Project ID: 152941be-0212-4cbc-8977-c7d8fa719760
stderr: 'pg_dump: error: connection to database "(database-nane)" failed: could not translate host name "(service-name).railway.internal" to address: Name does not resolve\n'
}
And also hasura API service is failing with a similar error
Error from hasura:
connection error","internal":"could not translate host name "(service-name)" to address: Name or service not known
I think I know the issue, private network initialisation takes about 2 seconds on average after container start, this can also effect calls to external APIs, add a timeout or increase DNS resolution time or add a retry mechanism
Hasura has a retry mechanism and it still fails
It was working about half an hour ago, but now after redeploying all services, they can't communicate through private networking
oh well thats not ideal
redeploy your services again? if the problem persists i will flag the team
Sadly, the problem still persists
Thread has been flagged to Railway team by @Brody.
did you rename the services by chance? I see some lookups with '-' in the service name, but the service names have '_' in them.
Yes I did
I would advise renaming everything to have
- in them, going to track down our validation logic - but _ definitely will cause issues in dns names, so we're probably missing some validation to prevent it
the service name is used to create the dns name, I think we had a warning when you try to rename the service - will follow that up as wellCurrently only the service name has but the private networking says its hostname containing -
If I renamed the service name to contain would it work?
if you could just rename the private networking endpoints to all have dashes
- and no underscores, and then reference stuff using that DNS name, that should work. There's an edit button on hover.
Did you rename both the endpoint and the service or just the service?
trying to work out how we got the service into this state since we do validate against underscores 😅
While renaming the endpoint now I discovered it had
Reverting back to - makes it work
Sorry for this silly mistake and thank you so much❤️❤️❤️
Yeah I noticed this the first time I renamed the endpoint but I don't remember that I renamed the endpoint to have again
i would have never had guessed that
no worries at all - We should definitely defend against that, going to to ticket internally to root cause. Cos it doesn't let me rename stuff with an underscore.
