Railway•15mo ago

railway internal network is unreadable

Solution:

try setting the ENABLE_ALPINE_PRIVATE_NETWORKING=true envvar per the docs on all the alpine containers?

Jump to solution

26 Replies

Percy•15mo ago

Project ID: 1f736e16-6743-4b65-ac36-e7b8aca242da

waltcowOP•15mo ago

projectID : 1f736e16-6743-4b65-ac36-e7b8aca242da

Brody•15mo ago

is that the correct domain name for the service?

waltcowOP•15mo ago

Brody•15mo ago

at what point in your apps lifecycle do you make a call this this internal service

waltcowOP•15mo ago

exactly the internal domain i use frontend web framework nextjs, and proxy api/* the backend internal sever call 'dify-flask'

waltcowOP•15mo ago

Brody•15mo ago

I'm sorry but that doesn't quite answer the question

waltcowOP•15mo ago

it use nodejs proxy to backend api server log show dns lookup fail

Brody•15mo ago

at what point in your apps lifecycle do you make a call this this internal service

waltcowOP•15mo ago

when browser call http://exapmle.com/api/foo/bar , nodejs proxy to backend http://dify-flask.railway.internal:5001/foo/bar

Brody•15mo ago

okay thank you has this exact code worked in the past?

waltcowOP•15mo ago

yes

Brody•15mo ago

have you tried a redoployment?

waltcowOP•15mo ago

redoployment twice . both frontend and backend

Brody•15mo ago

@fp could there be a problem with private network resolution?

Faraz•15mo ago

Interesting. Let me raise internally and find out.

char8•15mo ago

Is the container using alpine by chance? Or musl libc? The DNS resolution for that is slightly different and we have a note in the docs about it: https://docs.railway.app/reference/private-networking#workaround-for-alpine-based-images (We're working on a fix for that particular problem)

waltcowOP•15mo ago

FROM node:18.17.0-alpine

Solution

char8•15mo ago

try setting the ENABLE_ALPINE_PRIVATE_NETWORKING=true envvar per the docs on all the alpine containers?

waltcowOP•15mo ago

OK, i try it now

char8•15mo ago

to give some background, we put both our internal DNS server and a public DNS resolver in resolv.conf . The dns lookup functions in glibc will treat the second entry as a fallback (desired behaviour since we want public DNS to work even if our internal resolver is unavailable), MUSL libc (used by alpine) is different: it sends out parallel requests to all resolvers in resolv.conf and errors on the first failure. Because public DNS doesn't have the railway.internal zone, privnet lookups can fail on Alpine.

waltcowOP•15mo ago

it works now

char8•15mo ago

what the magic envvar above does is it removes the fallback resolv.conf entry, the only caveat with this is that it means public DNS resolutions are only available with privnet initialization (which is usually ~ 100ms after container startup). Again this sucks, so we're actively playing with a fix for it

waltcowOP•14mo ago

network issue occur again today

at connResetException (node:internal/errors:720:14)

at Socket.socketCloseListener (node:_http_client:474:25)

at Socket.emit (node:events:526:35)

at TCP.<anonymous> (node:net:323:12)

at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {

code: 'ECONNRESET'

}

Error: socket hang up

at connResetException (node:internal/errors:720:14)

at Socket.socketCloseListener (node:_http_client:474:25)

at Socket.emit (node:events:526:35)

at TCP.<anonymous> (node:net:323:12)

at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {

code: 'ECONNRESET'

}

at connResetException (node:internal/errors:720:14)

at Socket.socketCloseListener (node:_http_client:474:25)

at Socket.emit (node:events:526:35)

at TCP.<anonymous> (node:net:323:12)

at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {

code: 'ECONNRESET'

}

Error: socket hang up

at connResetException (node:internal/errors:720:14)

at Socket.socketCloseListener (node:_http_client:474:25)

at Socket.emit (node:events:526:35)

at TCP.<anonymous> (node:net:323:12)

at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {

code: 'ECONNRESET'

}

@char8

Brody•14mo ago

do not ping the team #🛂｜readme #5

Gaming

Programming

railway internal network is unreadable