Railway•10mo ago

Private Networking XMLRPC Server Connection Refused

I am using private networking for postgres and redis, but I also have an additional service that I want to connect to via private networking. However, when I try to establish the connection I get a connection refused error. This service uses a XMLRPC server and I do not control the source code for this service (unoserver), I simply deploy a dockerfile for the service. I am able to connect to the service if I use the railway TCP proxy, however I suspect this adds a significant performance penalty and incurs additional network egress/ingress costs since this service returns large responses (> 1mb) for every request. If I try to resolve the DNS record manually using socket.gethostbyname_ex I get the following error: socket.gaierror: [Errno -5] No address associated with hostname. This error makes it seem like the service is never attached to private networking. As far as I can tell this is a bug in railway, since every other service using private networking works fine. I've attached a screenshot of the relevant exception in the deployment logs.

31 Replies

Percy•10mo ago

Project ID: 46739752-566e-4538-85f5-d14c35ae1073

ReeceOP•10mo ago

46739752-566e-4538-85f5-d14c35ae1073

Adam•10mo ago

Are your services all in the same project?

ReeceOP•10mo ago

Yeah

Adam•10mo ago

Please add a 3 second sleep before your start command

ReeceOP•10mo ago

Frontend + Backend API + Backend Worker + Redis + Postgres + Custom Dockerfile

Adam•10mo ago

sleep 3; startcommand

ReeceOP•10mo ago

The connection is established only when a job is sent to the worker Long after the worker starts

Adam•10mo ago

ah understood you mentioned you try to resolve the dns manually, please try with just the private URL is there a different error?

ReeceOP•10mo ago

That error was when I used the private URL. (unoserver.railway.internal)

Adam•10mo ago

Unfortunately I'm out of ideas. This all sounds like it should be working. When you change to the public url, do you change anything else?

Brody•10mo ago

the "Uno" server needs to be listening on an ipv6 address, this is usually done by setting the host the app listens on to :: or [::]. currently the server is listening only on ipv4 and that's why you get connection refused. what kind of app is the "uno" server?

Krøn•10mo ago

how do we do that unoserver.railway.internal::PORT?

Brody•10mo ago

as mentioned above the host you would want to have your app listen on would be :: or [::]

Krøn•10mo ago

Noted I understand now Thank you

ReeceOP•10mo ago

ah interesting did not know that private networking requires ipv6 address binding. Let me try that. Here is a link to unoserver. It is a document conversion service. https://github.com/unoconv/unoserver

GitHub

GitHub - unoconv/unoserver

Contribute to unoconv/unoserver development by creating an account on GitHub.

Brody•10mo ago

yep the private network is ipv6 only

ReeceOP•10mo ago

neato Thanks for the help gang. Got it to work. Unfortunately unoserver only supports ipv4 address binding so I had to use socat to create a dual-stack proxy in the docker container, but it works now! Appreciate the help!

Brody•10mo ago

what server does unoserver use? I saw in those docs it allows you to pass in a host value

ReeceOP•10mo ago

It uses xmlrpc.server.SimpleXMLRPCServer. When they construct the server they use a ipv4-style tuple (len = 2) and not a ipv6-style tuple (len = 4) https://stackoverflow.com/questions/5358021/establishing-an-ipv6-connection-using-sockets-in-python https://github.com/unoconv/unoserver/blob/master/src/unoserver/server.py#L88

Stack Overflow

Establishing an IPv6 connection using sockets in python

I am trying to run this very basic socket example: import socket host = 'ipv6hostnamegoeshere' port=9091 ourSocket = socket.socket(socket.AF_INET6, socket.SOCK_STREAM, 0) ourSocket.connect((host,...

GitHub

unoserver/src/unoserver/server.py at master · unoconv/unoserver

Contribute to unoconv/unoserver development by creating an account on GitHub.

Brody•10mo ago

well yep that would definitely restrict you to ipv4

Krøn•10mo ago

@Brody i come once again

Krøn•10mo ago

after switching to :: my server starts and shuts down last time this happend was because i didnt set the port variable to 8001, now i dont know

ReeceOP•10mo ago

Are you using multiprocessing or async or both? Or threads?

Krøn•10mo ago

neither it worked i had to cancle my health check which im not sure if its a good idea i can have my gateway do it but its just a pain

Brody•10mo ago

what's your start command?

Krøn•10mo ago

none, i user a dockerfile

Brody•10mo ago

send it?

Krøn•10mo ago

FROM python:3.11

WORKDIR /app

COPY . /app

RUN pip install --no-cache -r requirements.txt

EXPOSE 8001

ARG SERVICE_DATABASE_URL
RUN echo $SERVICE_DATABASE_URL

ENV SERVICE_DATABASE_URL=$SERVICE_DATABASE_URL

RUN alembic upgrade head

CMD ["uvicorn", "app.main:app", "--host", "::", "--port", "8001"]

FROM python:3.11

WORKDIR /app

COPY . /app

RUN pip install --no-cache -r requirements.txt

EXPOSE 8001

ARG SERVICE_DATABASE_URL
RUN echo $SERVICE_DATABASE_URL

ENV SERVICE_DATABASE_URL=$SERVICE_DATABASE_URL

RUN alembic upgrade head

CMD ["uvicorn", "app.main:app", "--host", "::", "--port", "8001"]

ReeceOP•10mo ago

You should be using a start script for your app. You should not be running alembic in the docker file since it will only perform migrations when the container builds and not when the container is run. The startup script may look something like:

#!/bin/sh
set -e

PORT=${PORT:-8001}

alembic upgrade head

uvicorn app.main:app --host "::" --port "$PORT"

#!/bin/sh
set -e

PORT=${PORT:-8001}

alembic upgrade head

uvicorn app.main:app --host "::" --port "$PORT"

Then update the dockerfile to look something like:

FROM python:3.11

WORKDIR /app

COPY . /app

RUN pip install --no-cache -r requirements.txt

CMD ["/bin/sh", "/app/start.sh"]

FROM python:3.11

WORKDIR /app

COPY . /app

RUN pip install --no-cache -r requirements.txt

CMD ["/bin/sh", "/app/start.sh"]

As for why your app won't start, there are a few possible causes: - Does your app have an app module? If it does then in your source code root directory you will have a directory called app with a file called __init__.py in it. - If you have any life cycle hooks (on startup, on shutdown) make sure that you are not stopping the server - It is possible that you are doing something in your code that causes the worker process to be killed. This can happen if you allocate too much memory or use bugged python APIs. For example, I had an issue a couple years ago where using Thread and ThreadPoolExecutor in the same process caused python to silently shutdown like you are experiencing now. - If your asgi config is invalid, for example you have workers set to 0.

Krøn•10mo ago

ohh i understand, thank you very much

Gaming

Programming

Private Networking XMLRPC Server Connection Refused