Railway•16mo ago

Connecting prometheus instance with python app via internal network

Hey everyone, I am new to Railway. I deployed a Python App to railway via GitHub repo and Dockerimage and did the same for a Prometheus instance in a separated repo. Both apps are up and running. Within the Python app I have set the variable PORT = 80 because without this setting I would have been not able to access the app. The python app provides an endpoint /metrics form there I want the prometheus instance to poll the metrics. To do so one can configure a prometheus.yml. I managed that Prometheus is connecting to my app via the public domain, but I would like to have it connect via the internal network. I tried multiple things but it is still failing. My prometheus.yml looks like this:

scrape_configs:
  - job_name: internal-endpoint
    metrics_path: /metrics
    static_configs:
      # Replace the port with the port your /metrics endpoint is running on
      - targets: ['fastapi-on-railway.railway.internal']
    # For a real deployment, you would want the scrape interval to be
    # longer but for testing, you want the data to show up quickly
    scrape_interval: 200ms

  - job_name: external-endpoint
    metrics_path: /metrics
    static_configs:
      # Replace the port with the port your /metrics endpoint is running on
      - targets: ['fastapi-on-railway-production.up.railway.app:443']
    scheme: https
    # For a real deployment, you would want the scrape interval to be
    # longer but for testing, you want the data to show up quickly
    scrape_interval: 200ms

scrape_configs:
  - job_name: internal-endpoint
    metrics_path: /metrics
    static_configs:
      # Replace the port with the port your /metrics endpoint is running on
      - targets: ['fastapi-on-railway.railway.internal']
    # For a real deployment, you would want the scrape interval to be
    # longer but for testing, you want the data to show up quickly
    scrape_interval: 200ms

  - job_name: external-endpoint
    metrics_path: /metrics
    static_configs:
      # Replace the port with the port your /metrics endpoint is running on
      - targets: ['fastapi-on-railway-production.up.railway.app:443']
    scheme: https
    # For a real deployment, you would want the scrape interval to be
    # longer but for testing, you want the data to show up quickly
    scrape_interval: 200ms

I tried different target options for the internal one including: 0.0.0.0:80 , fastapi-on-railway.railway.internal:80 and basically fastapi-on-railway. If I look into the Prometheus frontend it takes the endpoint but reports that the service is down. Has anyone an idea how to make this connection within the Railway network? This would be amazing! Thank you and cheers Nele

Solution:

Jump to solution

18 Replies

Percy•16mo ago

Project ID: b2a21226-a962-4555-94d3-d08d309e1c32

Nlea•16mo ago

Project ID: b2a21226-a962-4555-94d3-d08d309e1c32

Brody•16mo ago

is this python app running with gunicorn? uvicorn?

Nlea•16mo ago

uvicorn

Brody•16mo ago

whats your start command for it?

Nlea•16mo ago

from the docker image it is: CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "80"]

Brody•16mo ago

use this instead

CMD uvicorn app.main:app --host '::' --port $PORT

CMD uvicorn app.main:app --host '::' --port $PORT

then in your railway service variables set PORT variable to 8080 and then this should be the target url

- targets: ['http://fastapi-on-railway.railway.internal:8080']

- targets: ['http://fastapi-on-railway.railway.internal:8080']

p.s. no need to put the cmd command into an array

Nlea•16mo ago

cool thanks! I will give it a try It works if I set the Variable Port to 8080 manually in the railway UI... but also now the public domain is not working anymore and I get back an "application failed" error. Do you might have an idea how both points can work?

Brody•16mo ago

i will look into it uvicorn doesn't seem to easily support dual stack binding, this is needed to access your app externally (ipv4) and to also access your app internally (ipv6) the easiest solution imo would be to use hypercorn instead: remove uvicorn from your requirements.txt file install hypercorn pip install hypercorn pin hypercorn to a fixed version in your requiremernts.txt file hypercorn==0.14.4 change your CMD command CMD hypercorn app.main:app --bind "[::]:$PORT" notes: keep the PORT = 8080 service variable your targets in your prometheus.yml stay the same example fast api server using hypercorn external: https://fast-api-hypercorn-test.up.railway.app/ internal: https://utilities.up.railway.app/request-test?url=http://fast-api-hypercorn-test.railway.internal:8080

Nlea•16mo ago

cool thanks! This is super helpful!

Brody•16mo ago

no problem, if you get stuck switching over, feel free to ask

Nlea•15mo ago

Hey @Brody , it is me again. I have now a second app I would like to connect to my running Prometheus instance on Railway. The Dockerfile uses: CMD uvicorn main:app --host='::' --port=$PORT And I set an Enviroment variable via the railway UI : PORT = 8000 In the Prometheus yaml I specified a second target: - targets: ['fastapi-on-railway.railway.internal:8000'] I see that the Prometheus instance tries to connect to the new target but it gets the following error:

Get "http://fastapi-on-railway.railway.internal:8000/metrics": dial tcp [fd12:860e:5888::b:ab40:b92b]:8000: connect: connection refused

I think I did everything the same as for the first application but I wondering why I can't connect it towards the second app. Am I missing something? Do you maybe have an idea?

Brody•15mo ago

can I see a screenshot of the deployment logs of this new fast api service?

Nlea•15mo ago

sure

Brody•15mo ago

I am truly stumped

Nlea•15mo ago

mh any idea how I can debugg this? It is super odd as the other app is working with the same setup. okay it started working now... no idea why but happy that it did!

Brody•15mo ago

very strange

simoelalj•12mo ago

I'm having exactly the same issue :

Get "http://prometheus-bull-exporter.railway.internal:9538/metrics": dial tcp [fd12:9515:ae15:0:8000:2:3d15:152a]:9538: connect: connection refused

Get "http://prometheus-bull-exporter.railway.internal:9538/metrics": dial tcp [fd12:9515:ae15:0:8000:2:3d15:152a]:9538: connect: connection refused

Did you managed to find what went wrong? EDIT: After some digging I found out that the metric exporters I was using was binding the server on the 0.0.0.0 IP instead of ::. So I had to fork that image and edit the value. And it it worked. (that was an Aha moment understanding that railway internal network uses only ipv6 addresses)

Gaming

Programming

Connecting prometheus instance with python app via internal network