R
Railway17mo ago
Nlea

Connecting prometheus instance with python app via internal network

Hey everyone, I am new to Railway. I deployed a Python App to railway via GitHub repo and Dockerimage and did the same for a Prometheus instance in a separated repo. Both apps are up and running. Within the Python app I have set the variable PORT = 80 because without this setting I would have been not able to access the app. The python app provides an endpoint /metrics form there I want the prometheus instance to poll the metrics. To do so one can configure a prometheus.yml. I managed that Prometheus is connecting to my app via the public domain, but I would like to have it connect via the internal network. I tried multiple things but it is still failing. My prometheus.yml looks like this:
scrape_configs:
- job_name: internal-endpoint
metrics_path: /metrics
static_configs:
# Replace the port with the port your /metrics endpoint is running on
- targets: ['fastapi-on-railway.railway.internal']
# For a real deployment, you would want the scrape interval to be
# longer but for testing, you want the data to show up quickly
scrape_interval: 200ms

- job_name: external-endpoint
metrics_path: /metrics
static_configs:
# Replace the port with the port your /metrics endpoint is running on
- targets: ['fastapi-on-railway-production.up.railway.app:443']
scheme: https
# For a real deployment, you would want the scrape interval to be
# longer but for testing, you want the data to show up quickly
scrape_interval: 200ms
scrape_configs:
- job_name: internal-endpoint
metrics_path: /metrics
static_configs:
# Replace the port with the port your /metrics endpoint is running on
- targets: ['fastapi-on-railway.railway.internal']
# For a real deployment, you would want the scrape interval to be
# longer but for testing, you want the data to show up quickly
scrape_interval: 200ms

- job_name: external-endpoint
metrics_path: /metrics
static_configs:
# Replace the port with the port your /metrics endpoint is running on
- targets: ['fastapi-on-railway-production.up.railway.app:443']
scheme: https
# For a real deployment, you would want the scrape interval to be
# longer but for testing, you want the data to show up quickly
scrape_interval: 200ms
I tried different target options for the internal one including: 0.0.0.0:80 , fastapi-on-railway.railway.internal:80 and basically fastapi-on-railway. If I look into the Prometheus frontend it takes the endpoint but reports that the service is down. Has anyone an idea how to make this connection within the Railway network? This would be amazing! Thank you and cheers Nele
Solution:
Hey everyone, I am new to Railway. I deployed a Python App to railway via GitHub repo and Dockerimage and did the same for a Prometheus instance in a separated repo. Both apps are up and running. Within the Python app I have set the variable PORT = 80 because without this setting I would have been not able to access the app. The python app provides an endpoint /metrics form there I want the prometheus instance to poll the metrics. ...
Jump to solution
18 Replies
Percy
Percy17mo ago
Project ID: b2a21226-a962-4555-94d3-d08d309e1c32
Nlea
NleaOP17mo ago
Project ID: b2a21226-a962-4555-94d3-d08d309e1c32
Brody
Brody17mo ago
is this python app running with gunicorn? uvicorn?
Nlea
NleaOP17mo ago
uvicorn
Brody
Brody17mo ago
whats your start command for it?
Nlea
NleaOP17mo ago
from the docker image it is: CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "80"]
Brody
Brody17mo ago
use this instead
CMD uvicorn app.main:app --host '::' --port $PORT
CMD uvicorn app.main:app --host '::' --port $PORT
then in your railway service variables set PORT variable to 8080 and then this should be the target url
- targets: ['http://fastapi-on-railway.railway.internal:8080']
- targets: ['http://fastapi-on-railway.railway.internal:8080']
p.s. no need to put the cmd command into an array
Nlea
NleaOP17mo ago
cool thanks! I will give it a try It works if I set the Variable Port to 8080 manually in the railway UI... but also now the public domain is not working anymore and I get back an "application failed" error. Do you might have an idea how both points can work?
Brody
Brody17mo ago
i will look into it uvicorn doesn't seem to easily support dual stack binding, this is needed to access your app externally (ipv4) and to also access your app internally (ipv6) the easiest solution imo would be to use hypercorn instead: remove uvicorn from your requirements.txt file install hypercorn pip install hypercorn pin hypercorn to a fixed version in your requiremernts.txt file hypercorn==0.14.4 change your CMD command CMD hypercorn app.main:app --bind "[::]:$PORT" notes: keep the PORT = 8080 service variable your targets in your prometheus.yml stay the same example fast api server using hypercorn external: https://fast-api-hypercorn-test.up.railway.app/ internal: https://utilities.up.railway.app/request-test?url=http://fast-api-hypercorn-test.railway.internal:8080
Nlea
NleaOP17mo ago
cool thanks! This is super helpful!
Brody
Brody17mo ago
no problem, if you get stuck switching over, feel free to ask
Nlea
NleaOP16mo ago
Hey @Brody , it is me again. I have now a second app I would like to connect to my running Prometheus instance on Railway. The Dockerfile uses: CMD uvicorn main:app --host='::' --port=$PORT And I set an Enviroment variable via the railway UI : PORT = 8000 In the Prometheus yaml I specified a second target: - targets: ['fastapi-on-railway.railway.internal:8000'] I see that the Prometheus instance tries to connect to the new target but it gets the following error: Get "http://fastapi-on-railway.railway.internal:8000/metrics": dial tcp [fd12:860e:5888::b:ab40:b92b]:8000: connect: connection refused I think I did everything the same as for the first application but I wondering why I can't connect it towards the second app. Am I missing something? Do you maybe have an idea?
Brody
Brody16mo ago
can I see a screenshot of the deployment logs of this new fast api service?
Nlea
NleaOP16mo ago
sure
Brody
Brody16mo ago
I am truly stumped
Nlea
NleaOP16mo ago
mh any idea how I can debugg this? It is super odd as the other app is working with the same setup. okay it started working now... no idea why but happy that it did!
Brody
Brody16mo ago
very strange
simoelalj
simoelalj12mo ago
I'm having exactly the same issue :
Get "http://prometheus-bull-exporter.railway.internal:9538/metrics": dial tcp [fd12:9515:ae15:0:8000:2:3d15:152a]:9538: connect: connection refused
Get "http://prometheus-bull-exporter.railway.internal:9538/metrics": dial tcp [fd12:9515:ae15:0:8000:2:3d15:152a]:9538: connect: connection refused
Did you managed to find what went wrong? EDIT: After some digging I found out that the metric exporters I was using was binding the server on the 0.0.0.0 IP instead of ::. So I had to fork that image and edit the value. And it it worked. (that was an Aha moment understanding that railway internal network uses only ipv6 addresses)
Want results from more Discord servers?
Add your server