NGINX, Uvicorn and FastAPI setup not working
I'm going to put as much information here because I'm so lost and hopefully it makes helping me easier.
I've got an uvicorn server running and have verified it works when I ssh into the pod.
Here's my nginx config
I've put that into /etc/nginx/sites-available/ and made a symlink to /etc/nginx/sites-enabled/
netstat gives this
So I can see the uvicorn server is listening, but nginx isn't listening on 8888
I've configured my pod to expose tcp port 8888


106 Replies
also I run nginx -s reload before starting my uvicorn server
your nginx service inst running
maybe, because nginx isnt listening
or theres some config error in your nginx

I'm not sure how to confirm it's running
but here's some command output and it still doesn't listen to the port i specified
maybe looking at the logs from the service will help
how can i do that?
cat /var/log/nginx/error.log
2025/04/02 19:12:36 [notice] 440#440: signal process started
2025/04/02 19:25:16 [notice] 747#747: signal process started
2025/04/02 19:33:27 [notice] 964#964: signal process started
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:9091 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:3001 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:7861 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:8081 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:8001 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:7270 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:9091 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:3001 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:7861 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:8081 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:8001 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:7270 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:9091 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:3001 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:7861 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:8081 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:8001 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:7270 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:9091 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:3001 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:7861 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:8081 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:8001 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:7270 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:9091 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:3001 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:7861 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:8081 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:8001 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: bind() to 0.0.0.0:7270 failed (98: Unknown error)
2025/04/03 12:46:52 [emerg] 1436#1436: still could not bind()
2025/04/03 12:47:04 [notice] 1491#1491: signal process started
2025/04/03 12:50:46 [notice] 1672#1672: signal process started
2025/04/03 12:56:26 [notice] 1786#1786: signal process started
2025/04/03 12:58:33 [notice] 1933#1933: signal process started
here's the nginx error log
why does it try to bind to random adresses
not sure?
yes
i don't know but my config i added should just do 8888
maybe because it is in a docker container?
chat gpt says try this
listen 0.0.0.0:8888;
doesn't seem to work, nothing changed with netstat or curl
aren't there any other configs
its really strange. why is it trying those ports
the runpod came with a default config
oh then
maybe nginx is already running
and you are trying to run 1 more
i mean not with services, in the docker CMD (or start command)
can you specify the image
maybe? im using the default pytorch 2.4 docker image
wait a sec
cat /etc/nginx/sites-available/default
You should look at the following URL's in order to grasp a solid understanding
of Nginx configuration files in order to fully unleash the power of Nginx.
https://www.nginx.com/resources/wiki/start/
https://www.nginx.com/resources/wiki/start/topics/tutorials/config_pitfalls/
https://wiki.debian.org/Nginx/DirectoryStructure
In most cases, administrators will remove this file from sites-enabled/ and
leave it as reference inside of sites-available where it will continue to be
updated by the nginx packaging team.
This file will automatically load configuration files provided by other
applications, such as Drupal or Wordpress. These applications will be made
available underneath a path with that package name, such as /drupal8.
Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples.
##
Default server configuration
server {
listen 80 default_server;
listen [::]:80 default_server;
# SSL configuration
#
# listen 443 ssl default_server;
# listen [::]:443 ssl default_server;
#
# Note: You should disable gzip for SSL traffic.
# See: https://bugs.debian.org/773332
#
# Read up on ssl_ciphers to ensure a secure configuration.
# See: https://bugs.debian.org/765782
#
# Self signed certs generated by the ssl-cert package
# Don't use them in a production server!
#
# include snippets/snakeoil.conf;
root /var/www/html;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
servername ;
location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
}
# pass PHP scripts to FastCGI server
#
#location ~ .php$ {
# include snippets/fastcgi-php.conf;
#
# # With php-fpm (or other unix sockets):
# fastcgi_pass unix:/run/php/php7.4-fpm.sock;
# # With php-cgi (or other tcp sockets):
# fastcgi_pass 127.0.0.1:9000;
#}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /.ht {
# deny all;
#}
}
NGINX Product Documentation
Learn how to deliver, manage, and protect your applications using F5 NGINX products.
NGINX Product Documentation
Learn how to deliver, manage, and protect your applications using F5 NGINX products.
here's the default nginx config
in the site-available folder i got llm_app that i added and default that came with the docker image

can you cat the /start.sh
cat /start.sh
#!/bin/bash
set -e # Exit the script if any statement returns a non-true return value
---------------------------------------------------------------------------- #
Function Definitions #
---------------------------------------------------------------------------- #
Start nginx service
start_nginx() {
echo "Starting Nginx service..."
service nginx start
}
Execute script if exists
execute_script() {
local script_path=$1
local script_msg=$2
if [[ -f ${script_path} ]]; then
echo "${script_msg}"
bash ${script_path}
fi
}
Setup ssh
setup_ssh() {
if [[ $PUBLIC_KEY ]]; then
echo "Setting up SSH..."
mkdir -p ~/.ssh
echo "$PUBLIC_KEY" >> ~/.ssh/authorized_keys
chmod 700 -R ~/.ssh
if [ ! -f /etc/ssh/ssh_host_rsa_key ]; then
ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -q -N ''
echo "RSA key fingerprint:"
ssh-keygen -lf /etc/ssh/ssh_host_rsa_key.pub
fi
if [ ! -f /etc/ssh/ssh_host_dsa_key ]; then
ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key -q -N ''
echo "DSA key fingerprint:"
ssh-keygen -lf /etc/ssh/ssh_host_dsa_key.pub
fi
if [ ! -f /etc/ssh/ssh_host_ecdsa_key ]; then
ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -q -N ''
echo "ECDSA key fingerprint:"
ssh-keygen -lf /etc/ssh/ssh_host_ecdsa_key.pub
fi
if [ ! -f /etc/ssh/ssh_host_ed25519_key ]; then
ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -q -N ''
echo "ED25519 key fingerprint:"
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
fi
service ssh start
echo "SSH host keys:"
for key in /etc/ssh/.pub; do
echo "Key: $key"
ssh-keygen -lf $key
done
fi
}
Export env vars
export_envvars() {
echo "Exporting environment variables..."
printenv | grep -E '^RUNPOD|^PATH=|^_=' | awk -F = '{ print "export " $1 "="" $2 """ }' >> /etc/rp_environment
echo 'source /etc/rp_environment' >> ~/.bashrc
}
Start jupyter lab
start_jupyter() {
if [[ $JUPYTER_PASSWORD ]]; then
echo "Starting Jupyter Lab..."
mkdir -p /workspace && \
cd / && \
nohup jupyter lab --allow-root --no-browser --port=8888 --ip= --FileContentsManager.delete_to_trash=False --ServerApp.terminado_settings='{"shell_command":["/bin/bash"]}' --ServerApp.token=$JUPYTER_PASSWORD --ServerApp.allow_origin=* --ServerApp.preferred_dir=/workspace &> /jupyter.log &
echo "Jupyter Lab started"
fi
}
---------------------------------------------------------------------------- #
Main Program #
---------------------------------------------------------------------------- #
start_nginx
execute_script "/pre_start.sh" "Running pre-start script..."
echo "Pod Started"
setup_ssh
start_jupyter
export_env_vars
execute_script "/post_start.sh" "Running post-start script..."
echo "Start script(s) finished, pod is ready to use."
sleep infinity
here you go
service nginx restart
and
service nginx status
and it should work
apparently nginx was running on the default template so you got the port conflicts
huh
look at the start of the /start.sh file
there's this
and then
see that
start_nginx
?yh?
so does it work?
ok its printing starting nginx nginx and nginx is running that follows
how about the ports?
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
udp UNCONN 0 0 127.0.0.11:39579 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:8001 0.0.0.0:*
tcp LISTEN 0 2048 0.0.0.0:8000 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:8081 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:7861 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:7270 0.0.0.0:*
tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
tcp LISTEN 0 1024 127.0.0.1:36967 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:3001 0.0.0.0:*
tcp LISTEN 0 4096 127.0.0.11:34187 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:9091 0.0.0.0:*
tcp LISTEN 0 128 [::]:22 [::]:* still doesn't seem to be listening to 8888 also i verified the public ip works when i curl the ip without a port on my machine it returns
udp UNCONN 0 0 127.0.0.11:39579 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:8001 0.0.0.0:*
tcp LISTEN 0 2048 0.0.0.0:8000 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:8081 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:7861 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:7270 0.0.0.0:*
tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
tcp LISTEN 0 1024 127.0.0.1:36967 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:3001 0.0.0.0:*
tcp LISTEN 0 4096 127.0.0.11:34187 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:9091 0.0.0.0:*
tcp LISTEN 0 128 [::]:22 [::]:* still doesn't seem to be listening to 8888 also i verified the public ip works when i curl the ip without a port on my machine it returns
maybe server_name does not equal to the public ip?
no
why is it not opening the port tho
so confusing
did u do service nginx restart?
just the service nginx start
maybe do
service nginx restart
and then see if the port appearsok
still nothing on netstat
im confused on how did it reply on port 80 if its not listening
on port 80 on the netstat
u pinged the host from a computer at home?
yh
if so, there's a port mapping on the docker layer
so its probably not your container's number 80 port
actually ping doesn't work but curl returned that 301 response
anyways if that doesn't work
maybe try baking the config to /etc/nginx/nginx.conf directly
i did a before and after when i stopped nginx
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
udp UNCONN 0 0 127.0.0.11:47547 0.0.0.0:*
tcp LISTEN 0 1024 127.0.0.1:38753 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:9091 0.0.0.0:*
tcp LISTEN 0 4096 127.0.0.11:41723 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:7270 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:7861 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:8001 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:8081 0.0.0.0:*
tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:3001 0.0.0.0:*
tcp LISTEN 0 128 [::]:22 [::]:*
root@4f8ee7782bb7:/workspace# ss -tuln Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
udp UNCONN 0 0 127.0.0.11:47547 0.0.0.0:*
tcp LISTEN 0 1024 127.0.0.1:38753 0.0.0.0:*
tcp LISTEN 0 4096 127.0.0.11:41723 0.0.0.0:*
tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
tcp LISTEN 0 128 [::]:22 [::]:*
udp UNCONN 0 0 127.0.0.11:47547 0.0.0.0:*
tcp LISTEN 0 1024 127.0.0.1:38753 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:9091 0.0.0.0:*
tcp LISTEN 0 4096 127.0.0.11:41723 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:7270 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:7861 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:8001 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:8081 0.0.0.0:*
tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:3001 0.0.0.0:*
tcp LISTEN 0 128 [::]:22 [::]:*
root@4f8ee7782bb7:/workspace# ss -tuln Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
udp UNCONN 0 0 127.0.0.11:47547 0.0.0.0:*
tcp LISTEN 0 1024 127.0.0.1:38753 0.0.0.0:*
tcp LISTEN 0 4096 127.0.0.11:41723 0.0.0.0:*
tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
tcp LISTEN 0 128 [::]:22 [::]:*
oh here's the nginx.conf content, it explains all these ports
maybe i should change my configuration to their format
because they seem to work
append your config file to that list of servers and then try again?
yeah
ok
and jupyterlab's port seems to be 8888
so change the port to sth else just in case
oh
you're so right with putting it in the nginx conf itself
udp UNCONN 0 0 127.0.0.11:47547 0.0.0.0:*
tcp LISTEN 0 1024 127.0.0.1:38753 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:8888 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:9091 0.0.0.0:*
tcp LISTEN 0 4096 127.0.0.11:41723 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:7270 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:7861 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:8001 0.0.0.0:*
tcp LISTEN 0 2048 0.0.0.0:8000 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:8081 0.0.0.0:*
tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:3001 0.0.0.0:*
tcp LISTEN 0 128 [::]:22 [::]:* i'm still not getting an response but the port is listened to now
tcp LISTEN 0 1024 127.0.0.1:38753 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:8888 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:9091 0.0.0.0:*
tcp LISTEN 0 4096 127.0.0.11:41723 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:7270 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:7861 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:8001 0.0.0.0:*
tcp LISTEN 0 2048 0.0.0.0:8000 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:8081 0.0.0.0:*
tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
tcp LISTEN 0 511 0.0.0.0:3001 0.0.0.0:*
tcp LISTEN 0 128 [::]:22 [::]:* i'm still not getting an response but the port is listened to now
finally
u need to open the ports(sort of) in runpod

this one
in container settings
i got it like that
on http port or tcp port?
http
then
click on the access service (port 8888) thing
same as when u access jupyterlab
erm i think i should change port number
to not overlap with jupyter lab
yeah
and then expose the port on runpod and
try again
8080 should be fine right?
maybe go with an uncommon port
like 8465 or sth like that
8080 is used a lot so
hmm ill try 8080 for now but yh will try an uncommon port later
omg it works with the tcp port

i saw this
and tried to host on 9999 and then connecting to 12946
congrats
what is the purpose of this port mapping and how can i predict what port will be mapped to what??
what's the purpose:
what u r getting is a docker container
sorry im pretty new to this stuff uni didnt teach me any of this :/
no wonder barely any graduates are getting jobs
and in docker there's port mapping
so basically the host (the computer with gpus that's hosting the docker daemon) maps a random port and forwards requests that comes to that port to the port you requested
im a uni student too ๐
i see, so because the gpu machine/data centre could have multiple people wanting 9999 so they remap their ports to distribute them?
:0
so in ur case the host got a free 12946 port and selected it and then forwarded all tcp packets coming to that port to your docker container's 9999port
the host selects whatever port is left
so the second question: how can i predict...
the answer is you can't
ah...
erm
oh damn
there's an alternative tho
you prepared
just an enthusiast
third year uni panic upskilling the actually useful stuff
anyways
but why did you decide to run nginx
erm i want to run my own llm endpoint but didnt want to just use an premade solution so i can learn
ah
and then go on to tinker with the models myself
maybe chanings bits and bobs of the architecture
Expose ports | RunPod Documentation
Exposing ports on your pod to the outside world: Learn how to expose ports via RunPod's Proxy or TCP Public IP, and discover the benefits and limitations of each method, including symmetrical port mapping requests and proxy timeout considerations.
it seems like http ports arent remapped? like jupyter lab is still on 8888 instead of a random port
its proxied through cloudflare
oh
and it has a short timeout so not ideal for llm apps without streaming support

perhaps if i can access the port mapping in the pod i can first run a http request to get tcp port and then continue with tcp port
omg there is

in the env
it tells you the mapping
didnt know that
okay that's perfect
i just had a hunch
they might nicely put it there
but how do you know the port in your app
probs just use the os library
os.getenv("RUNPOD_TCP_PORT_9999")
i thought it was a application (like not webapp)
lol
right now its just some python scripts that i call an api endpoint
maybe will develop into some app for cv sakes
and maybe learn docker and build your own dockerfile
if only i saw that proxy port mapping before i started doing this stuff
yeah im still rudimentry with docker
learning it really helps
not sure how to inherit the pytorch one and add my own stuff
would be convinent to not have to reinstall stuff everytime the container restarts
FROM runpod/pytroch:something
//your other logic here
but what about the CMD they have in the pytorch one
would it still execute
no
the templates are all here
https://github.com/runpod/containershttps://github.com/runpod/containers
start.sh: https://github.com/runpod/containers/blob/main/container-template/start.sh
pytorch template:https://github.com/runpod/containers/blob/main/official-templates/pytorch/Dockerfile
GitHub
containers/container-template/start.sh at main ยท runpod/containers
๐ณ | Dockerfiles for the RunPod container images used for our official templates. - runpod/containers
GitHub
containers/official-templates/pytorch/Dockerfile at main ยท runpod/...
๐ณ | Dockerfiles for the RunPod container images used for our official templates. - runpod/containers
it just does sth and then calls "/start.sh"
kk thank you for all the help
so if you build your own image you can do
FROM image:tag
//some logic (like installing sth)
CMD ["/start.sh", "&", "./do_sth.sh"]
just a bored uni student
๐
idk that cmd is right tho (that & operator)
ill probs look more into that when i get sick of reinstalling the python extension for the vscode server
lol