container crashed
container worked and everything was working well but container crashed...not sure why..Here's the projectid - 1b883543-ff8a-4389-9f72-1fe4b7b73668
93 Replies
Project ID:
1b883543-ff8a-4389-9f72-1fe4b7b73668Is there anything in the logs that might show why it crashed?
/bin/bash: line 1: web:: command not found
container event container restart
/bin/bash: line 1: web:: command not found
container event container died
Also when the app was up, it timedout on few api calls and I tried web: gunicorn app:app --timeout 120
but I am back to web: gunicorn app:app in procfile
web: is a Procfile thing, you should be able to set your start command to just gunicorn app:web --timeout 120
Railway hasn't supported procfiles in a long timeFile "/opt/venv/lib/python3.11/site-packages/gunicorn/util.py", line 402, in import_app
app = getattr(mod, name)
^^^^^^^^^^^^^^^^^^
AttributeError: module 'app' has no attribute 'web'
Failed to find attribute 'web' in 'app'.
[2024-09-13 14:09:51 +0000] [4] [INFO] Worker exiting (pid: 4)
[2024-09-13 14:09:51 +0000] [1] [INFO] Shutting down: Master
[2024-09-13 14:09:51 +0000] [1] [INFO] Reason: App failed to load.
container event container restart
[2024-09-13 14:09:52 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2024-09-13 14:09:52 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
[2024-09-13 14:09:52 +0000] [1] [INFO] Using worker: sync
[2024-09-13 14:09:52 +0000] [4] [INFO] Booting worker with pid: 4
Traceback (most recent call last):
File "/opt/venv/lib/python3.11/site-packages/gunicorn/util.py", line 402, in import_app
app = getattr(mod, name)
^^^^^^^^^^^^^^^^^^
AttributeError: module 'app' has no attribute 'web'
Failed to find attribute 'web' in 'app'.
[2024-09-13 14:09:52 +0000] [4] [INFO] Worker exiting (pid: 4)
[2024-09-13 14:09:52 +0000] [1] [INFO] Shutting down: Master
[2024-09-13 14:09:52 +0000] [1] [INFO] Reason: App failed to load.
container event container died
[2024-09-13 14:09:53 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2024-09-13 14:09:53 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
[2024-09-13 14:09:53 +0000] [1] [INFO] Using worker: sync
[2024-09-13 14:09:53 +0000] [4] [INFO] Booting worker with pid: 4
Traceback (most recent call last):
File "/opt/venv/lib/python3.11/site-packages/gunicorn/util.py", line 402, in import_app
app = getattr(mod, name)
^^^^^^^^^^^^^^^^^^
AttributeError: module 'app' has no attribute 'web'
Failed to find attribute 'web' in 'app'.
[2024-09-13 14:09:53 +0000] [4] [INFO] Worker exiting (pid: 4)
container event container restart
[2024-09-13 14:09:54 +0000] [1] [INFO] Shutting down: Master
[2024-09-13 14:09:54 +0000] [1] [INFO] Reason: App failed to load.
[2024-09-13 14:09:55 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2024-09-13 14:09:55 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
[2024-09-13 14:09:55 +0000] [1] [INFO] Using worker: sync
[2024-09-13 14:09:55 +0000] [4] [INFO] Booting worker with pid: 4
Traceback (most recent call last):
File "/opt/venv/lib/python3.11/site-packages/gunicorn/util.py", line 402, in import_app
app = getattr(mod, name)
^^^^^^^^^^^^^^^^^^
AttributeError: module 'app' has no attribute 'web'
Failed to find attribute 'web' in 'app'.
[2024-09-13 14:09:55 +0000] [4] [INFO] Worker exiting (pid: 4)
[2024-09-13 14:09:55 +0000] [1] [INFO] Shutting down: Master
[2024-09-13 14:09:55 +0000] [1] [INFO] Reason: App failed to load.
container event container died
[2024-09-13 14:09:56 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2024-09-13 14:09:56 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
[2024-09-13 14:09:56 +0000] [1] [INFO] Using worker: sync
[2024-09-13 14:09:56 +0000] [4] [INFO] Booting worker with pid: 4
Traceback (most recent call last):
File "/opt/venv/lib/python3.11/site-packages/gunicorn/util.py", line 402, in import_app
app = getattr(mod, name)
^^^^^^^^^^^^^^^^^^
AttributeError: module 'app' has no attribute 'web'
Failed to find attribute 'web' in 'app'.
[2024-09-13 14:09:56 +0000] [4] [INFO] Worker exiting (pid: 4)
[2024-09-13 14:09:56 +0000] [1] [INFO] Shutting down: Master
[2024-09-13 14:09:56 +0000] [1] [INFO] Reason: App failed to load.
container event container died
[2024-09-13 14:09:57 +0000] [1] [INFO] Using worker: sync
Can you share your repo? Your application might be called app and not web, in which case, you would use
app:appyes we do
That is true.
I am restarting with app:app
Now the app is up. testing the api calls to see if it times out
Same issue
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): feed-api.cielo.finance:443
[2024-09-13 14:16:51 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:4)
[2024-09-13 14:16:51 +0000] [4] [INFO] Worker exiting (pid: 4)
[2024-09-13 14:16:51 +0000] [5] [INFO] Booting worker with pid: 5
It is a worker timeout
Check the update
how long do you expect a single request to take
it depends on the response but should not take more than 5-7 seconds tbh
when testing, how many RPS are you doing?
what is RPS?
locally it works
requests per second
It is adata analysis application hitting many different apis
so def. 100 req/sec
then you are likely running into issues with having run gunicorn with sync workers
start it with the uvicorn event worker
how do I do it?
what's your current start command
gunicorn app:app --timeout 120
gunicorn app:app --timeout 120
no need to ping reply
what kind of app? fastapi, flask?
flask
your start command would become
gunicorn app:app --timeout 120 --worker-class eventlet
you will also need to make sure eventlet is in your requirements.txt fileHere's my current req
Flask==2.0.3
Werkzeug==2.0.3
Jinja2==3.0.3
itsdangerous==2.0.1
click==8.0.4
gunicorn==20.1.0
requests==2.26.0
prettytable==2.5.0
Flask-Login==0.5.0
what version should I use for eventlet
the latest is probably fine
ok, made changes and redeploying
Error: class uri 'eventlet' invalid or not found:
[Traceback (most recent call last):
File "/opt/venv/lib/python3.11/site-packages/gunicorn/util.py", line 99, in load_class
mod = importlib.import_module('.'.join(components))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.nix-profile/lib/python3.11/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/opt/venv/lib/python3.11/site-packages/gunicorn/workers/geventlet.py", line 20, in <module>
from eventlet.wsgi import ALREADY_HANDLED as EVENTLET_ALREADY_HANDLED
ImportError: cannot import name 'ALREADY_HANDLED' from 'eventlet.wsgi' (/opt/venv/lib/python3.11/site-packages/eventlet/wsgi.py)
]
container event container died
do you have it in your requirements.txt?
Flask==2.0.3
Werkzeug==2.0.3
Jinja2==3.0.3
itsdangerous==2.0.1
click==8.0.4
gunicorn==20.1.0
requests==2.26.0
prettytable==2.5.0
Flask-Login==0.5.0
eventlet==0.33.3
whats your start command
shoujd I use gevent?
^
gunicorn app:app --timeout 120 --worker-class eventlet
I have a Procfile
does it matter for Railway
no, we support procfiles
okay try gevent
trying now
invalid type: string "gunicorn --timeout 120 --worker-class gevent app:app", expected a ma
did I get format right?
try moving
app:app to after gunicorncontext: 5f53c0eeaf434d6d998e9ed8ac0a2c0f
Nixpacks build failed
Error: Reading Procfile
Caused by:
invalid type: string "gunicorn app:app --timeout 120 --worker-class gevent", expected a map
just way beyond my understanding...
gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/nix/store/wwg6376ymkhpjxy6c3w9gkbhv6l2gmqy-libxcrypt-4.4.36/include -fPIC -DLIBEV_EMBED=1 -DEV_COMMON= -DEV_CLEANUP_ENABLE=0 -DEV_EMBED_ENABLE=0 -DEV_PERIODIC_ENABLE=0 -DEV_USE_REALTIME=1 -DEV_USE_MONOTONIC=1 -DEV_USE_FLOOR=1 -Isrc/gevent/libev -I/nix/store/k3701zl6gmx3la7y4dnflcvf3xfy88kh-python3-3.11.9/include/python3.11 -I/root/.nix-profile/include/python3.11 -I/tmp/pip-install-mht4g7p5/gevent_7b95d12f6294466681fc5ce8f6126ea3/deps -I/tmp/pip-install-mht4g7p5/gevent_7b95d12f6294466681fc5ce8f6126ea3/src/gevent/libev -I/tmp/pip-install-mht4g7p5/gevent_7b95d12f6294466681fc5ce8f6126ea3/deps/libev -Isrc/gevent -Isrc/gevent/libev -Isrc/gevent/resolver -I. -I/opt/venv/include -I/nix/store/k3701zl6gmx3la7y4dnflcvf3xfy88kh-python3-3.11.9/include/python3.11 -c src/gevent/libev/callbacks.c -o build/temp.linux-x86_64-cpython-311/src/gevent/libev/callbacks.o
#11 103.8 gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/nix/store/wwg6376ymkhpjxy6c3w9gkbhv6l2gmqy-libxcrypt-4.4.36/include -fPIC -DLIBEV_EMBED=1 -DEV_COMMON= -DEV_CLEANUP_ENABLE=0 -DEV_EMBED_ENABLE=0 -DEV_PERIODIC_ENABLE=0 -DEV_USE_REALTIME=1 -DEV_USE_MONOTONIC=1 -DEV_USE_FLOOR=1 -Isrc/gevent/libev -I/nix/store/k3701zl6gmx3la7y4dnflcvf3xfy88kh-python3-3.11.9/include/python3.11 -I/root/.nix-profile/include/python3.11 -I/tmp/pip-install-mht4g7p5/gevent_7b95d12f6294466681fc5ce8f6126ea3/deps -I/tmp/pip-install-mht4g7p5/gevent_7b95d12f6294466681fc5ce8f6126ea3/src/gevent/libev -I/tmp/pip-install-mht4g7p5/gevent_7b95d12f6294466681fc5ce8f6126ea3/deps/libev -Isrc/gevent -Isrc/gevent/libev -Isrc/gevent/resolver -I. -I/opt/venv/include -I/nix/store/k3701zl6gmx3la7y4dnflcvf3xfy88kh-python3-3.11.9/include/python3.11 -c src/gevent/libev/corecext.c -o build/temp.linux-x86_64-cpython-311/src/gevent/libev/corecext.o
#11 103.8 In file included from src/gevent/libev/libev.h:9,
#11 103.8 from src/gevent/libev/corecext.c:1319:
#11 103.8 /tmp/pip-install-mht4g7p5/gevent_7b95d12f6294466681fc5ce8f6126ea3/deps/libev/ev.c:580:48: warning: "/" within comment [-Wcomment]
#11 103.8 580 | /#define MIN_INTERVAL 0.00000095367431640625 /* 1/2**20, good till 2200 */
#11 103.8 |
#11 103.8 /tmp/pip-install-mht4g7p5/gevent_7b95d12f6294466681fc5ce8f6126ea3/deps/libev/ev.c: In function ‘ecb_binary32_to_binary16’:
gone back to eventlet
File "/opt/venv/lib/python3.11/site-packages/eventlet/greenio/base.py", line 32, in <module>
socket_timeout = eventlet.timeout.wrap_is_timeout(socket.timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/eventlet/timeout.py", line 166, in wrap_is_timeout
base.istimeout = property(lambda : True)
^^^^^^^^^^^^^^^
TypeError: cannot set 'is_timeout' attribute of immutable type 'TimeoutError'
]
container event container died
can I use python-3.9.7
yes, set it in a runtime.txt file
I did that but no idea how to resolve the issue with timeouts
neither gevent or eventlet is working on railway
is your app now using gevent
gevent had build issues so I went back to eventlet
but eventlet is crashing the docker
Error: class uri 'eventlet' invalid or not found:
you woud need to get one of them working
runtime.txt does not seem to be getting picked up
why do you say that
File "/opt/venv/lib/python3.11
using python3.11
i cant verify that, you are about to delete the service
I deleted the service
started a new one.
and I got things back to a place where it is working
not using eventlet or gevent
just timeout
I want to see if the error persists
atleast now I don't have build issues or deployment issues
this wasnt an issue with railway so the timeout issue isnt going to change
no timeouts locally
local dev works fine
you are not running sync workers locally
are you sure this is flask?
Yes
app = Flask(name)
same error
[2024-09-13 16:16:28 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:4)
[2024-09-13 16:16:28 +0000] [4] [INFO] Worker exiting (pid: 4)
[2024-09-13 16:16:28 +0000] [5] [INFO] Booting worker with pid: 5
you need to use an async worker
that will require changes ot my code?
I just want to replicate my local environment on railway
im not sure
in a production environment you must use gunicorn
yes that works
and in your case you also need to use async workers
but the moment I add eventlet, it is not even building
the moment I add gevent, docker is crashing
try gthread then
increasaed timeout sufficiently large so that works but towards the end, the final request the response got chopped off
is there a limit on the size of response for railway?
no, only a 5 minute time limit
this was the last step before rendering if you see logs
increasing the time-out here is just a Band-Aid solution
still dont kknow why railway has issue.
works fine on local
works well on a vps server
fails on railway
that not not indicate an issue with railway, that indicates an issue with your configurations on railway
you need to use an async worker so that your app can handle more then 3 requests concurrently
yea, will need to implement celery and redis but was hoping that I can do that later.
Maybe for now, I just use VPS and evaluate railway later on
you can do celery and redis later, right now you need to use an async worker
didn't we try that?
eventlet and gevent both crash the docker for whatever reason
you have something misconfigured and will need to look into why that's happening
I'll modify the code to see if it works
by importing asyncio & aiohttp and adding async with in front of all my request fetch functions
Figured out a way.
Is there a way to hard refresh a web page?
feels like it is serving some cached version
railway doesnt do any asset caching, is it behind cloudflare?
I migrated the Flask App to NodeJS app
is the startup command - web: node app.js
container is failing right now
Build is successful
please provide the error
/bin/bash: line 1: web:: command not found
container event container died
what is the contents of your procfile
web: node app.js
I'll check back with this when I'm back at the computer so I can view your logs, I don't think you are sending me the correct error
\
wait, aren't you deploying from a Dockerfile? why do you have a procfile
I connected my github project from railway
railway handles everything
I can remove the procfile
you want me to try that?
haha slow down there
ok, let me know
trying my luck ...
send me your Dockerfile please
How do I do it?
I only have the node dev environment on my end. Then I commit my changes to Github. Railway auto deploys
you do not have a Dockerfile?
Nope
Never had one for flask app to
Didn’t know we need one
delete your Procfile
ok
deleted
what about the start command in update config?
you should have a start script in your package.json
"scripts": {
"start": "node app.js",
"dev": "app.js",
"test": "echo "Error: no test specified" && exit 1"
},
yea, it has always been there
then you have no need for a procfile
ok. but I am still getting the same problem
thats not valid syntax
what should I enter in there?
just node app.js?
figured it out. TY for your help!
no problem!
when you upgrade the plan, does the app needs to be redeloyed to take advantage of the new RAM etc?
yes it does