Railway•2y ago

Booting worker looping infinitely

Hi there, I'm facing an issue where "Booting worker with pid: XXX" is looping infinitely despite no exit signals. The logs are:

[2023-05-20 02:24:28 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2023-05-20 02:24:28 +0000] [1] [INFO] Listening at: http://0.0.0.0:6516 (1)
[2023-05-20 02:24:28 +0000] [1] [INFO] Using worker: sync
[2023-05-20 02:24:28 +0000] [10] [INFO] Booting worker with pid: 10
Loading model...
Model loaded
Starting prediction...
[2023-05-20 02:25:49 +0000] [58] [INFO] Booting worker with pid: 58
Loading model...
Model loaded
Starting prediction...
[2023-05-20 02:25:55 +0000] [106] [INFO] Booting worker with pid: 106

[2023-05-20 02:24:28 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2023-05-20 02:24:28 +0000] [1] [INFO] Listening at: http://0.0.0.0:6516 (1)
[2023-05-20 02:24:28 +0000] [1] [INFO] Using worker: sync
[2023-05-20 02:24:28 +0000] [10] [INFO] Booting worker with pid: 10
Loading model...
Model loaded
Starting prediction...
[2023-05-20 02:25:49 +0000] [58] [INFO] Booting worker with pid: 58
Loading model...
Model loaded
Starting prediction...
[2023-05-20 02:25:55 +0000] [106] [INFO] Booting worker with pid: 106

I've looked at https://github.com/benoitc/gunicorn/issues/1663 and https://medium.com/@soleilstudio/booting-worker-with-pid-n-in-looping-infinitely-1c70a604231e and it seems to be memory problem. I've upgraded to developer plan and I do see spikes to 512MB under Metrics > Memory so I suspect that is the issue. Project ID: f1c85024-f36a-4e32-89f1-694999669ee1

GitHub

"Booting worker" is looping infinitely despite no exit signals · Is...

I'm trying to get gunicorn set up on Docker. It works well locally, and the production image is exactly the same as the local image, but I'm getting this strange behaviour on the production...

Medium

Booting worker with pid N in looping infinitely

The docker with gunicorn was running ok in my localhost and the build server, as the same code version, the same docker image. But when I…

Solution:

have you since re-deployed your app after upgrading to dev plan?

Jump to solution

21 Replies

Percy•2y ago

Project ID: f1c85024-f36a-4e32-89f1-694999669ee1

Solution

Brody•2y ago

have you since re-deployed your app after upgrading to dev plan?

howard_2435OP•2y ago

oh let me try that takes about 7min to deploy so it'll be a while

Brody•2y ago

damn, why?

howard_2435OP•2y ago

by the way, is there a way to set usage limits even on a usage-based plan?

Brody•2y ago

you pip freeze your entire systems python packages into the requirements.txt file?

howard_2435OP•2y ago

I'm trying to deploy a simple AI model and I suspect it's a combination of the weights as well as installing pytorch and torchvision

Brody•2y ago

chonky

howard_2435OP•2y ago

click==7.1.2
Flask==1.1.2
gunicorn==20.0.4
itsdangerous==1.1.0
Jinja2==2.11.3
MarkupSafe==1.1.1
Werkzeug==1.0.1

matplotlib>=3.2.2
numpy>=1.18.5,<1.24.0
Pillow>=7.1.2
PyYAML>=5.3.1
requests>=2.23.0
scipy>=1.4.1
torch>=1.7.0,!=1.12.0
torchvision>=0.8.1,!=0.13.0
tqdm>=4.41.0
protobuf<4.21.3

opencv-python-headless==4.7.0.72

click==7.1.2
Flask==1.1.2
gunicorn==20.0.4
itsdangerous==1.1.0
Jinja2==2.11.3
MarkupSafe==1.1.1
Werkzeug==1.0.1

matplotlib>=3.2.2
numpy>=1.18.5,<1.24.0
Pillow>=7.1.2
PyYAML>=5.3.1
requests>=2.23.0
scipy>=1.4.1
torch>=1.7.0,!=1.12.0
torchvision>=0.8.1,!=0.13.0
tqdm>=4.41.0
protobuf<4.21.3

opencv-python-headless==4.7.0.72

yeahhh

Brody•2y ago

oh wow that's not even big can you get pre-compiled versions of py torch? there's not, it's on the roadmap though

howard_2435OP•2y ago

perfect, thanks! I'll look into that once this works, thanks for the tip Build time: 154.82 seconds build time itself isn't too long, just 2.5min

Brody•2y ago

damn, I have build times around 20 seconds but I'm not doing ai stuff, so it's not comparable

howard_2435OP•2y ago

yeahh gigantic libraries are the bane of my existence finaly docker image is 4.9GB

Brody•2y ago

holy shit, right now in #🎤｜chit-chat we are talking about image sizes and have come to the concussion that anything above 100mb is gigantic

howard_2435OP•2y ago

.-. chonk

Brody•2y ago

you indeed have a chonker

howard_2435OP•2y ago

anyways it works now, thanks! getting a different error but seems to be more code/dependency related rather than memory

Brody•2y ago

maybe I might know how to help with the error, maybe not, let's see the error?

howard_2435OP•2y ago

it's 'Upsample' object has no attribute 'recompute_scale_factor'``

Brody•2y ago

oh yeah code issue 🙂 was thinking it was more along the lines of a missing system library okay you are all set, happy coding

howard_2435OP•2y ago

thank you!

Gaming

Programming

Booting worker looping infinitely