R
Railway6mo ago
Joshie

v2 Builder bug: uses nixpacks over dockerfile

I have a Dockerfile in my directories root. But yet, the build logs keeps showing the use of nixpacks:
╔══════════════════════════════ Nixpacks v1.24.1 ══════════════════════════════╗
║ setup │ python311, postgresql, gcc ║
║──────────────────────────────────────────────────────────────────────────────║
║ install │ python -m venv --copies /opt/venv && . /opt/venv/bin/activate ║
║ │ && pip install -r requirements.txt ║
║──────────────────────────────────────────────────────────────────────────────║
║ start │ python manage.py migrate && gunicorn BBWare.wsgi ║
╚══════════════════════════════════════════════════════════════════════════════╝
╔══════════════════════════════ Nixpacks v1.24.1 ══════════════════════════════╗
║ setup │ python311, postgresql, gcc ║
║──────────────────────────────────────────────────────────────────────────────║
║ install │ python -m venv --copies /opt/venv && . /opt/venv/bin/activate ║
║ │ && pip install -r requirements.txt ║
║──────────────────────────────────────────────────────────────────────────────║
║ start │ python manage.py migrate && gunicorn BBWare.wsgi ║
╚══════════════════════════════════════════════════════════════════════════════╝
When I have a look at the details of the deployment, it shows that the builder is dockerfile (correct). But why is nixpacks being used? It is ignoring my dockerfile!? Why might this be going on?
Solution:
hey joshie, I'm told this has been fixed, would you want to give the v2 builder another try?
Jump to solution
63 Replies
Percy
Percy6mo ago
Project ID: 40943c2e-d9a8-4f3c-b414-73ea73eccd8f
Joshie
JoshieOP6mo ago
40943c2e-d9a8-4f3c-b414-73ea73eccd8f
Joshie
JoshieOP6mo ago
Nope I am wrong. I updated my question. It really isn't touching the dockerfile at all. But the builder thinks it should use dockerfile
No description
No description
Brody
Brody6mo ago
do you have a root directory set?
Joshie
JoshieOP6mo ago
Yes
No description
Brody
Brody6mo ago
what's your dockerfile path set to exactly?
Joshie
JoshieOP6mo ago
The file it should be using (the one it even recognizes in the details) is /BBWare/Dockerfile I found the issue
Brody
Brody6mo ago
what's the issue?
Joshie
JoshieOP6mo ago
It is the new builder. I had it enabled. It was messing things up. I am guessing this is not intended behavior. But I turned it off and it worked as expected
Brody
Brody6mo ago
it shouldn't, I use the v2 builder exclusively with Dockerfiles for all my own projects and I've never seen this issue would you mind enabling the v2 builder again so we can try a few things?
Joshie
JoshieOP6mo ago
There seems to be some other issue with my deployment. But it looks like a me issue at this point. The thing actually is able to build and deploy without crashing :D. Yea, we can mess with it before I start fixing the CMD issue
Brody
Brody6mo ago
well now you got me curious about the CMD issue
Joshie
JoshieOP6mo ago
I think it is just that I have too many things chaned together. I really really don't think it is the cause. But I mean,,, anything is possible
Brody
Brody6mo ago
okay we'll come back to that then let me know when you're back on the v2 builder
Joshie
JoshieOP6mo ago
Yea, just switching it cuases it to use the wrong deployment method. As you can see in these two photos, the builder is docker. But the actual build is being done with nixpacks?
Brody
Brody6mo ago
so what method are you using to set the location of the dockerfile?
Joshie
JoshieOP6mo ago
I am not. I am letting it be found by the system.
Brody
Brody6mo ago
I see
Joshie
JoshieOP6mo ago
.
├── BBWare
├── Dockerfile
├── breaker
├── docker-entrypoint.sh
├── gunicorn.conf.py
├── manage.py
├── permissions
├── purchase
├── requirements.txt
├── sale
└── stock
.
├── BBWare
├── Dockerfile
├── breaker
├── docker-entrypoint.sh
├── gunicorn.conf.py
├── manage.py
├── permissions
├── purchase
├── requirements.txt
├── sale
└── stock
This is my directory. (docker-entrypoint.sh is for local dev only)
Brody
Brody6mo ago
I think the v2 builder is still looking in the root of the repo for the dockerfile, but your dockerfile is in the BBWare subfolder, right?
Joshie
JoshieOP6mo ago
Yes. Sorry, the naming of this project is confussing. The root root of this project is
.
├── BBWare
├── Caddyfile
├── README.md
├── compose.yaml
├── frontend
├── proxy
└── pyproject.toml
.
├── BBWare
├── Caddyfile
├── README.md
├── compose.yaml
├── frontend
├── proxy
└── pyproject.toml
gotta love that double naming of nested directories 😄
Brody
Brody6mo ago
okay so if the v2 builder looked for a dockerfile in your set root directory this would work that sound about right?
Joshie
JoshieOP6mo ago
Yea. It should
Brody
Brody6mo ago
cool, I'll try to spin up a repo to test that theory. but just as a little last ditch workaround, can you try setting a service variable RAILWAY_DOCKERFILE_PATH=/BBWare/Dockerfile with the v2 builder
Joshie
JoshieOP6mo ago
Didn't do anything
Brody
Brody6mo ago
was worth a try, the v2 builder might not even have a concept of that variable, thanks for trying anyway you can remove that variable and switch back to the legacy builder and I'll work on reproducing this and then reporting to the v2 builder's dev
Joshie
JoshieOP6mo ago
👍 sounds good. Glad I could help. And more glad that I was able to figure out my issue
Brody
Brody6mo ago
this thread is definitely a big help!
Joshie
JoshieOP6mo ago
I actually have a different directory in this same project that has 2 files in it. I can switch it to the new builder and see if it does the same thing Directory:
proxy
├── Caddyfile
└── Dockerfile
proxy
├── Caddyfile
└── Dockerfile
Brody
Brody6mo ago
if you're down to try that more information would always be great
Joshie
JoshieOP6mo ago
Well yea. It couldn't even begin to guess what this service is supposed to be
No description
Brody
Brody6mo ago
same results though, still tried to use nixpacks useful information
Joshie
JoshieOP6mo ago
Generated config
{
"$schema": "https://railway.app/railway.schema.json",
"build": {
"builder": "DOCKERFILE",
"dockerfilePath": "proxy/Dockerfile",
"buildEnvironment": "V2"
},
"deploy": {
"runtime": "V2",
"numReplicas": 1,
"sleepApplication": false,
"restartPolicyType": "ALWAYS"
}
}
{
"$schema": "https://railway.app/railway.schema.json",
"build": {
"builder": "DOCKERFILE",
"dockerfilePath": "proxy/Dockerfile",
"buildEnvironment": "V2"
},
"deploy": {
"runtime": "V2",
"numReplicas": 1,
"sleepApplication": false,
"restartPolicyType": "ALWAYS"
}
}
Yea. But becuase it had no files for nixpacks to guess what the service was, it just failed. So I guess you were right. The new builder isn't recognizing dockerfile (sub directory at least)
Brody
Brody6mo ago
on paper it sounds like an easy fix, so I hope it's an easy fix in implementation, I'm afk right now so won't be able to put together a repo to reproduce, but will update you when things happen thanks again! was able to easily reproduce, and have reported it to the team, but the dev is offline right now if you're curious, heres my minimal repo that can reproduce this - https://github.com/brody192/dockerfile_subfolder if you wanna go over that CMD issue you mentioned id be happy to help
Joshie
JoshieOP6mo ago
No rush (at least on my end). Not like I need the new build system for any reason. I appreciate the willingness to help. I know exactly what the issue is and have a solution written up. Going to test it tomorrow though. It is late where I live. But if I end up being wrong and needing help ,,,,
Brody
Brody6mo ago
sounds good, I'd like to hear the problem and solution once you figure that out!
Brody
Brody6mo ago
ray what lol
No description
Ray
Ray6mo ago
shh
Joshie
JoshieOP6mo ago
TL;DR Django manage based actions can only be done through commands. So having a line in like RUN ./manage migrate is illegal. So, I did basically CMD ./manage migrate && ./manage runserver but it made the startup CMD enter an infinite loop. The way to get out of that, is to have the CMD be some bash script. Hence that docker-entrypoint.sh file. I ended up modifying it handle both local dev and deployment. That being said, I am now seeing a different issue; my goto gunicorn config is to have workers = multiprocessing.cpu_count() * 2 + 1. But when I run with this config, it seems that railway doesn't like it:
[2024-06-14 14:35:13 +0000] [65] [INFO] Booting worker with pid: 65
[2024-06-14 14:35:13 +0000] [67] [INFO] Booting worker with pid: 67
[2024-06-14 14:35:17 +0000] [3] [ERROR] Worker (pid:28) was sent SIGKILL! Perhaps out of memory?
[2024-06-14 14:35:20 +0000] [3] [ERROR] Worker (pid:30) was sent SIGKILL! Perhaps out of memory?
[2024-06-14 14:35:20 +0000] [3] [ERROR] Worker (pid:35) was sent SIGKILL! Perhaps out of memory?
[2024-06-14 14:35:22 +0000] [68] [INFO] Booting worker with pid: 68
[2024-06-14 14:35:22 +0000] [3] [ERROR] Worker (pid:36) was sent SIGKILL! Perhaps out of memory?
[2024-06-14 14:35:24 +0000] [3] [ERROR] Worker (pid:32) was sent SIGKILL! Perhaps out of memory?
[2024-06-14 14:35:24 +0000] [69] [INFO] Booting worker with pid: 69
[2024-06-14 14:35:25 +0000] [70] [INFO] Booting worker with pid: 70
[2024-06-14 14:35:28 +0000] [3] [ERROR] Worker (pid:34) was sent SIGKILL! Perhaps out of memory?
[2024-06-14 14:35:13 +0000] [65] [INFO] Booting worker with pid: 65
[2024-06-14 14:35:13 +0000] [67] [INFO] Booting worker with pid: 67
[2024-06-14 14:35:17 +0000] [3] [ERROR] Worker (pid:28) was sent SIGKILL! Perhaps out of memory?
[2024-06-14 14:35:20 +0000] [3] [ERROR] Worker (pid:30) was sent SIGKILL! Perhaps out of memory?
[2024-06-14 14:35:20 +0000] [3] [ERROR] Worker (pid:35) was sent SIGKILL! Perhaps out of memory?
[2024-06-14 14:35:22 +0000] [68] [INFO] Booting worker with pid: 68
[2024-06-14 14:35:22 +0000] [3] [ERROR] Worker (pid:36) was sent SIGKILL! Perhaps out of memory?
[2024-06-14 14:35:24 +0000] [3] [ERROR] Worker (pid:32) was sent SIGKILL! Perhaps out of memory?
[2024-06-14 14:35:24 +0000] [69] [INFO] Booting worker with pid: 69
[2024-06-14 14:35:25 +0000] [70] [INFO] Booting worker with pid: 70
[2024-06-14 14:35:28 +0000] [3] [ERROR] Worker (pid:34) was sent SIGKILL! Perhaps out of memory?
What is the actual real way to deploy a gunicorn instance? To my understanding, to get the most of gunicorn, you should have 2-4X workers to cores. But I guess that isn't possible here? So I should just use 1 worker (default value)? This is also not really related to this thread though; I can totally open a new one if need be
Brody
Brody6mo ago
that will evaluate to 65 workers because cpu_count will return 32 despite you being on the hobby plan.
Joshie
JoshieOP6mo ago
Well ,,,, I tried doing a deploy with just the default ...
[2024-06-14 14:46:41 +0000] [72] [INFO] Booting worker with pid: 72
[2024-06-14 14:46:41 +0000] [73] [INFO] Booting worker with pid: 73
[2024-06-14 14:46:43 +0000] [75] [INFO] Booting worker with pid: 75
[2024-06-14 14:46:43 +0000] [30] [ERROR] Worker (pid:41) was sent SIGKILL! Perhaps out of memory?
--- Logging error ---
[2024-06-14 14:46:43 +0000] [74] [INFO] Booting worker with pid: 74
Traceback (most recent call last):
stream.write(msg + self.terminator)
RuntimeError: reentrant call inside <_io.BufferedWriter name='<stderr>'>

Traceback (most recent call last):
File "/usr/local/lib/python3.12/logging/__init__.py", line 1163, in emit
self.reap_workers()
File "/opt/venv/lib/python3.12/site-packages/gunicorn/arbiter.py", line 554, in reap_workers
File "/opt/venv/lib/python3.12/site-packages/gunicorn/glogging.py", line 272, in error
File "/usr/local/lib/python3.12/logging/__init__.py", line 1568, in error
File "/usr/local/lib/python3.12/logging/__init__.py", line 1684, in _log
File "/usr/local/lib/python3.12/logging/__init__.py", line 1700, in handle
File "/usr/local/lib/python3.12/logging/__init__.py", line 1762, in callHandlers
File "/usr/local/lib/python3.12/logging/__init__.py", line 1028, in handle
self.handleError(record)
sys.stderr.write('--- Logging error ---\n')
RuntimeError: reentrant call inside <_io.BufferedWriter name='<stderr>'>
Call stack:
File "/opt/venv/bin/gunicorn", line 8, in <module>
File "/opt/venv/lib/python3.12/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
super().run()
Arbiter(self).run()
File "/opt/venv/lib/python3.12/site-packages/gunicorn/arbiter.py", line 202, in run
File "/opt/venv/lib/python3.12/site-packages/gunicorn/arbiter.py", line 571, in manage_workers
File "/opt/venv/lib/python3.12/site-packages/gunicorn/arbiter.py", line 642, in spawn_workers
File "/opt/venv/lib/python3.12/site-packages/gunicorn/arbiter.py", line 593, in spawn_worker
File "/opt/venv/lib/python3.12/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
Arguments: ()
[2024-06-14 14:46:43 +0000] [76] [INFO] Booting worker with pid: 76
[2024-06-14 14:46:44 +0000] [30] [ERROR] Worker (pid:42) was sent SIGKILL! Perhaps out of memory?
[2024-06-14 14:46:44 +0000] [77] [INFO] Booting worker with pid: 77
[2024-06-14 14:46:41 +0000] [72] [INFO] Booting worker with pid: 72
[2024-06-14 14:46:41 +0000] [73] [INFO] Booting worker with pid: 73
[2024-06-14 14:46:43 +0000] [75] [INFO] Booting worker with pid: 75
[2024-06-14 14:46:43 +0000] [30] [ERROR] Worker (pid:41) was sent SIGKILL! Perhaps out of memory?
--- Logging error ---
[2024-06-14 14:46:43 +0000] [74] [INFO] Booting worker with pid: 74
Traceback (most recent call last):
stream.write(msg + self.terminator)
RuntimeError: reentrant call inside <_io.BufferedWriter name='<stderr>'>

Traceback (most recent call last):
File "/usr/local/lib/python3.12/logging/__init__.py", line 1163, in emit
self.reap_workers()
File "/opt/venv/lib/python3.12/site-packages/gunicorn/arbiter.py", line 554, in reap_workers
File "/opt/venv/lib/python3.12/site-packages/gunicorn/glogging.py", line 272, in error
File "/usr/local/lib/python3.12/logging/__init__.py", line 1568, in error
File "/usr/local/lib/python3.12/logging/__init__.py", line 1684, in _log
File "/usr/local/lib/python3.12/logging/__init__.py", line 1700, in handle
File "/usr/local/lib/python3.12/logging/__init__.py", line 1762, in callHandlers
File "/usr/local/lib/python3.12/logging/__init__.py", line 1028, in handle
self.handleError(record)
sys.stderr.write('--- Logging error ---\n')
RuntimeError: reentrant call inside <_io.BufferedWriter name='<stderr>'>
Call stack:
File "/opt/venv/bin/gunicorn", line 8, in <module>
File "/opt/venv/lib/python3.12/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
super().run()
Arbiter(self).run()
File "/opt/venv/lib/python3.12/site-packages/gunicorn/arbiter.py", line 202, in run
File "/opt/venv/lib/python3.12/site-packages/gunicorn/arbiter.py", line 571, in manage_workers
File "/opt/venv/lib/python3.12/site-packages/gunicorn/arbiter.py", line 642, in spawn_workers
File "/opt/venv/lib/python3.12/site-packages/gunicorn/arbiter.py", line 593, in spawn_worker
File "/opt/venv/lib/python3.12/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
Arguments: ()
[2024-06-14 14:46:43 +0000] [76] [INFO] Booting worker with pid: 76
[2024-06-14 14:46:44 +0000] [30] [ERROR] Worker (pid:42) was sent SIGKILL! Perhaps out of memory?
[2024-06-14 14:46:44 +0000] [77] [INFO] Booting worker with pid: 77
.... "default" gunicorn BBWare.wsgi -b [::]:${PORT:-8000}
Brody
Brody6mo ago
you're on the trial plan? weren't you on hobby yesterday? I don't know what your app does but its very possible it's trying to use more than 500mb
Joshie
JoshieOP6mo ago
This is totally possible. But then again, these numbers don't seem reasonable for a program that is idle. I am using a trail account right now to gauge the viability of railway for my needs. A rabbit hole I really didn't need to go down. But too deep into it now :p But once I confirm it is, I would switch this to a pro plan Anyways, I feel like this issue is escaping the purpose of the thread. I will open a new thread
No description
Brody
Brody6mo ago
sounds good, I'll see you there!
Solution
Brody
Brody6mo ago
hey joshie, I'm told this has been fixed, would you want to give the v2 builder another try?
Joshie
JoshieOP6mo ago
Sure I can do that
Brody
Brody6mo ago
thanks!
Joshie
JoshieOP6mo ago
I am a pro when it comes to breaking things 😄 If I had to guess, it is becuase I have this attached to a repo that has been removed. Even if that is the cause, two things should happen (instead of what does). 1) the change that failed to "commit" should be reverted (it isn't) and 2) the error message should be better. Ideally.
Brody
Brody6mo ago
why does it sound like you are in the middle of a data center
Joshie
JoshieOP6mo ago
And this isn't a UI bug either (the setting not reverting) I refresh with no cache and it remins that way Becuase I am Becuase there is running water right next to me.
Brody
Brody6mo ago
when the redeploy happens via deploying changes it does not grab your repo, so repo or not shouldn't matter, but maybe it doesn't like that it doesn't exist and it's breaking some flow and it's not like it failed it just didn't trigger a redeploy, you can do that manually after all
Joshie
JoshieOP6mo ago
Can confirm. I removed the repo link, without changing anything else. And it was able to build. So yea, it can still build without a repo link but having a broken one cuases a crash.
Brody
Brody6mo ago
I'll try to reproduce this
Joshie
JoshieOP6mo ago
Yes. The two issues is the bad error message (not informative) and the fact that it did commit my changes even if it was unable to attempt a redeploy. Idk what it should do in that case. But it just feels like it should revert the settings. But maybe other people would disagree? I guess it depends on what settings are changed. Like in this case, with the builder being changed, I want a revert. But if I was changing variables, maybe not. :shrug:
Brody
Brody6mo ago
I don't think it should revert, since the settings will be applied on the next deployment
Joshie
JoshieOP6mo ago
Yea. After writting it out and thinking about it, I guess that is correct.
Brody
Brody6mo ago
either way, once you manually triggered a redeploy, did it work?
Joshie
JoshieOP6mo ago
The website is being a little buggy / laggy. But let us ignore that part This does work now. It was able to deploy and build properly.
Brody
Brody6mo ago
awesome then I will mark this as solved
Joshie
JoshieOP6mo ago
@Bug Basher please 🫴 (jk, I don't actually need it if this doesn't qualify for that :p)
Brody
Brody6mo ago
why doesn't it? you are the first person to catch this dockerfile bug
Joshie
JoshieOP6mo ago
Thanks 😄 (I more so didn't want to come off as pushy for it. I don't require it or anything.)
Brody
Brody6mo ago
badges are fun
Want results from more Discord servers?
Add your server