Upload large file via CLI
Hi there, I have two 200mb files that are ignored by git. I want to upload to the railway instance. How do I do that so I can deploy? Thanks
Solution:Jump to solution
for context, this was solved with a custom dockerfile and fixing a lot of dependency conflicts
59 Replies
Project ID:
N/Aservice_id: 92e27f59-0420-4bb1-aa1f-625ef3e927f3
the cli upload limit is something like 50mb, and railway doesn't support git lfs (your 200mb files would be in git lfs)
the best way for you to go about this would be to have code to download the files from a storage bucket
How would I do that exactly? These are files which are used in conjunction with a module...they have to be put in a specific spot in the file system
even locally
write code to download the files from a storage bucket to the needed location
Well haha thank you sorry I'm not too well versed in this stuff
I mean is this a script that runs on build?
If so, how would I do that?
run in build or before start, whatever is easiest for you
I appreciate the patience
you would need to add another phase to run your code that downloads the file
https://nixpacks.com/docs/configuration/file#phases
Will give it a shot thanks
🫡
no problem!
Hey so my script relies on a package that needs to be downloaded
So I get an error thrown when I run it with this nix.toml:
[phases.setup]
nixPkgs = ["...", "tesseract"]
[phases.downloadModules]
cmds = ["python downloadModules.py"]
The second phase is what is impt
Whats the usecase you are trying to solve here?
200mb file needs to be downloaded to root directory for use with a module
I'm storing in a bucket
Want to download from bucket
I need to do it at build time...it's a chromium module for dealing with PDFs
The error that gets thrown is a module not found:#10 0.418 ModuleNotFoundError: No module named 'supabase'
Which I think just means the phase needs to run after packages are installed
then you might need a
dependsOn = ['install'] in your download phase
just so that the download phase is ran after the install phaseWill try:
[phases.setup]
nixPkgs = ["...", "tesseract"]
[phases.downloadModules]
dependsOn = ['install']
cmds = ["python3 -m pip install apryse-sdk --extra-index-url=https://pypi.apryse.com", "python downloadModules.py"]
nah that looks hacky
put that pip package in your requirements.txt file
ooof
The extra index thing causes errors
why is it needed?
Not sure lol. I think they switched package names or something but are enterprise software so maybe it's for legacy stuff. Support said to make sure it's there
Technically this deployed. Lemme see if it runs
if it really does need to be there, and it won't work in the requirements.txt file, just extend the install phase cmds array with that command
Gotcha. Will do. There's one more command that needs to be added to get this run so will check that. Thx
🤞
you're learning fast, you got this for sure
Okay so that works...kinda:
Exception: ./html2pdf_chromium.so: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory
Exit code: 0X7F
Some weird shared library issue.
Not gonna make you debug this with me LOL but libnss3 is an apt thing, ya?
one sec
you will need to add these packages to your nixPkgs array
remember, 99% of the time, make sure you are extending the arrays, you've already shown that you have done that for nixPkgs so this is just a reminder
Gotcha and yep will give it a shot rn
Hoping this is the one man
it might be, fingers crossed
Hmm not liking those packages
who? you or nixpacks?
haha nixpacks. The third one errors
please hold
Sounds good. Where'd ya get that list btw ? just curious
that was from a github issue on nixpacks, I'm going to look at the current librarys nixpacks installs though
kk. Yeah actually I justed tried the first one
[phases.setup]
nixPkgs = ["...", "tesseract", "libnss3"]
and it failed
it's for node, but these are the packages nixpacks installs for puppeteer, and I'm assuming you're using the python equivalent, so it would likely need all the same packages
https://github.com/railwayapp/nixpacks/blob/main/src/providers/node/mod.rs#L119
Ah I see
Yeah hmm I wonder why libnss3 is undefined
show me the error?
error: undefined variable 'libnss3'
I did just see this issue:https://github.com/railwayapp/nixpacks/issues/294
GitHub
Install
libnss when puppeteer is used · Issue #294 · railwayapp...We need to install libnss via apt. It is probably worth overhauling the current solution to check if packages are installed and make one-function, purpose-built for this.
screenshot of error please
Oh duh haha that's just a pointer to what you sent
send your full nixpacks.toml please
At that time it was this:
[phases.install]
cmds = ["...", "python3 -m pip install apryse-sdk --extra-index-url=https://pypi.apryse.com"]
[phases.setup]
nixPkgs = ["...", "tesseract", "libnss3"]
[phases.downloadModules]
dependsOn = ['install']
cmds = ["python downloadModules.py", "chmod +x /app/html2pdf_chromium.so"]
Same error happens when doing the full list of packages too
show me the nixpacks.toml will all the packages please
[phases.install]
cmds = ["...", "python3 -m pip install apryse-sdk --extra-index-url=https://pypi.apryse.com"]
[phases.setup]
nixPkgs = ["...", "tesseract", "libnss3", "libatk1.0-0", "libatk-bridge2.0-0", "libcups2","libgbm1","libasound2","libpangocairo-1.0-0","libxss1","libgtk-3-0"]
[phases.downloadModules]
dependsOn = ['install']
cmds = ["python downloadModules.py", "chmod +x /app/html2pdf_chromium.so"]
and you get this error with the nixpacks.toml you just sent
Sorry let me clear. The error I sent was for the single. This is the error for that one (just ran again). I assume it's bc parses the names first for exceptions
Sorry for miscommunication
in that build error it looks like you just haven't quoted the packages
I did. I think it just logs it like that?
you know, there's always the option of a dockerfile based build?
Yeah...i just don't know how to do that haha
would it be any different tho? Same installations right?
quite different, yeah
could you share your repo so I can get a better idea of what's going on, that will help me create a dockerfile for you
Uh yeah that's fine. What's your github?
brody192
Invited. Gonna DM you so we can discuss more personal things to that repo
okay!
Solution
for context, this was solved with a custom dockerfile and fixing a lot of dependency conflicts