Very weird "bug" - Celery (Redis)
Hi, I experience a very weird bug.
I deployed a FLASK web app, with as main .py files
main.py, at the root of the project, with most of the code, incl. a function which triggers a celery task located in tasks.py
celery_config.py (with the Celery configuration)
tasks.py, where I have my celery task
I have 2 services in my project
- a Redis database
- the Flask Web App
When I trigger in my web app the action which is supposed to launch the celery task, I see an entry (celery) in Redis, which proves that the web app could communicate with Redis, but the task doesn't actually start.
But then when I spin Celery in my local development instance (using the same start command as the one in my Procfile), on my computer, the task is actually triggered and it's completed.
The problem is that the output file which was supposed to be generated and stored in the cloud by the web app is then stored on my local machine, since the process actually completes on my local machine, not in the web app....
In my Procfile (for the Flask web app deployment), I inserted the 2 lines which are supposed to both spin the server for the web app and the celery service.
web: gunicorn main:app
worker: celery -A celery_config worker --loglevel=info
What am I missing?
Why isn't Celery working as expected in the cloud version of my app?
It works fine locally (connecting to the same Railway Redis database as a broker, via the same URL).
Solution:Jump to solution
i changed my mind, we'll try to do this all in a single config file, so lets see if this works..
add a railway.json to your project with this in it:
```json
{
"$schema": "https://schema.up.railway.app/railway.schema.json",...
93 Replies
Project ID:
N/AProject ID: a7e74948-3de2-4e70-afa7-622b30e041b0
railway does not natively support running parallel commands in a single service the same way heroku does.
you instead need two railway services, one that runs gunicorn, the other that runs clery.
this is achieved by having two near duplicate services in your railway project with the applicable start commands set in each service.
So I need to deploy twice my web app basically? With 2 different Procfile ?
yes but forego the procfile entirely, set the applicable start commands in the service settings
here?
So for one version:
web: gunicorn main:app
And for its clone:
worker: celery -A celery_config worker --loglevel=info
?
correct
What is the impact price wise of duplicating a full app?
the full app is not duplicated in the sence you think, one service will only be running gunicorn, the other service will only run celery
So no negative impact price wise?
im sure theres some, but this is simply how its done on railway
Why do you have to clone the full app?
Can't you just isolate some .py files for the Celery part?
why would that be necessary? if your start command is set to run celery than that is all that will be ran
So you could have some dummy empty project just to run Celery? Not the full Flask web app?
why would you want that? again, if your start command is set to run celery that is the only thing that will be ran on that induvial service
So in that service you would just need basically what's required to spin celery (in the requirements and in the code) but not the full code of the full app since Celery isn't used in the full app, just for 2 tasks...
again, why would you want to do that? you are trying to make this far harder on yourself development wise by breaking the repo up
By the way, I haven't cloned yet the full app but the first instance of the service doesn't start anymore.
errors...
the start command field is not in Procfile format, forego the
web: partgood to know...
and for Celery?
same thing
so no worker:
Just
celery -A celery_config worker --loglevel=info
you'd likely want to set a fixed concurrency value there, but yes, the start command field is not in Procfile format
Which concurrency value do you recommend?
It's a process which concatenates mp3 files, not machine learning heavy
start with 1, you can always increase at any time
celery -A celery_config worker --loglevel=info -c 1 ?
personally, id use the long form flag
celery -A celery_config worker --loglevel=info --concurrency 1
let's see if this thing will work now...
I spent 4 hours trying to figure out how to make Celery work on Railway.
I should have asked earlier....
haha yes you should have
you are missing the equals sign between that last flag, not sure if it matters but probably best to have it
OK so...
The "task" now completes.
But when I try to access the file where it's supposed to be... it isn't there.
I suspect that the web app triggers the Celery task, which is performed, kind of generates the expected file but it's not stored in the expected location.
where is the file supposed to be saved to?
in a folder at the root of the web app project
/output (the folder exists)
railway does not provide a native way to inspect the contents of the container, so im not sure how you are verifying that
I know that the folder exists because I force the creation of folders if they don't exist when the web app starts
the problem is that the file doesn't get saved... in the folder
(it's all fine in local development where both services are spun from the Terminal in the same environment)
the two services are in isolated containers
So what I suspect...
is that the file gets saved in the Celery app 🙂
I mean in the Celery version of the app...
which can't be accessed by the actual user-facing app
the flask service can not access the celery service's filesystem, that is correct
so basically I did all of that... for nothing
since my Celery task can't actually save a file available to the web app which triggered the task
wonderful
i was not aware you where doing such things
i was under the impression that you saved the file to storage that was accessible to both services
In the first place I didn't know I needed two cloned versions of the same app...
so my logic is just based on stuff happening in one app
not in version A & B spun separately for technical reasons 🙂
two services for separate apps is standard practice on railway, developing like you are using heroku is going to bring only headaches
I have never developed on Heroku... I've been a proud Railway customer for months.
It's just that it's the first time I need this Celery thing
in fact, ive heard through the grape vine that heroku regrets allowing the web and workers in the same container
Procfile is a herkou thing, railway only has minor support for it
OK, so basically, what I have to do is revamp my dual-head app and store the final file on S3
and communicate the S3 URL back to the download button instead of a file system location...
i mean there are better options then s3, but if you don't want to change code then we can run two apps in the same service if you would stronly prefer not to change code
in a custom way? Or is this a feature?
you could call it a custom way, is that something you are interested in?
What does it require at my end?
two config files that id write for you
OK, let me know exactly what I'm supposed to do and I'll test the result.
this means you are going back down to one service, so delete the celery service, remove the custom start command, and delete the Procfile
OK, Procfiles were removed from both. I'm deleting now the celery service.
from both?
I used the start command in both when they were separate services
im getting the feeling you duplicated your repo?
yes that's what I did
one starting as web, the other one starting as celery worker
one repo, two services
two services created from one single repo
with 2 different start commands
as instructed
but I can now delete the one we used to start Celery
why did you create a duplicate github repo?
I didn't create a duplicate github repo
I created 2 services on Railway from one repo
you duplicated your repo? yes that's what I did
Well, sorry for the misunderstanding... I'm getting a bit tired 🙂
So shall I delete the "duplicate" on Railway published to spin Celery?
yes
done
so I'm left now with one service with a gunicorn start command
I'm documenting what I'm doing so that this thread can be useful for random strangers
remove the start command, then save without deploying
alt + shift + enteralt shift enter doesn't seem to save on Windows (after removing the content of the field)
shall I press on the v ?
ok so
I pressed on the v and then I did alt shift enter
which worked
V?
the little v icon next to the x icon
im drawing a blank here, please show me the V icon
I removed the content of the field, pressed on v and then did alt-shift-enter
we're good
next = the custom file which will do the magic
the secret sauce
bruh thats a check mark
well yeah, which is a little "v" 🙂 anyway, we got there
show me your magic now
go to the optometrist after this 🤣
I promise
Solution
i changed my mind, we'll try to do this all in a single config file, so lets see if this works..
add a railway.json to your project with this in it:
OK, just one thing, I don't know if it has an impact.
I have a nickpacks.toml
[phases.setup]
nixPkgs = ["...", "ffmpeg"]
because I'm using ffmepg in the project
Do you need to add anything to the JSON for that?
i think they'd get merged, lets find out
OK, I will deploy this JSON
Building...
While we're waiting, you told me that S3 wasn't your preferred option for file storage. Anything special to suggest?
i only said that with the assumption you would have used an s3 storage provider not hosted on railway, for what you're doing using minio on railway would be the best choice if you needed s3
I didn't know about minio
I'm testing the new setup
It worked
alleluia
this isnt my first rodeo
Thanks so much!
I understood from your profile that you're a non-Railway volunteer?
yeah i dont work for railway, just help out
From what I can see you help out full time. What a dedication 👏
I hope you have free credits for life 🙂
haha thanks, i do
I've sent you a TrainCoffee
thank you very much!
Extra gift, created with one my apps on Railway, the AI Jingle Maker.
The one we've fixed today is the upcoming project, the AI Show Maker.
very cool!
#labouroflove