Railway•2y ago

Deploying Django with celery

Hi everyone. I have a simple django api service, which makes use of celery. for handling the background tasks. Usually in local, I start django in one shell

python manage.py runserver

python manage.py runserver

and celery in another shell

celery -A backend worker -l INFO -P gevent

celery -A backend worker -l INFO -P gevent

and its works just fine. But I am not sure how to do that in railway. My current proc file looks like this .

web: python manage.py migrate && python manage.py runserver 0.0.0.0:$PORT
worker: railway run celery -A backend worker --pool=prefork --concurrency=8

web: python manage.py migrate && python manage.py runserver 0.0.0.0:$PORT
worker: railway run celery -A backend worker --pool=prefork --concurrency=8

Thanks. Any help is much appreciated?

36 Replies

Percy•2y ago

Project ID: N/A

hiransarkarOP•2y ago

N/A

Brody•2y ago

1. you aren't using gunicorn, you should definitely be using gunicorn 2. railway's Procfile support is not 1:1 functionally equivalent to heroku's Procfile support, thus railway does not support defining two processes to be run in parallel, you will want to use a single command with a single & the order I think would work best would be to start the celery worker first. when on railway, you don't need to use the railway cli anymore

hiransarkarOP•2y ago

I am using daphne for my websockets. Is it okay if i just use daphne instead of gunicorn? It will look something like this

web: celery -A backend worker --pool=prefork --concurrency=8 & python manage.py migrate && daphne -b 0.0.0.0 -p $PORT backend.asgi:application

web: celery -A backend worker --pool=prefork --concurrency=8 & python manage.py migrate && daphne -b 0.0.0.0 -p $PORT backend.asgi:application

Brody•2y ago

yeah looks good to me, give it a try?

hiransarkarOP•2y ago

its building.

hiransarkarOP•2y ago

Just got hit with this error.

message.txt

Brody•2y ago

the errors itself tells you how to fix that

hiransarkarOP•2y ago

Yeah, it does. I should add the code snippet, in my django project itself. Sorry, about my previous message. I don't think there is a way to do that in the proc file

Brody•2y ago

imo create a python file that does the download and run it before you start daphne

hiransarkarOP•2y ago

okay. but how do run that file in the ProcFile? Something like this?

web: celery -A backend worker --pool=prefork --concurrency=8 & python manage.py migrate && python3 download_punkt.py && daphne -b 0.0.0.0 -p $PORT backend.asgi:application

web: celery -A backend worker --pool=prefork --concurrency=8 & python manage.py migrate && python3 download_punkt.py && daphne -b 0.0.0.0 -p $PORT backend.asgi:application

Brody•2y ago

yes, see you know stuff!

hiransarkarOP•2y ago

it worked! But got this error. even though i tried to update the file which download punkt to also download the chrome driver. here is the file

import nltk
from webdriver_manager.chrome import ChromeDriverManager


# downloading punkt
nltk.download('punkt')

# downloading chrome driver
ChromeDriverManager().install()

import nltk
from webdriver_manager.chrome import ChromeDriverManager


# downloading punkt
nltk.download('punkt')

# downloading chrome driver
ChromeDriverManager().install()

message.txt

Brody•2y ago

what are you using chrome for?

hiransarkarOP•2y ago

the django app uses chrome to do some scraping.

Brody•2y ago

does the site you are scrapping allow scrapping?

hiransarkarOP•2y ago

yes, it does.

Brody•2y ago

🤨

hiransarkarOP•2y ago

or I think so. I am scraping one of the pages of langchain docs

Brody•2y ago

sounds harmless enough

hiransarkarOP•2y ago

yeah. i didn't think it would be a problem, since it was open source

Brody•2y ago

you are installing the chrome driver, but dont you also need to install chrome too

hiransarkarOP•2y ago

okay.. I think then the proc file should look like this?

celery -A backend worker --pool=prefork --concurrency=8 & python manage.py migrate && python3 download_resources.py && apt-get update && apt-get install -y google-chrome-stable && daphne -b 0.0.0.0 -p $PORT backend.asgi:application

celery -A backend worker --pool=prefork --concurrency=8 & python manage.py migrate && python3 download_resources.py && apt-get update && apt-get install -y google-chrome-stable && daphne -b 0.0.0.0 -p $PORT backend.asgi:application

does proc file support apt-get install?

Brody•2y ago

it would work, but do not do it like that hold remove the apt-get command from your procfile and use this as a nixpacks.toml file

[phases.setup]
nixPkgs = ["...", "chromium"]

[phases.setup]
nixPkgs = ["...", "chromium"]

hiransarkarOP•2y ago

does railway automcatically picks up the nixpacks.toml file form the repository? or do i need to mention it in the proc file? thanks!

Brody•2y ago

automatically

hiransarkarOP•2y ago

Still the same error

selenium.common.exceptions.WebDriverException: Message: Service /root/.cache/selenium/chromedriver/linux64/114.0.5735.90/chromedriver unexpectedly exited. Status code was: 127

selenium.common.exceptions.WebDriverException: Message: Service /root/.cache/selenium/chromedriver/linux64/114.0.5735.90/chromedriver unexpectedly exited. Status code was: 127

This is what i got when i searched the build logs for

```
Using Nixpacks
Nixpacks file based configuration is experimental and may change
╔═══════════════════════════════ Nixpacks v1.9.0 ══════════════════════════════╗
#3 [internal] load metadata for ghcr.io/railwayapp/nixpacks:ubuntu-1684957838
#4 [stage-0  1/10] FROM ghcr.io/railwayapp/nixpacks:ubuntu-1684957838@sha256:eb26a5ad60faad269d01ac896a4eefb5e7987040d39549dbe5d8cdfd1a830b75
#7 [stage-0 3/8] COPY .nixpacks/nixpkgs-293a28df6d7ff3dec1e61e37cc4ee6e6c0fb0847.nix .nixpacks/nixpkgs-293a28df6d7ff3dec1e61e37cc4ee6e6c0fb0847.nix
#8 [stage-0 4/8] RUN nix-env -if .nixpacks/nixpkgs-293a28df6d7ff3dec1e61e37cc4ee6e6c0fb0847.nix && nix-collect-garbage -d

```
Using Nixpacks
Nixpacks file based configuration is experimental and may change
╔═══════════════════════════════ Nixpacks v1.9.0 ══════════════════════════════╗
#3 [internal] load metadata for ghcr.io/railwayapp/nixpacks:ubuntu-1684957838
#4 [stage-0  1/10] FROM ghcr.io/railwayapp/nixpacks:ubuntu-1684957838@sha256:eb26a5ad60faad269d01ac896a4eefb5e7987040d39549dbe5d8cdfd1a830b75
#7 [stage-0 3/8] COPY .nixpacks/nixpkgs-293a28df6d7ff3dec1e61e37cc4ee6e6c0fb0847.nix .nixpacks/nixpkgs-293a28df6d7ff3dec1e61e37cc4ee6e6c0fb0847.nix
#8 [stage-0 4/8] RUN nix-env -if .nixpacks/nixpkgs-293a28df6d7ff3dec1e61e37cc4ee6e6c0fb0847.nix && nix-collect-garbage -d

This is what my app details looks like

Brody•2y ago

looks like you might want to move to a Dockerfile based build

hiransarkarOP•2y ago

could you point me to sources that i can refer to for a Dockerfile based build. I am not really familiar with docker.

Brody•2y ago

Google lol it seems like a cop out answer, but since I haven't done a python and chrome dockerfile myself, I'd be using Google

hiransarkarOP•2y ago

yeah. i thought maybe railway, provides the template for a dockerfile. or something you wouldn't believe me when i tell this but, you helped me a lot today. thanks man.

Brody•2y ago

unfortunately not, such specific use cases are up to the users thank you 🙂

hiransarkarOP•2y ago

once i have the docker file, do i need to delete the proc file, runtime.txt and nixpacks.toml?

Brody•2y ago

correct none of that stuff is applicable to a Dockerfile build

hiransarkarOP•2y ago

Hi. I buit the dockerfile, but I am having issues with versioning for Chromedriver and Chromium. They need to be the same.

[nltk_data]   Package punkt is already up-to-date!
Traceback (most recent call last):
File "download_resources.py", line 12, in <module>
chrome_driver_path = ChromeDriverManager(chrome_type=ChromeType.CHROMIUM, version=chromium_version).install()
File "/usr/local/lib/python3.8/site-packages/webdriver_manager/chrome.py", line 39, in install
driver_path = self._get_driver_path(self.driver)
File "/usr/local/lib/python3.8/site-packages/webdriver_manager/core/manager.py", line 30, in _get_driver_path
file = self._download_manager.download_file(driver.get_driver_download_url())
File "/usr/local/lib/python3.8/site-packages/webdriver_manager/core/download_manager.py", line 28, in download_file
response = self._http_client.get(url)
File "/usr/local/lib/python3.8/site-packages/webdriver_manager/core/http.py", line 37, in get
self.validate_response(resp)
File "/usr/local/lib/python3.8/site-packages/webdriver_manager/core/http.py", line 16, in validate_response
raise ValueError(f"There is no such driver by url {resp.url}")
ValueError: There is no such driver by url https://chromedriver.storage.googleapis.com/90.0.4430/chromedriver_linux64.zip
Operations to perform:

[nltk_data]   Package punkt is already up-to-date!
Traceback (most recent call last):
File "download_resources.py", line 12, in <module>
chrome_driver_path = ChromeDriverManager(chrome_type=ChromeType.CHROMIUM, version=chromium_version).install()
File "/usr/local/lib/python3.8/site-packages/webdriver_manager/chrome.py", line 39, in install
driver_path = self._get_driver_path(self.driver)
File "/usr/local/lib/python3.8/site-packages/webdriver_manager/core/manager.py", line 30, in _get_driver_path
file = self._download_manager.download_file(driver.get_driver_download_url())
File "/usr/local/lib/python3.8/site-packages/webdriver_manager/core/download_manager.py", line 28, in download_file
response = self._http_client.get(url)
File "/usr/local/lib/python3.8/site-packages/webdriver_manager/core/http.py", line 37, in get
self.validate_response(resp)
File "/usr/local/lib/python3.8/site-packages/webdriver_manager/core/http.py", line 16, in validate_response
raise ValueError(f"There is no such driver by url {resp.url}")
ValueError: There is no such driver by url https://chromedriver.storage.googleapis.com/90.0.4430/chromedriver_linux64.zip
Operations to perform:

Here is the code, which gets the version of the chromium and then downloads the chrome driver for that specific version

import nltk
from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.core.utils import ChromeType, read_version_from_cmd, PATTERN

# downloading punkt
nltk.download('punkt')

# Get the version of Chromium
chromium_version = read_version_from_cmd("/usr/bin/chromium --version", PATTERN[ChromeType.CHROMIUM])

# Install the corresponding version of ChromeDriver
chrome_driver_path = ChromeDriverManager(chrome_type=ChromeType.CHROMIUM, version=chromium_version).install()

import nltk
from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.core.utils import ChromeType, read_version_from_cmd, PATTERN

# downloading punkt
nltk.download('punkt')

# Get the version of Chromium
chromium_version = read_version_from_cmd("/usr/bin/chromium --version", PATTERN[ChromeType.CHROMIUM])

# Install the corresponding version of ChromeDriver
chrome_driver_path = ChromeDriverManager(chrome_type=ChromeType.CHROMIUM, version=chromium_version).install()

I was wondering if you have seen anything like this. I got it working! Chromium was a real pain in the ass. I am using google instead. Even though chromium is lightweight. I think I will stick to google chrome for a while. Here is the dockerfile

# Base image
FROM python:3.8.2

# Set the working directory in the container
WORKDIR /app

# Copy the project files to the container
COPY . .

# downloading chrome
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list
RUN apt-get update && apt-get install -y google-chrome-stable

# Install Python dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Expose the port if needed
# EXPOSE <port_number>

# Set environment variables if needed
# ENV <variable_name>=<value>

# Run the command
CMD celery -A backend worker -l INFO --pool=prefork --concurrency=8 & python manage.py migrate && python3 download_resources.py && daphne -b 0.0.0.0 -p $PORT backend.asgi:application

# Base image
FROM python:3.8.2

# Set the working directory in the container
WORKDIR /app

# Copy the project files to the container
COPY . .

# downloading chrome
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list
RUN apt-get update && apt-get install -y google-chrome-stable

# Install Python dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Expose the port if needed
# EXPOSE <port_number>

# Set environment variables if needed
# ENV <variable_name>=<value>

# Run the command
CMD celery -A backend worker -l INFO --pool=prefork --concurrency=8 & python manage.py migrate && python3 download_resources.py && daphne -b 0.0.0.0 -p $PORT backend.asgi:application

Thanks for all the help

Brody•2y ago

hey thats awasome, great work!!

Gaming

Programming

Deploying Django with celery