Deploying Django with celery

Hi everyone. I have a simple django api service, which makes use of celery. for handling the background tasks. Usually in local, I start django in one shell
python manage.py runserver
python manage.py runserver
and celery in another shell
celery -A backend worker -l INFO -P gevent
celery -A backend worker -l INFO -P gevent
and its works just fine. But I am not sure how to do that in railway. My current proc file looks like this .
web: python manage.py migrate && python manage.py runserver 0.0.0.0:$PORT
worker: railway run celery -A backend worker --pool=prefork --concurrency=8
web: python manage.py migrate && python manage.py runserver 0.0.0.0:$PORT
worker: railway run celery -A backend worker --pool=prefork --concurrency=8
Thanks. Any help is much appreciated?
36 Replies
Percy
Percy2y ago
Project ID: N/A
hiransarkar
hiransarkar2y ago
N/A
Brody
Brody2y ago
1. you aren't using gunicorn, you should definitely be using gunicorn 2. railway's Procfile support is not 1:1 functionally equivalent to heroku's Procfile support, thus railway does not support defining two processes to be run in parallel, you will want to use a single command with a single & the order I think would work best would be to start the celery worker first. when on railway, you don't need to use the railway cli anymore
hiransarkar
hiransarkar2y ago
I am using daphne for my websockets. Is it okay if i just use daphne instead of gunicorn? It will look something like this
web: celery -A backend worker --pool=prefork --concurrency=8 & python manage.py migrate && daphne -b 0.0.0.0 -p $PORT backend.asgi:application
web: celery -A backend worker --pool=prefork --concurrency=8 & python manage.py migrate && daphne -b 0.0.0.0 -p $PORT backend.asgi:application
Brody
Brody2y ago
yeah looks good to me, give it a try?
hiransarkar
hiransarkar2y ago
its building.
hiransarkar
hiransarkar2y ago
Just got hit with this error.
Brody
Brody2y ago
the errors itself tells you how to fix that
hiransarkar
hiransarkar2y ago
Yeah, it does. I should add the code snippet, in my django project itself. Sorry, about my previous message. I don't think there is a way to do that in the proc file
Brody
Brody2y ago
imo create a python file that does the download and run it before you start daphne
hiransarkar
hiransarkar2y ago
okay. but how do run that file in the ProcFile? Something like this?
web: celery -A backend worker --pool=prefork --concurrency=8 & python manage.py migrate && python3 download_punkt.py && daphne -b 0.0.0.0 -p $PORT backend.asgi:application
web: celery -A backend worker --pool=prefork --concurrency=8 & python manage.py migrate && python3 download_punkt.py && daphne -b 0.0.0.0 -p $PORT backend.asgi:application
Brody
Brody2y ago
yes, see you know stuff!
hiransarkar
hiransarkar2y ago
it worked! But got this error. even though i tried to update the file which download punkt to also download the chrome driver. here is the file
import nltk
from webdriver_manager.chrome import ChromeDriverManager


# downloading punkt
nltk.download('punkt')

# downloading chrome driver
ChromeDriverManager().install()
import nltk
from webdriver_manager.chrome import ChromeDriverManager


# downloading punkt
nltk.download('punkt')

# downloading chrome driver
ChromeDriverManager().install()
Brody
Brody2y ago
what are you using chrome for?
hiransarkar
hiransarkar2y ago
the django app uses chrome to do some scraping.
Brody
Brody2y ago
does the site you are scrapping allow scrapping?
hiransarkar
hiransarkar2y ago
yes, it does.
Brody
Brody2y ago
🤨
hiransarkar
hiransarkar2y ago
or I think so. I am scraping one of the pages of langchain docs
Brody
Brody2y ago
sounds harmless enough
hiransarkar
hiransarkar2y ago
yeah. i didn't think it would be a problem, since it was open source
Brody
Brody2y ago
you are installing the chrome driver, but dont you also need to install chrome too
hiransarkar
hiransarkar2y ago
okay.. I think then the proc file should look like this?
celery -A backend worker --pool=prefork --concurrency=8 & python manage.py migrate && python3 download_resources.py && apt-get update && apt-get install -y google-chrome-stable && daphne -b 0.0.0.0 -p $PORT backend.asgi:application
celery -A backend worker --pool=prefork --concurrency=8 & python manage.py migrate && python3 download_resources.py && apt-get update && apt-get install -y google-chrome-stable && daphne -b 0.0.0.0 -p $PORT backend.asgi:application
does proc file support apt-get install?
Brody
Brody2y ago
it would work, but do not do it like that hold remove the apt-get command from your procfile and use this as a nixpacks.toml file
[phases.setup]
nixPkgs = ["...", "chromium"]
[phases.setup]
nixPkgs = ["...", "chromium"]
hiransarkar
hiransarkar2y ago
does railway automcatically picks up the nixpacks.toml file form the repository? or do i need to mention it in the proc file? thanks!
Brody
Brody2y ago
automatically
hiransarkar
hiransarkar2y ago
Still the same error
selenium.common.exceptions.WebDriverException: Message: Service /root/.cache/selenium/chromedriver/linux64/114.0.5735.90/chromedriver unexpectedly exited. Status code was: 127
selenium.common.exceptions.WebDriverException: Message: Service /root/.cache/selenium/chromedriver/linux64/114.0.5735.90/chromedriver unexpectedly exited. Status code was: 127
This is what i got when i searched the build logs for
```
Using Nixpacks
Nixpacks file based configuration is experimental and may change
╔═══════════════════════════════ Nixpacks v1.9.0 ══════════════════════════════╗
#3 [internal] load metadata for ghcr.io/railwayapp/nixpacks:ubuntu-1684957838
#4 [stage-0 1/10] FROM ghcr.io/railwayapp/nixpacks:ubuntu-1684957838@sha256:eb26a5ad60faad269d01ac896a4eefb5e7987040d39549dbe5d8cdfd1a830b75
#7 [stage-0 3/8] COPY .nixpacks/nixpkgs-293a28df6d7ff3dec1e61e37cc4ee6e6c0fb0847.nix .nixpacks/nixpkgs-293a28df6d7ff3dec1e61e37cc4ee6e6c0fb0847.nix
#8 [stage-0 4/8] RUN nix-env -if .nixpacks/nixpkgs-293a28df6d7ff3dec1e61e37cc4ee6e6c0fb0847.nix && nix-collect-garbage -d
```
Using Nixpacks
Nixpacks file based configuration is experimental and may change
╔═══════════════════════════════ Nixpacks v1.9.0 ══════════════════════════════╗
#3 [internal] load metadata for ghcr.io/railwayapp/nixpacks:ubuntu-1684957838
#4 [stage-0 1/10] FROM ghcr.io/railwayapp/nixpacks:ubuntu-1684957838@sha256:eb26a5ad60faad269d01ac896a4eefb5e7987040d39549dbe5d8cdfd1a830b75
#7 [stage-0 3/8] COPY .nixpacks/nixpkgs-293a28df6d7ff3dec1e61e37cc4ee6e6c0fb0847.nix .nixpacks/nixpkgs-293a28df6d7ff3dec1e61e37cc4ee6e6c0fb0847.nix
#8 [stage-0 4/8] RUN nix-env -if .nixpacks/nixpkgs-293a28df6d7ff3dec1e61e37cc4ee6e6c0fb0847.nix && nix-collect-garbage -d
This is what my app details looks like
Brody
Brody2y ago
looks like you might want to move to a Dockerfile based build
hiransarkar
hiransarkar2y ago
could you point me to sources that i can refer to for a Dockerfile based build. I am not really familiar with docker.
Brody
Brody2y ago
Google lol it seems like a cop out answer, but since I haven't done a python and chrome dockerfile myself, I'd be using Google
hiransarkar
hiransarkar2y ago
yeah. i thought maybe railway, provides the template for a dockerfile. or something you wouldn't believe me when i tell this but, you helped me a lot today. thanks man.
Brody
Brody2y ago
unfortunately not, such specific use cases are up to the users thank you 🙂
hiransarkar
hiransarkar2y ago
once i have the docker file, do i need to delete the proc file, runtime.txt and nixpacks.toml?
Brody
Brody2y ago
correct none of that stuff is applicable to a Dockerfile build
hiransarkar
hiransarkar2y ago
Hi. I buit the dockerfile, but I am having issues with versioning for Chromedriver and Chromium. They need to be the same.
[nltk_data] Package punkt is already up-to-date!
Traceback (most recent call last):
File "download_resources.py", line 12, in <module>
chrome_driver_path = ChromeDriverManager(chrome_type=ChromeType.CHROMIUM, version=chromium_version).install()
File "/usr/local/lib/python3.8/site-packages/webdriver_manager/chrome.py", line 39, in install
driver_path = self._get_driver_path(self.driver)
File "/usr/local/lib/python3.8/site-packages/webdriver_manager/core/manager.py", line 30, in _get_driver_path
file = self._download_manager.download_file(driver.get_driver_download_url())
File "/usr/local/lib/python3.8/site-packages/webdriver_manager/core/download_manager.py", line 28, in download_file
response = self._http_client.get(url)
File "/usr/local/lib/python3.8/site-packages/webdriver_manager/core/http.py", line 37, in get
self.validate_response(resp)
File "/usr/local/lib/python3.8/site-packages/webdriver_manager/core/http.py", line 16, in validate_response
raise ValueError(f"There is no such driver by url {resp.url}")
ValueError: There is no such driver by url https://chromedriver.storage.googleapis.com/90.0.4430/chromedriver_linux64.zip
Operations to perform:
[nltk_data] Package punkt is already up-to-date!
Traceback (most recent call last):
File "download_resources.py", line 12, in <module>
chrome_driver_path = ChromeDriverManager(chrome_type=ChromeType.CHROMIUM, version=chromium_version).install()
File "/usr/local/lib/python3.8/site-packages/webdriver_manager/chrome.py", line 39, in install
driver_path = self._get_driver_path(self.driver)
File "/usr/local/lib/python3.8/site-packages/webdriver_manager/core/manager.py", line 30, in _get_driver_path
file = self._download_manager.download_file(driver.get_driver_download_url())
File "/usr/local/lib/python3.8/site-packages/webdriver_manager/core/download_manager.py", line 28, in download_file
response = self._http_client.get(url)
File "/usr/local/lib/python3.8/site-packages/webdriver_manager/core/http.py", line 37, in get
self.validate_response(resp)
File "/usr/local/lib/python3.8/site-packages/webdriver_manager/core/http.py", line 16, in validate_response
raise ValueError(f"There is no such driver by url {resp.url}")
ValueError: There is no such driver by url https://chromedriver.storage.googleapis.com/90.0.4430/chromedriver_linux64.zip
Operations to perform:
Here is the code, which gets the version of the chromium and then downloads the chrome driver for that specific version
import nltk
from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.core.utils import ChromeType, read_version_from_cmd, PATTERN

# downloading punkt
nltk.download('punkt')

# Get the version of Chromium
chromium_version = read_version_from_cmd("/usr/bin/chromium --version", PATTERN[ChromeType.CHROMIUM])

# Install the corresponding version of ChromeDriver
chrome_driver_path = ChromeDriverManager(chrome_type=ChromeType.CHROMIUM, version=chromium_version).install()
import nltk
from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.core.utils import ChromeType, read_version_from_cmd, PATTERN

# downloading punkt
nltk.download('punkt')

# Get the version of Chromium
chromium_version = read_version_from_cmd("/usr/bin/chromium --version", PATTERN[ChromeType.CHROMIUM])

# Install the corresponding version of ChromeDriver
chrome_driver_path = ChromeDriverManager(chrome_type=ChromeType.CHROMIUM, version=chromium_version).install()
I was wondering if you have seen anything like this. I got it working! Chromium was a real pain in the ass. I am using google instead. Even though chromium is lightweight. I think I will stick to google chrome for a while. Here is the dockerfile
# Base image
FROM python:3.8.2

# Set the working directory in the container
WORKDIR /app

# Copy the project files to the container
COPY . .

# downloading chrome
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list
RUN apt-get update && apt-get install -y google-chrome-stable

# Install Python dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Expose the port if needed
# EXPOSE <port_number>

# Set environment variables if needed
# ENV <variable_name>=<value>

# Run the command
CMD celery -A backend worker -l INFO --pool=prefork --concurrency=8 & python manage.py migrate && python3 download_resources.py && daphne -b 0.0.0.0 -p $PORT backend.asgi:application
# Base image
FROM python:3.8.2

# Set the working directory in the container
WORKDIR /app

# Copy the project files to the container
COPY . .

# downloading chrome
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list
RUN apt-get update && apt-get install -y google-chrome-stable

# Install Python dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Expose the port if needed
# EXPOSE <port_number>

# Set environment variables if needed
# ENV <variable_name>=<value>

# Run the command
CMD celery -A backend worker -l INFO --pool=prefork --concurrency=8 & python manage.py migrate && python3 download_resources.py && daphne -b 0.0.0.0 -p $PORT backend.asgi:application
Thanks for all the help
Brody
Brody2y ago
hey thats awasome, great work!!
Want results from more Discord servers?
Add your server