Railway•10mo ago

How to setup celery worker in django+redis project?

I have a django+celery+redis setup. I can use django and redis no problem. I can also trigger endpoint to start task. task func is defined within the django project. Problem is those task dont get completed bc they are not sent to any celery worker. How do I setup a celery worker in railway? I tried to start a new celery service and I cant find it from options. Do I pull it from a docker image? This is my docker-compose.yaml for how things are getting setup in my dev, if this can help understand my setup better

# docker-compose.yaml
version: '3.8'

services:
  redis:
    image: redis:7.0.11-alpine
    ports:
      - "6379:6379"


  django:
    container_name: django
    build:
      context: ./dj_project
    command: python manage.py runserver
    volumes:
      - ./dj_project:/usr/src/app
    ports:
      - 8001:8000
    depends_on:
      - redis 


  celery_1:
    container_name: celery_1
    build:
      context: ./dj_project
    command: celery -A dj_project worker -l INFO -Q queue1
    volumes:
      - ./dj_project:/usr/src/app # c1 is within dj_app folder
    depends_on:
      - redis
      - django


  celery_2:
    container_name: celery_2
    build:
      context: ./celery_2
      dockerfile: Dockerfile
    command: celery -A celery_2 worker -l INFO -Q queue2
    volumes:
      - ./celery_2:/usr/src/app # c2 is outside dj_app folder
    depends_on:
      - redis
      - django


  flower:
    image: mher/flower
    ports:
      - "5555:5555"
    environment: 
       - CELERY_BROKER_URL=redis://redis:6379/0
    depends_on:
      - redis

# docker-compose.yaml
version: '3.8'

services:
  redis:
    image: redis:7.0.11-alpine
    ports:
      - "6379:6379"


  django:
    container_name: django
    build:
      context: ./dj_project
    command: python manage.py runserver
    volumes:
      - ./dj_project:/usr/src/app
    ports:
      - 8001:8000
    depends_on:
      - redis 


  celery_1:
    container_name: celery_1
    build:
      context: ./dj_project
    command: celery -A dj_project worker -l INFO -Q queue1
    volumes:
      - ./dj_project:/usr/src/app # c1 is within dj_app folder
    depends_on:
      - redis
      - django


  celery_2:
    container_name: celery_2
    build:
      context: ./celery_2
      dockerfile: Dockerfile
    command: celery -A celery_2 worker -l INFO -Q queue2
    volumes:
      - ./celery_2:/usr/src/app # c2 is outside dj_app folder
    depends_on:
      - redis
      - django


  flower:
    image: mher/flower
    ports:
      - "5555:5555"
    environment: 
       - CELERY_BROKER_URL=redis://redis:6379/0
    depends_on:
      - redis

3 Replies

Percy•10mo ago

Project ID: 187a5a9d-3902-45ea-ab25-003405c7240a

Evil AnyaOP•10mo ago

187a5a9d-3902-45ea-ab25-003405c7240a I solved my problem. So I created a duplicate service, linking to the same github repo, same root path. In settings, custom start command, I added celery -A dj_project worker -l INFO -Q queue1 In entrypoint.sh used by dockerfile, I added a condition

#!/bin/bash

# Check if the first argument passed is "celery". Gotta do this method bc one worker is wrapped within DJ proj
if [ "$1" = "celery" ]; then
    # Start Celery worker
    echo "Starting Celery worker..."
    shift  # This removes the first argument ("celery") from the arguments list
    exec celery -A dj_project "$@"
else
    # migrating db changes
    echo "migrating db changes..."
    python manage.py migrate

    # Collect static files
    echo "Collecting static files..."
    python manage.py collectstatic --noinput
    
    # # Start Gunicorn server
    PORT=${PORT:-8000} # default port (8000) if the PORT environment variable is not set.
    echo "PORT is set to '$PORT'"
    echo "Running on http://localhost:$PORT"
    exec gunicorn dj_project.wsgi:application --bind 0.0.0.0:$PORT --log-level info # change to debug to view more logs
fi

#!/bin/bash

# Check if the first argument passed is "celery". Gotta do this method bc one worker is wrapped within DJ proj
if [ "$1" = "celery" ]; then
    # Start Celery worker
    echo "Starting Celery worker..."
    shift  # This removes the first argument ("celery") from the arguments list
    exec celery -A dj_project "$@"
else
    # migrating db changes
    echo "migrating db changes..."
    python manage.py migrate

    # Collect static files
    echo "Collecting static files..."
    python manage.py collectstatic --noinput
    
    # # Start Gunicorn server
    PORT=${PORT:-8000} # default port (8000) if the PORT environment variable is not set.
    echo "PORT is set to '$PORT'"
    echo "Running on http://localhost:$PORT"
    exec gunicorn dj_project.wsgi:application --bind 0.0.0.0:$PORT --log-level info # change to debug to view more logs
fi

Brody•10mo ago

yeah that's certainly one way to do it

Gaming

Programming

How to setup celery worker in django+redis project?