Django Redis Queue Application Failed to Respond
Hi everyone, my Django web app is down with a 503 error. I'm using Django RQ with one worker started by default as a separate service with this command
/bin/bash -l -c "python3 manage.py rqworker default" which was working. The deployment of my main Django repo is successful, but I have these deploy logs:
In which a worker timeouts, exits, and then a new one boots over and over. I have the worker timeout setting at 3600 (1 hour) for my three queues. The one worker I have started by default's pid does not seem to match any of the pids in my deploy logs after checking with this code:
And
How should I continue debugging this?
Project ID: 26297924-d91d-448c-b1fa-e00612d5579550 Replies
Project ID:
26297924-d91d-448c-b1fa-e00612d55795what protocol do rq works communicate with?
I think they use RESP (REdis Serialization Protocol)
railway only supports http services
My bad, it seems they use the Redis-based Python queuing library RQ
if its not http it wont work on railway
Thank you for letting me know. I will continuing doing research 👍
This is an explanation generated by ChatGPT and verified using phind.com (ChatGPT search engine). It aims to shed light on the workings of HTTP in the context mentioned above:
Django RQ integrates with Django, which is a web framework for building web applications in Python. Django handles the HTTP part of your application, including handling incoming requests, routing, views, and generating responses.
Django RQ complements Django by providing a way to offload time-consuming tasks to background workers using Redis as the message broker. When a Django view or other part of your Django application enqueues a task using Django RQ, it sends a message to Redis with the task details. The RQ workers, which are separate processes or threads, then pick up these tasks from Redis and execute them in the background.
So, while Django RQ itself does not directly handle HTTP communication, it works in conjunction with Django, which takes care of the HTTP aspects of your application. Django RQ focuses on the queuing and execution of tasks in the background, leaving Django to handle the web-related functionality.
Glad to learn more about this! Additionally, I would like to mention that the RQ functionality of my application was functioning properly until yesterday.
now what service are you getting a 503 from, the django app itself?
I'm getting the 503 from the dev tools console of my website. Here's the link: samplegen.com
It loads for a while, then says Application failed to respond.
and that is the django app in question
Yes.
Thank you for helping me investigate this!
do you have a repo for this app?
Yes
I'm not sure if I can really share it though.
understandable, do you have a procfile?
No, but I have these start commands, which replaced the Procfile when I switched from Heroku:
Django Web App:
/bin/bash -l -c "python manage.py migrate && gunicorn config.wsgi && python3 manage.py collectstatic --no-input"
Redis Worker (Same repo, separate service and different start command):
/bin/bash -l -c "python3 manage.py rqworker default"
1. how are you telling railway to run these commands?
2. it doesnt look like you are starting a gunicorn server?
no gunicorn server, no app
Sorry, I didn't copy the whole command when I first wrote that. Here are the full start commands:
Django Web App:
/bin/bash -l -c "python manage.py migrate && gunicorn config.wsgi && python3 manage.py collectstatic --no-input"
Redis Worker (Same repo, separate service and different start command):
/bin/bash -l -c "python3 manage.py rqworker default"
However, I noticed it looks like I accidently put this in the build command:
/bin/bash -l -c "python3 manage.py collectstatic --no-input" 🤦♂️
It's redeploying now, I'll keep you posted.
i still dont see gunicorn?
This is part of the start command for my Django web app repo:
gunicorn config.wsgi
The full start command is:
/bin/bash -l -c "python manage.py migrate && gunicorn config.wsgi && python3 manage.py collectstatic --no-input"
the full start command does not include gunicorn though??
without seeing your repo, i am quite confused
Sorry about that, I will see what I can do about showing the repo.
It looks like guincon is starting from this command:
gunicorn config.wsgi
Here are my deploy logs:
do you have some sort of gunicorn config file?
No, by default Django handles gunicorn in the background and runs the web app with a command like this
gunicorn project_name.wsgi using the wsgi.py file. It is possible to add a gunicorn config file to Django for further customization though.
Thank you for spending the time to help! 🙂something is telling gunicorn to listen on port 8000
show me a screenshot of your service variables please?
1. ive never seen someone with so many variables
2. what are those little numbers for?
3. try deleting the PORT variable, railway will make one for you
True, gotta appease the microservices 😂
I shared my variables because my RQ worker service needs them as it's the same repo as my website service repo running with a different start command (I may not need my RQ worker service anymore as I was successfully starting workers dynamically for a few weeks before today)
I deleted the PORT variable - will keep you updated.
Here are my Deploy Logs after deleting the PORT variable:
I got a new port number 🙂
yep now railway is randomly generating a PORT for you
this is a good thing, and preferred over setting your own
what is the railway domain for this service?
Thank you!
Here's the railway domain:
djangogeneratesamples-production.up.railway.app
just keeps loading
not even a 503
Hmm, interesting
I think I'm going to try rolling back to an earlier deployment that worked previously.
and im about all out of ideas
wait
It works!
rollback?
I didn't rollback. Something you helped me with worked 😃
sounds tempermental
do a redeploy
Okay
It just finished redeploying and the website is now loading endlessly.
has the deploy logs already displayed
Listening at: http://0.0.0.0:PORTYes, this is that line from the deploy logs:
[2023-06-01 02:30:37 +0000] [54] [INFO] Listening at: http://0.0.0.0:6688 (54)do you see worker timeouts?
Yes, these are the full deploy logs:
okay so gunicorn is starting correctly, but your app itself isnt responding
Yes, I don't know why it just finished loading 🤔
sounds like you have some debuging to do
since this is an issue with your code, theres not much more i can do for you unfortunately
No problem, you've been amazing! Thank you so much! I'm glad we could narrow it down 😃
glad i could help somewhat!
id be very intrested to know what the cause was once you nail this issue down
Definitely, I'll keep you updated! Learning and using Redis queues for this project has been interesting.
feel free to ping when solved 🙂
Okay awesome 🙂