Celery connection issue
Hi! Just recently I've been having issues with my celery instance. It was working all this time. It can't connect to redis (everything is dockerized). Here is the following issue: Cannot connect to {redacted redis address} Lookup timed out.. - Has anyone had the same issue?
64 Replies
Project ID:
N/AN/A
the redis address would be very important to help us debug this
redis-15409.c281.us-east-1-2.ec2.cloud.redislabs.com:15409
🙂
oh so not on railway
is there some kind of whitelist you need to do?
well I also tried railway as well
I'm having the same issue no matter the redis address
however it works just fine when connecting locally
can we focus on getting you connected to a railway hosted redis database?
local instance to production redis
that's dine with me
fine*
works better for me since i know nothing about redislabs
monorail.proxy.rlwy.net:34378
can i see a screenshot of your railway project please?
sure
give me a sec, let me configure redis on railway
if redis isn't configured, where did you get that domain from?
so that domain is from a heroku instance, I'm having the same issue and decided to try railway to see if it was a isolated instance
My application had been working just fine up until today
im talking about this domain
Oh I had tested a setup
earlier
I just tried connecting to it and i do get the same error
[2024-02-26 17:31:11,560: ERROR/MainProcess] consumer: Cannot connect to redis://default:**@monorail.proxy.rlwy.net:34378/0: Error -3 connecting to monorail.proxy.rlwy.net:34378. Lookup timed out..
please show me the screenshot of your project
just a normal screenshot? or is there a railway function
sorry, new to this
ignore the flask
okay can show me how you are trying to connect to it
let me make a quick loom
Thanks for the help!
I truly appreciate it
very helpful video, what do you have
REDIS_ENDPOINT and REDIS_PORT set to in your service variables?give me one sec
REDIS_ENDPOINT=default:**@monorail.proxy.rlwy.net
REDIS_PORT=34378
asterisk is the password btw
instead of trying to build the url yourself, just use a
REDIS_URL variable in the celery config file, then in your service variables set REDIS_URL to ${{Redis.REDIS_PRIVATE_URL}}
this is unlikely to fix the main issue, but its best we do this insteadokay got it - would my config look like this then?
import os
from dotenv import load_dotenv
load_dotenv()
REDIS_URL = os.environ.get('REDIS_URL', '6379') # default to 6379 if not provided
imports = ["tasks"]
broker_url = REDIS_URL
result_backend = REDIS_URL
task_serializer = 'json'
result_serializer = 'json'
accept_content = ['json']
timezone = 'America/Chicago'
enable_utc = True
broker_connection_retry_on_startup = True
lets omit the dotenv for now, since it looks like you are committing that file to your repo
okay gotcha
oaky got it - railway gives me two options REDIS_URL and REDIS_PRIVATE_URL
when I'm setting env varible
please reference my message
okay got it, deploying now
okay deployed -s ame issue
show me the new error please
[2024-02-26 17:54:49,592: ERROR/MainProcess] consumer: Cannot connect to redis://default:**@redis.railway.internal:6379//: Error -3 connecting to redis.railway.internal:6379. Lookup timed out..
cool, send your dockerfile please, and going forward please enclose logs and code (or similar) in code blocks
sorry haha , how do i enclose in code blocks
triple backticks
try this instead
trying
same issue
interesting
yeah, I'm not sure if it's a rediscloud issue but status have been okay with a few urgent fixes that happend last two days
as I said before, everything worked properly until today
does railway use the rediscloud service?
railway deploys a redis docker image
hmm.
yeah, it's really odd
driving me insane
increase the sleep to 10 seconds?
sure ill try it
same issue :/
im kinda out of ideas, you sure you are usng the dockerfile i provided?
Yes I am :9
😦
I'm dumbfounded as well. It literally worked for months before today
and you haven't modified anything at all about the redis database on railway right? absolutely nothing at all?
Nope
nothing
I even tried rabbitmq instead of redis
same issue
let me do some thinking, could you share your repo, or add me to it?
Thank you! This is under a client repo but I can create another and add you in
yes please
brody192
Hmm
well, here's another thing
I tried a skeleton celery
and it's connecting
I'm going to look into this further
I'll keep you posted 🙂
Thank you Brody
I will definitley buy you some coffee
theres no need but i very much appreciate the gesture
Here you go ! https://www.buymeacoffee.com/brody192/c/8635280 - The world needs mroe people like you. Much appreciated
Buy Me a Coffee
Brody is Helping in the Railway Community Discord server!
Hello! 👋I don't work for Railway ðŸ«
I figured out what it was - it was the concurrency pool. Eventlet doesn't want to play nice with Docker I guess
sorry i couldn’t solve this but happy you have! and thank you so much for the train!!
a sleep is still needed, but you can bump the sleep down to 3 seconds
Okay sounds good! Why do you recommend the sleep?
You still helped, I appreciate it
because the private network takes about 3 seconds to be able to respond to dns lookups
Ah okay gotcha
makes sense
thank you again
happy to help where i can!