Redis down?
Hello,
There seems to be connection problems to both Postgres servers and Redis servers (mostly Redis).
I get this error all the time and I cannot get the servers to connect:
Error: connect ECONNREFUSED 127.0.0.1:6379
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1555:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '127.0.0.1',
port: 6379
}
I tried redeploying, restarting. Did not work.
I have not changed anything, it just stopped working.
Project ID: 312b74d3-7061-448f-a297-539dd2bfbb7e
28 Replies
Project ID:
312b74d3-7061-448f-a297-539dd2bfbb7esame for me for a pg database, private db url does not work anymore
public one does
It worked for ten minutes and now it does not work again. Very bad. I have made no code changes that could impact this.
Very hard to rely on Railway when these things happens now and then 😦
I'm experiencing something similar where my Postgres and Redis on Digital Ocean can't be reached from Railway services 😬
Yes it seems that there is an internal networking issue of some sort on Railway because the databases themselfs can be access through for example Postico.
@Brody what is happening?
It works and then 10 min later it stops working (often after new deploy)
@Brody Any ETA when this will be fixed, been many hours now.
#🛂｜readme #5
Well, sorry about the breach of rules but the servers has been down for hours and I get no response and this is serious. My service is unusable.
I completely understand, hopefully someone sees this soon 🙏
If it's that critical u should consider upgrading to the Team plan.
And get email support with the team
I am on the PRO plan and that includes Email Support I see now. I thought this was the place to get help.
This is community help
A lot of ppl are getting the same issue as you, but i can't tell if it was alerted to the team since seems like no conductors are active rn
But it should be adressed soon.
We're investigating this right now. Apologies for the delay
We're currently experiencing intermittent networking issues impacti...
We're currently experiencing intermittent networking issues impacting some services in europe-west4 - Incident details - Railway Status
@dkrusenstrahle
Error: connect ECONNREFUSED 127.0.0.1:6379Are you sure you're using the correct connection string for Redis? Your issue might be unrelated
That one may be unrelated perhaps, but the variable is redis_url and that points to the Redis server. But I get other issues. It time out because the connection is not working.
No code changed and it has worked up till today and it works for 10-20 min sometimes and then fails again (server crashes after timeout).
Gotcha
And Redis is the one timing out, right?
From what I can gather from the logs but sometimes postgres also times out.
This is the latest logs:
at Socket.<anonymous> (/app/node_modules/ioredis/built/Redis.js:170:41)
at Object.onceWrapper (node:events:631:28)
at Socket.emit (node:events:517:28)
at Socket._onTimeout (node:net:598:8)
at listOnTimeout (node:internal/timers:569:17)
at process.processTimers (node:internal/timers:512:7) {
errorno: 'ETIMEDOUT',
code: 'ETIMEDOUT',
syscall: 'connect'
}
Error connecting to PostgreSQL: ConnectionAcquireTimeoutError [SequelizeConnectionAcquireTimeoutError]: Operation timeout
at ConnectionManager.getConnection (/app/node_modules/sequelize/lib/dialects/abstract/connection-manager.js:206:15)
at async /app/node_modules/sequelize/lib/sequelize.js:305:26
at async Sequelize.authenticate (/app/node_modules/sequelize/lib/sequelize.js:457:5)
at async PostgresService.connect (/app/services/postgres_service.js:39:7) {
parent: TimeoutError: Operation timeout
at Timeout.<anonymous> (/app/node_modules/sequelize-pool/lib/Deferred.js:17:25)
at listOnTimeout (node:internal/timers:569:17)
at process.processTimers (node:internal/timers:512:7),
original: TimeoutError: Operation timeout
at Timeout.<anonymous> (/app/node_modules/sequelize-pool/lib/Deferred.js:17:25)
at listOnTimeout (node:internal/timers:569:17)
at process.processTimers (node:internal/timers:512:7)
}
node:internal/process/promises:288
triggerUncaughtException(err, true /* fromPromise */);
^
ConnectionAcquireTimeoutError [SequelizeConnectionAcquireTimeoutError]: Operation timeout
at ConnectionManager.getConnection (/app/node_modules/sequelize/lib/dialects/abstract/connection-manager.js:206:15)
at async /app/node_modules/sequelize/lib/sequelize.js:305:26
at async Sequelize.authenticate (/app/node_modules/sequelize/lib/sequelize.js:457:5)
at async PostgresService.connect (/app/services/postgres_service.js:39:7) {
parent: TimeoutError: Operation timeout
at Timeout.<anonymous> (/app/node_modules/sequelize-pool/lib/Deferred.js:17:25)
at listOnTimeout (node:internal/timers:569:17)
at process.processTimers (node:internal/timers:512:7),
original: TimeoutError: Operation timeout
at Timeout.<anonymous> (/app/node_modules/sequelize-pool/lib/Deferred.js:17:25)
at listOnTimeout (node:internal/timers:569:17)
at process.processTimers (node:internal/timers:512:7)
}
Node.js v18.18.2
error: script "start" exited with code 1 (SIGHUP)
Its like the authentication to the servers are not working. I have changed no code and its been working fine for weeks.
I am using the public url if that helps (not internal one)Can you paste a link to where which service these logs are from?
I see them in the backend service logs
What link do you need?
I think Ray is trying to figure out which service those logs are from, so a link to your service
it would look something like this:
railway.app/project/43d250e5-ab9b-4341-8828-5e344d20524b/service/8f1b6722-7196-4cdd-8cb0-27ff50aa7a21Railway
404 - Page not found
Railway is an infrastructure platform where you can provision infrastructure, develop with that infrastructure locally, and then deploy to the cloud.
Are you still seeing this issue?
@dkrusenstrahle
@dkrusenstrahle
Seems to work now
What are Railway doing to make sure a thing like this will not happen again. It was very painful and many hours gone to waste. Will there be better safe guards out in place?
@dkrusenstrahle - What company/Pro workspace are you here on behalf for? I would love to refund you for this month and will share some mitigation steps so that we can try to restore your confidence on the platform.
Thank you @Angelo , very nice of you! My project id is 312b74d3-7061-448f-a297-539dd2bfbb7e if that helps?
BTW, there seems to be some other issues right now. One of my projects has been deploying for 20 minutes now (and counting), it usually takes 3-4 minutes.
I really love the simplicity of Railway, you guys really rocks in terms of functionality, support and ease of use. Just get on top of these stability issues and you will be the best. After all, stability is number one all day long.
Your deployment failing to go out is possibly related to: https://status.railway.app/clq29e6po141569bkokxcrhik8b
Should be fixed soon 🙂
Yepp, works now, thanks!