Railway•13mo ago

My server failed to response after migrating to new database

Checking the log is still ok but calling the custom domain (as well as my-server.up.railway.app domain) still can not response. It happens on production! Please help!

Solution:

look like redeploy the redis helps

Jump to solution

67 Replies

Percy•13mo ago

Project ID: 28677a80-2aaf-46d7-8fff-e2898fa09eb1

Sang DangOP•13mo ago

28677a80-2aaf-46d7-8fff-e2898fa09eb1 I tried to restart, as well as redeployed, still not respond. 🥹

Brody•13mo ago

nothing bad in the logs at all?

Sang DangOP•13mo ago

yeah, the log is green, just fail when trying to reach the domain the last thing I did is Migrate your plugin to a V2 database

Brody•13mo ago

the log is green? have you check the deployment logs though?

Sang DangOP•13mo ago

hmm, let me double check I found

2023-11-14 21:35:02.468 UTC [7] LOG:  starting PostgreSQL 15.4 (Ubuntu 15.4-2.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit

2023-11-14 21:35:02.468 UTC [7] LOG:  listening on IPv4 address "0.0.0.0", port 5432

2023-11-14 21:35:02.468 UTC [7] LOG:  listening on IPv6 address "::", port 5432

2023-11-14 21:35:02.473 UTC [7] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"

2023-11-14 21:35:02.481 UTC [34] LOG:  database system was interrupted; last known up at 2023-11-14 21:33:57 UTC

2023-11-14 21:35:02.523 UTC [34] LOG:  database system was not properly shut down; automatic recovery in progress

2023-11-14 21:35:02.468 UTC [7] LOG:  starting PostgreSQL 15.4 (Ubuntu 15.4-2.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit

2023-11-14 21:35:02.468 UTC [7] LOG:  listening on IPv4 address "0.0.0.0", port 5432

2023-11-14 21:35:02.468 UTC [7] LOG:  listening on IPv6 address "::", port 5432

2023-11-14 21:35:02.473 UTC [7] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"

2023-11-14 21:35:02.481 UTC [34] LOG:  database system was interrupted; last known up at 2023-11-14 21:33:57 UTC

2023-11-14 21:35:02.523 UTC [34] LOG:  database system was not properly shut down; automatic recovery in progress

and this

1:C 14 Nov 2023 21:43:55.103 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

1:C 14 Nov 2023 21:43:55.103 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

Brody•13mo ago

think I'm gonna tag in @jr here

Sang DangOP•13mo ago

the migration look like ok

[ OK ] Successfully saved dump to plugin_dump.sql

Dump file size: 14M

==== Restoring database to NEW_URL ====

DO

[ OK ] Successfully restored database to NEW_URL

Migration completed successfully

[ OK ] Successfully saved dump to plugin_dump.sql

Dump file size: 14M

==== Restoring database to NEW_URL ====

DO

[ OK ] Successfully restored database to NEW_URL

Migration completed successfully

Do I HAVE TO delete old services?

Brody•13mo ago

Sang DangOP•13mo ago

ok I will leave it there at least for a couple days

jr•13mo ago

Hey, Just checking things out here. The migration and databases look okay

Sang DangOP•13mo ago

after migrating the Postgres, I do the Redis as well

jr•13mo ago

It looks like the server is just failing to start fully But there are no errors or anything. Can you trying to kick off another deployment

Sang DangOP•13mo ago

I see one thing

node:events:492

throw er; // Unhandled 'error' event

^

Error: read ECONNRESET

at TCP.onStreamRead (node:internal/stream_base_commons:217:20)

Emitted 'error' event on Commander instance at:

at RedisSocket.<anonymous> (/app/node_modules/.pnpm/@[email protected]/node_modules/@redis/client/dist/lib/client/index.js:396:14)

at RedisSocket.emit (node:events:514:28)

at RedisSocket._RedisSocket_onSocketError (/app/node_modules/.pnpm/@[email protected]/node_modules/@redis/client/dist/lib/client/socket.js:217:10)

at Socket.<anonymous> (/app/node_modules/.pnpm/@[email protected]/node_modules/@redis/client/dist/lib/client/socket.js:191:127)

at Object.onceWrapper (node:events:629:26)

at Socket.emit (node:events:514:28)

at emitErrorNT (node:internal/streams/destroy:151:8)

at emitErrorCloseNT (node:internal/streams/destroy:116:3)

at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {

errno: -104,

code: 'ECONNRESET',

syscall: 'read'

}

Node.js v18.17.1

node:events:492

throw er; // Unhandled 'error' event

^

Error: read ECONNRESET

at TCP.onStreamRead (node:internal/stream_base_commons:217:20)

Emitted 'error' event on Commander instance at:

at RedisSocket.<anonymous> (/app/node_modules/.pnpm/@[email protected]/node_modules/@redis/client/dist/lib/client/index.js:396:14)

at RedisSocket.emit (node:events:514:28)

at RedisSocket._RedisSocket_onSocketError (/app/node_modules/.pnpm/@[email protected]/node_modules/@redis/client/dist/lib/client/socket.js:217:10)

at Socket.<anonymous> (/app/node_modules/.pnpm/@[email protected]/node_modules/@redis/client/dist/lib/client/socket.js:191:127)

at Object.onceWrapper (node:events:629:26)

at Socket.emit (node:events:514:28)

at emitErrorNT (node:internal/streams/destroy:151:8)

at emitErrorCloseNT (node:internal/streams/destroy:116:3)

at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {

errno: -104,

code: 'ECONNRESET',

syscall: 'read'

}

Node.js v18.17.1

jr•13mo ago

where do you see that? Perhaps I am looking at the wrong deployment

Sang DangOP•13mo ago

like the server can not connect to redis server (while on migration...)

jr•13mo ago

can you send the full link to these logs please

Sang DangOP•13mo ago

https://railway.app/project/28677a80-2aaf-46d7-8fff-e2898fa09eb1/logs?context=2023-11-14T21%3A45%3A24.495310000Z

jr•13mo ago

ty those errors appear to be from an older deployment Can you try redeploying plz

Sang DangOP•13mo ago

yeah, I will do it now

jr•13mo ago

hmm yeah, so seems to just halt after initialing the FilesModule. Can we try changing the start command to sleep 3 && node apps/server/dist/main.js

Sang DangOP•13mo ago

still the same

jr•13mo ago

hmm its hanging at the same spot. Can you try pointing at the previous databases. Trying to narrow down what the issue is. You can update the variables

DATABASE_URL=${{Postgres Legacy.DATABASE_URL}}
REDIS_URL=${{Redis Legacy.REDIS_URL}}

DATABASE_URL=${{Postgres Legacy.DATABASE_URL}}
REDIS_URL=${{Redis Legacy.REDIS_URL}}

If possible it might also be useful to increase the verbosity of Nestjs logging?

Sang DangOP•13mo ago

I will update the db url first

jr•13mo ago

Interesting, so that is fully crashing around the same spot. Likely the same thing

Sang DangOP•13mo ago

6379 is the redis, right?

jr•13mo ago

oh mb! It should be ${{ Redis Legacy.REDIS_URL }}. Not DATABASE_URL I've updated this

Sang DangOP•13mo ago

oh damn I did not see it let's wait for it redeploys

jr•13mo ago

ah okay that works. So it must be an issue connecting to one of the databases. Can you try just connecting to the new postgres? So we can narrow it down

Sang DangOP•13mo ago

using the new DATABASE_URL, I can connect using TablePlus app

jr•13mo ago

Yeah both of the migrated databases look okay It seems to be an issue with the app connecting

Sang DangOP•13mo ago

is there anything change (in technology) with the new? we use Prisma with PostgresQL client adapter under the hood

Brody•13mo ago

pg 14 to pg 15 is one thing i can think of

jr•13mo ago

Can you try narrowing down which database is having the issue

Sang DangOP•13mo ago

actually I can not log in using old database 😦 ~~I still can see the tables using TablePlus, but my NestJS app authentication can not login~~

jr•13mo ago

do you know where the auth is failing? Nothing was changed on the old database at all

Sang DangOP•13mo ago

1 sec my bad, it works well with old database I was using wrong username/password so, the old db works perfect

jr•13mo ago

I am unsure if it is failing to connect to redis or postgres

Sang DangOP•13mo ago

hmmm

Sang DangOP•13mo ago

the new redis

Sang DangOP•13mo ago

should I try to redeploy the new redis?

jr•13mo ago

Can you connect with that command?

Sang DangOP•13mo ago

I need to install redis cli 1 min

jr•13mo ago

I think that UI is timing out. But I'm not sure why the legacy one is not

Sang DangOP•13mo ago

yes I can connect to it

jr•13mo ago

Hmm. Can you try deploying the server while only connecting to one of the new databases?

Sang DangOP•13mo ago

ok I will swithc to use Postgres with the new one first yes it works with only new Postgres

jr•13mo ago

okay so it is a problem connecting to the redis. How does your app connect to it?

jr•13mo ago

ah and the module after the FileModule is the CacheModule, so that makes sense

jr•13mo ago

are you using something like the redis or ioredis npm packages

Sang DangOP•13mo ago

let me check the name - @nestjs/cache-manager - redisStore from 'cache-manager-redis-yet'; that cache-manager-redis-yet uses:

"dependencies": {
    "@redis/bloom": "^1.2.0",
    "@redis/client": "^1.5.8",
    "@redis/graph": "^1.1.0",
    "@redis/json": "^1.0.4",
    "@redis/search": "^1.1.3",
    "@redis/time-series": "^1.0.4",
    "cache-manager": "^5.2.3",
    "redis": "^4.6.7"
  },

"dependencies": {
    "@redis/bloom": "^1.2.0",
    "@redis/client": "^1.5.8",
    "@redis/graph": "^1.1.0",
    "@redis/json": "^1.0.4",
    "@redis/search": "^1.1.3",
    "@redis/time-series": "^1.0.4",
    "cache-manager": "^5.2.3",
    "redis": "^4.6.7"
  },

Look like it uses Redis under the hood

jr•13mo ago

hmm. testing a few things out

Sang DangOP•13mo ago

my cache module is simple

@Module({
  imports: [
    NestCacheModule.registerAsync<RedisClientOptions>({
      imports: [ConfigModule],
      useFactory: async (config: ConfigService) => ({
        isGlobal: true,
        store: await redisStore({ url: config.get<string>('REDIS_URL') }),
        ttl: config.get('CACHE_TTL'),
      }),
      inject: [ConfigService],
    }),
  ],
  providers: [CacheService],
  exports: [CacheService],
})
export class CacheModule {}

@Module({
  imports: [
    NestCacheModule.registerAsync<RedisClientOptions>({
      imports: [ConfigModule],
      useFactory: async (config: ConfigService) => ({
        isGlobal: true,
        store: await redisStore({ url: config.get<string>('REDIS_URL') }),
        ttl: config.get('CACHE_TTL'),
      }),
      inject: [ConfigService],
    }),
  ],
  providers: [CacheService],
  exports: [CacheService],
})
export class CacheModule {}

jr•13mo ago

I'm trying to reproduce this locally with the new redis database and am unable to. Can you connect to them locally? We can try restarting/redeploying the redis database as well

Sang DangOP•13mo ago

I still use redis in my docker for development but I can connect to it using the redis-cli let's redeploy the new redis

jr•13mo ago

What is so confusing is that there are no errors or anything I am not that familiar with NestJS, but if there is anyway to get more debug info, that would be really helpful We could also try deploying the same app to another prjoect with a new postgres/redis to see if that works. That would let us know if it is that specific redis, or if it is all of the new redis's that are not connectable by this server

Sang DangOP•13mo ago

I am not expert in nestjs as well, 30 mins ago I read the doc again and might try to push it if the redeploy does not help yeah, so strange! oh, look like it works

Sang DangOP•13mo ago

https://tenor.com/view/dancing-peachu-goma-and-peach-dudu-and-bubu-milk-mocha-sugar-and-bear-gif-26946040

Tenor

jr•13mo ago

oh wat

Sang DangOP•13mo ago

have no idea

jr•13mo ago

thats... good haha I'm so confused

Solution

Sang Dang•13mo ago

look like redeploy the redis helps

jr•13mo ago

very strange. Glad it worked. I'll definitely keep this in mind in the future And will explore some things later to see if I can determine what happened (not using your databases of course)

Sang DangOP•13mo ago

I guess because I redeployed the server right after the redis deploying, so it has something...

jr•13mo ago

awesome! glad everything is working now Feel free to keep the legacy databases around for a bit so you know everything is all good with the new ones

Sang DangOP•13mo ago

thanks for your patient I have to go to bed, it's 2:00 AM here 😄

jr•13mo ago

Good night 😴 👋

Gaming

Programming

My server failed to response after migrating to new database