Railway•6mo ago

postgres terminating abnormally

for some reason when both the ui and i try to connect to my postgres instance, we seem to be getting failures. the error message i am seeing is

psql: error: connection to server at "monorail.proxy.rlwy.net" (000.000.000.000), port 00000 failed: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

psql: error: connection to server at "monorail.proxy.rlwy.net" (000.000.000.000), port 00000 failed: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

wondering how to approach this / debug it or maybe there is something i am doing wrong. my start command is

docker-entrypoint.sh -c 'shared_buffers=1GB' -c 'temp_buffers=32MB' -c 'max_connections=500' -c 'work_mem=256MB' -c 'maintenance_work_mem=256MB' -c 'max_wal_size=1GB' -c 'listen_addresses=*'

docker-entrypoint.sh -c 'shared_buffers=1GB' -c 'temp_buffers=32MB' -c 'max_connections=500' -c 'work_mem=256MB' -c 'maintenance_work_mem=256MB' -c 'max_wal_size=1GB' -c 'listen_addresses=*'

my docker container is https://hub.docker.com/r/gibsfinance/assets-postgres the only other difference is that i am using

/var/lib/postgresql/data/pgdata13

/var/lib/postgresql/data/pgdata13

for my PGDATA value. i did this change in an effort to get around this weird error that i was seeing where the db was complaining that the database was created by postgres v16, when i built and everything is using v13

Solution:

hey @gibsfinance the team just pushed a fix for this, can you let me know if you are able to remove those extra services?

Jump to solution

56 Replies

Percy•6mo ago

Project ID: 42bfb463-d11d-431d-b46e-4bceb96bd943

gibsfinanceOP•6mo ago

42bfb463-d11d-431d-b46e-4bceb96bd943 what is more is i don't even see any connection break in the logs. they are just chilling, waiting to accept connections

Brody•6mo ago

can you check to see if the database is using the legacy or V2 runtime? please also provide the deploy logs for the database, and a screenshot of the tcp proxy https://bookmarklets.up.railway.app/log-downloader/

gibsfinanceOP•6mo ago

seems to happen for both

gibsfinanceOP•6mo ago

deployment_logs_post...

gibsfinanceOP•6mo ago

monorail.proxy.rlwy.net:15788

Brody•6mo ago

can I get that screenshot please?

gibsfinanceOP•6mo ago

gibsfinanceOP•6mo ago

oh, wait, is the proxy forwarding to a different port? / do i need to fix the pg port on 5432? that was it i guess that's what i get for working too late at night

Brody•6mo ago

beat me to it though whenever you can, definitely connect to the database via the private network so you can avoid the egress fees

gibsfinanceOP•6mo ago

ah, interesting. ok. so the "private" network is available from outside of railway?

Brody•6mo ago

nope, wouldn't be very private then that's why I said whenever you can, because you aren't going to be able to use the private network to access the database from your local computer, but your app that runs in the same environment will be able to

gibsfinanceOP•6mo ago

ok. cool. thank you

Brody•6mo ago

let me know if you run into any more difficulties or have any more questions

gibsfinanceOP•6mo ago

actually, i am having some difficulty for some reason with accessing the db from other services. when i read, it says that it is unable to find the row i am looking for, which is correct, the row does not exist the migrations seem to have worked, but during data collection/seeding i hit this issues of being unable to aquire connection i am also having difficulty for some reason downloading git submodules that i access during seeding

Brody•6mo ago

can you share the specfic error?

gibsfinanceOP•6mo ago

yeah, so there are 2

'    at Client_PG.acquireConnection (/usr/src/app/node_modules/knex/lib/client.js:311:13)\n' +

'    at Runner.ensureConnection (/usr/src/app/node_modules/knex/lib/execution/runner.js:305:46)\n' +

'    at Runner.run (/usr/src/app/node_modules/knex/lib/execution/runner.js:30:30)\n' +

'    at Target.then (/usr/src/app/node_modules/knex/lib/builder-interface-augmenter.js:24:43)\n' +

'    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)'

'    at Client_PG.acquireConnection (/usr/src/app/node_modules/knex/lib/client.js:311:13)\n' +

'    at Runner.ensureConnection (/usr/src/app/node_modules/knex/lib/execution/runner.js:305:46)\n' +

'    at Runner.run (/usr/src/app/node_modules/knex/lib/execution/runner.js:30:30)\n' +

'    at Target.then (/usr/src/app/node_modules/knex/lib/builder-interface-augmenter.js:24:43)\n' +

'    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)'

Brody•6mo ago

thats only 5 log lines, please use this - https://bookmarklets.up.railway.app/log-downloader/

gibsfinanceOP•6mo ago

deployment_logs_athl...

gibsfinanceOP•6mo ago

the other error is the no such file or directory, scandir line which i also find odd because it should have downloaded during the docker build

Brody•6mo ago

Dockerfile or nixpacks?

gibsfinanceOP•6mo ago

should be dockerfile

{
  "$schema": "https://railway.app/railway.schema.json",
  "build": {
    "builder": "DOCKERFILE",
    "dockerfilePath": "Dockerfile.worker"
  },
  "deploy": {
    "runtime": "LEGACY",
    "numReplicas": 1,
    "cronSchedule": "0 0 * * * *",
    "startCommand": "npm run collect",
    "sleepApplication": false,
    "restartPolicyType": "ON_FAILURE",
    "restartPolicyMaxRetries": 10
  }
}

{
  "$schema": "https://railway.app/railway.schema.json",
  "build": {
    "builder": "DOCKERFILE",
    "dockerfilePath": "Dockerfile.worker"
  },
  "deploy": {
    "runtime": "LEGACY",
    "numReplicas": 1,
    "cronSchedule": "0 0 * * * *",
    "startCommand": "npm run collect",
    "sleepApplication": false,
    "restartPolicyType": "ON_FAILURE",
    "restartPolicyMaxRetries": 10
  }
}

Brody•6mo ago

i dont even think 0 0 * * * * is a valid cron expression would you even want to be running a worker on a cron schedule? a worker is usually a long lived task

gibsfinanceOP•6mo ago

oh, it's just what i called it. it does one task for a couple minutes then exits

Brody•6mo ago

oh okay cool

gibsfinanceOP•6mo ago

Brody•6mo ago

looks like whatever frontend library railway is using can parse it, i would use the more standard 0 * * * * though

gibsfinanceOP•6mo ago

i think i may have figured out an issue with the git submodules

Brody•6mo ago

side question, are you seeing these characters in your logs on railway too? c-200-erc-20[24D[7B[10A[2K⠼ phux

gibsfinanceOP•6mo ago

so i deleted locally and tried rebuilding and that seemed to fail this is... i'll get you the package

Brody•6mo ago

right but do they show up in the logs window on railway?

gibsfinanceOP•6mo ago

@topcli/spinner oh, no they show up as the character. but the log line is updated so a lot of them show

Brody•6mo ago

can i see a screenshot of your logs from railway?

gibsfinanceOP•6mo ago

Brody•6mo ago

for the connection issue, try changing LEGACY to V2 thanks

gibsfinanceOP•6mo ago

ok, so i think the reason i am getting the submodules error is because it does not seem to have the submodules before the dockerfile is run

Brody•6mo ago

i unfortunately wouldnt know anything about submodules

gibsfinanceOP•6mo ago

is there a command that i can run before the docker file is run? if not i can just create a container i suppose this would be closer to editing the command for git pull https://www.answeroverflow.com/m/1082106108447043686

Brody•6mo ago

whats stopping you from making the necessary modifications to your Dockerfile so that it works properly?

gibsfinanceOP•6mo ago

it's just a wierd pathway. one that i did not expect. i have to initialize a git repository, then add the origin from github, then pull all the submodules is that the official way to do it? this works for me locally because i have all the submodules already but it seems like railway does not have them when it starts the docker build

Brody•6mo ago

right, the .git folder is not pulled

gibsfinanceOP•6mo ago

cool cool. working through this pathway am i charged for building large images? 2.45GB is what my local is showing

Brody•6mo ago

railway does not currently charge for the builder or the registry storage

gibsfinanceOP•6mo ago

ok, another interesting wrinkle nope. nevermind. application logic

gibsfinanceOP•6mo ago

ok, here's a question. i am reading files from the submodules i mentioned above

deployment_logs_athl...

Brody•6mo ago

you gotta turn off that spinner thing lol

gibsfinanceOP•6mo ago

i am seeing these fetch failures where it should be going to that path under usr fetch failure 'pls369' -> '/usr/src/app/submodules/pulsechain-assets/blockchain/pulsechain/assets/0xB64ef51C888972c908CFacf59B47C1AfBC0Ab8aC/logo.png' i am wondering if there are permissions that i need to be concerned about like fs permissions i'll work on it

Brody•6mo ago

"fetch failure" is no too helpfull, do you have a more verbose error?

gibsfinanceOP•6mo ago

working on that too basically, it failed to read from the file system

Brody•6mo ago

does the file exist at that location?

gibsfinanceOP•6mo ago

hm. thinking it's a different weird error. i'll investigate

gibsfinanceOP•6mo ago

is this anything to be concerned about? when i remove the active ones the ui eventually shows the top one as active

Brody•6mo ago

yeah I've been seeing that a lot

Solution

Brody•6mo ago

hey @gibsfinance the team just pushed a fix for this, can you let me know if you are able to remove those extra services?

gibsfinanceOP•6mo ago

hey, looks good on my end. thanks!

Brody•6mo ago

awsome!

Gaming

Programming

postgres terminating abnormally