Railway•6mo ago

previous deployment crash

When I deploy the API (express, nodejs, socketio) everything goes well, but right before the new deployment will replace the old one, I see on the dashboard and I get an email notification
Deployment crashed for [service name] in [project name]! why this happens? is there something I can do? project id: b799250d-dd7b-47bb-be65-0ae878afd304

Solution:

I don't think yarn is passing signals down, can you have railway start index.js directly with node?

Jump to solution

33 Replies

Percy•6mo ago

Project ID: b799250d-dd7b-47bb-be65-0ae878afd304

Brody•6mo ago

did the deployment actually crash or is it a false report? I don't think you specified this information

whitetownOP•6mo ago

I think it's a false report. however it's slightly annoying. The new deployment is completely fine, deployed, works, and never crashes. But the previous one, when it's killed by railway, marked as CRASHED in the control panel and I also get an email about this. with the next deployment it happens again. unfortunately, there are no logs for previous deployments or explanation the source of the crash.

Brody•6mo ago

is your app handing sigterm properly?

whitetownOP•6mo ago

I guess, it did not. maybe this is what I am looking for

Brody•6mo ago

its not like you are using nodemon or pm2 right?

whitetownOP•6mo ago

no, I do not use nothing external. pure railway

whitetownOP•6mo ago

Brody•6mo ago

what do the logs say of that crashed service?

whitetownOP•6mo ago

nothing in logs, yarn run v1.22.22 $ node index.js Connection has been established successfully. server started at 3040

Brody•6mo ago

those are logs from the service that was marked as crashed?

whitetownOP•6mo ago

yes found something like How do I shut down my Express server gracefully... trying this. thanks for the tip

Brody•6mo ago

let me know how that goes! but, are you on the legacy or v2 runtime? check your service settings

whitetownOP•6mo ago

the service/project was created last month. I did not choose the runtime, and I hope that by default it's latest

Brody•6mo ago

well could you please check

whitetownOP•6mo ago

I do not know where yo see it

Brody•6mo ago

the service settings

whitetownOP•6mo ago

aha.. legacy shall I change it?

Brody•6mo ago

you can try it

whitetownOP•6mo ago

process.on('SIGTERM', shutDown);
process.on('SIGINT', shutDown);

function shutDown() {
    console.log('Received kill signal, shutting down gracefully');
    server.close(() => {
        console.log('Closed out remaining connections');
        process.exit(0);
    });

    setTimeout(() => {
        console.error('Could not close connections in time, forcefully shutting down');
        process.exit(1);
    }, 10000);
}

process.on('SIGTERM', shutDown);
process.on('SIGINT', shutDown);

function shutDown() {
    console.log('Received kill signal, shutting down gracefully');
    server.close(() => {
        console.log('Closed out remaining connections');
        process.exit(0);
    });

    setTimeout(() => {
        console.error('Could not close connections in time, forcefully shutting down');
        process.exit(1);
    }, 10000);
}

neither that code, nor v2 did not help. okay, I will try to make some experiments and come back later

Brody•6mo ago

do you see the Received kill signal, shutting down gracefully logs on the crashed service?

whitetownOP•6mo ago

no 😦

Brody•6mo ago

back to the drawing board

whitetownOP•6mo ago

okay, locally it catches sigterm and sigint. if I run the api locally and kill it somehow, I get in the console

yarn start
yarn run v1.22.21
$ node index.js
Connection has been established successfully.
server started at 3040
Shutting down gracefully...

yarn start
yarn run v1.22.21
$ node index.js
Connection has been established successfully.
server started at 3040
Shutting down gracefully...

on Railway this message does not appear. https://expressjs.com/en/advanced/healthcheck-graceful-shutdown.html

Brody•6mo ago

what is your start command on railway?

whitetownOP•6mo ago

just yarn start

Brody•6mo ago

thanks, what is your start script?

whitetownOP•6mo ago

{
    "$schema": "https://railway.app/railway.schema.json",
    "build": {
        "builder": "NIXPACKS",
        "watchPatterns": ["/apps/ai-api"]
    },
    "deploy": {
        "numReplicas": 1,
        "startCommand": "yarn --cwd apps/ai-api start",
        "healthcheckPath": "/",
        "sleepApplication": false,
        "restartPolicyType": "ON_FAILURE",
        "restartPolicyMaxRetries": 10
    }
}

{
    "$schema": "https://railway.app/railway.schema.json",
    "build": {
        "builder": "NIXPACKS",
        "watchPatterns": ["/apps/ai-api"]
    },
    "deploy": {
        "numReplicas": 1,
        "startCommand": "yarn --cwd apps/ai-api start",
        "healthcheckPath": "/",
        "sleepApplication": false,
        "restartPolicyType": "ON_FAILURE",
        "restartPolicyMaxRetries": 10
    }
}

Brody•6mo ago

what is the start script for apps/ai-api?

whitetownOP•6mo ago

"scripts": { "dev": "node --env-file=.env.local index.js", "start": "node index.js" },

Solution

Brody•6mo ago

I don't think yarn is passing signals down, can you have railway start index.js directly with node?

whitetownOP•6mo ago

ok @Brody I owe you a beer. Resolved. Thanks a lot!

Brody•6mo ago

no problem!

Gaming

Programming

previous deployment crash