Railway•2y ago

NextJS Application Random Crash

Hey 👋 I have a NextJS service running on Railway which suddenly crashed (there were no active users at that time so the app was idle). After a restart of the service, everything went back to normal. On the deploy logs I see a SIGTERM signal (see below), I'm not sure though if it's there because of the restart attempts. 1. If the crash wasn't because of the SIGTERM, how can I investigate this more thoroughly? 2. If the crash was because of the SIGTERM - why did that happen, and how can I prevent it or at least make sure Railway automatically restarts the service? Thanks!

> [email protected] start
> next start
ready - started server on 0.0.0.0:4200, url: http://localhost:4200
npm ERR! path /app/dist/apps/client-app
npm ERR! command failed
npm ERR! signal SIGTERM
npm ERR! command sh -c -- next start
npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2023-03-15T09_43_22_663Z-debug-0.log
npm WARN config production Use `--omit=dev` instead.
> [email protected] start
> next start
ready - started server on 0.0.0.0:4200, url: http://localhost:4200
npm ERR! path /app/dist/apps/client-app
npm ERR! command failed
npm ERR! signal SIGTERM
npm ERR! command sh -c -- next start
npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2023-03-19T00_29_49_514Z-debug-0.log

> [email protected] start
> next start
ready - started server on 0.0.0.0:4200, url: http://localhost:4200
npm ERR! path /app/dist/apps/client-app
npm ERR! command failed
npm ERR! signal SIGTERM
npm ERR! command sh -c -- next start
npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2023-03-15T09_43_22_663Z-debug-0.log
npm WARN config production Use `--omit=dev` instead.
> [email protected] start
> next start
ready - started server on 0.0.0.0:4200, url: http://localhost:4200
npm ERR! path /app/dist/apps/client-app
npm ERR! command failed
npm ERR! signal SIGTERM
npm ERR! command sh -c -- next start
npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2023-03-19T00_29_49_514Z-debug-0.log

31 Replies

Percy•2y ago

Project ID: 4c244bec-c9dc-4d97-b44d-c09dcf0b97d3

Percy•2y ago

You might find these helpful: - npm error when I shut down the server - Railway automatically removes deployment

⚠️ experimental feature

guypyOP•2y ago

4c244bec-c9dc-4d97-b44d-c09dcf0b97d3

Brody•2y ago

could you link your discord to railway please?

guypyOP•2y ago

Sure, linked it

Brody•2y ago

where you on the trial plan at the time?

guypyOP•2y ago

No, haven't been on trial for about 2 months I think

guypyOP•2y ago

This happened again today, restarting the service didn't help, but a redeploy seemed to fix the issue for now. Any ideas? @Brody

Brody•2y ago

#🛂｜readme #5

guypyOP•2y ago

oops sorry, won't happen again. Would appreciate any help on this issue 🙏

Brody•2y ago

send your repo please and show me what your metrics look like during these crashes

guypyOP•2y ago

In the 2nd image you can see the downtimes, started around 03:30am IDT, ended after I redeployed the serviced at 07:30am. The metrics seem stable 😕

guypyOP•2y ago

The Github repo is private unfortunately, is there anything specific I could share that might be helpful?

Brody•2y ago

package.json how much traffic does this site get?

guypyOP•2y ago

It's a monorepo, so this package.json is shared between all the services i have on the project

package.json

guypyOP•2y ago

Not much traffic at all, we're still on a closed beta around the time the crashes started I see there was no activity at all

Brody•2y ago

could you send the nx.json?

guypyOP•2y ago

{
  "npmScope": "cubic",
  "affected": {
    "defaultBase": "main"
  },
  "tasksRunnerOptions": {
    "default": {
      "runner": "nx/tasks-runners/default",
      "options": {
        "cacheableOperations": [
          "build",
          "lint",
          "test",
          "e2e"
        ]
      }
    }
  },
  "generators": {
    "@nrwl/react": {
      "application": {
        "babel": true
      }
    },
    "@nrwl/next": {
      "application": {
        "style": "@emotion/styled",
        "linter": "eslint"
      }
    }
  },
  "defaultProject": "client-app",
  "$schema": "./node_modules/nx/schemas/nx-schema.json",
  "targetDefaults": {
    "build": {
      "dependsOn": [
        "^build"
      ],
      "inputs": [
        "production",
        "^production"
      ]
    },
    "e2e": {
      "inputs": [
        "default",
        "^production"
      ]
    },
    "test": {
      "inputs": [
        "default",
        "^production",
        "{workspaceRoot}/jest.preset.js"
      ]
    },
    "lint": {
      "inputs": [
        "default",
        "{workspaceRoot}/.eslintrc.json"
      ]
    }
  },
  "namedInputs": {
    "default": [
      "{projectRoot}/**/*",
      "sharedGlobals"
    ],
    "sharedGlobals": [
      "{workspaceRoot}/babel.config.json"
    ],
    "production": [
      "default",
      "!{projectRoot}/**/?(*.)+(spec|test).[jt]s?(x)?(.snap)",
      "!{projectRoot}/tsconfig.spec.json",
      "!{projectRoot}/jest.config.[jt]s",
      "!{projectRoot}/.eslintrc.json"
    ]
  }
}

{
  "npmScope": "cubic",
  "affected": {
    "defaultBase": "main"
  },
  "tasksRunnerOptions": {
    "default": {
      "runner": "nx/tasks-runners/default",
      "options": {
        "cacheableOperations": [
          "build",
          "lint",
          "test",
          "e2e"
        ]
      }
    }
  },
  "generators": {
    "@nrwl/react": {
      "application": {
        "babel": true
      }
    },
    "@nrwl/next": {
      "application": {
        "style": "@emotion/styled",
        "linter": "eslint"
      }
    }
  },
  "defaultProject": "client-app",
  "$schema": "./node_modules/nx/schemas/nx-schema.json",
  "targetDefaults": {
    "build": {
      "dependsOn": [
        "^build"
      ],
      "inputs": [
        "production",
        "^production"
      ]
    },
    "e2e": {
      "inputs": [
        "default",
        "^production"
      ]
    },
    "test": {
      "inputs": [
        "default",
        "^production",
        "{workspaceRoot}/jest.preset.js"
      ]
    },
    "lint": {
      "inputs": [
        "default",
        "{workspaceRoot}/.eslintrc.json"
      ]
    }
  },
  "namedInputs": {
    "default": [
      "{projectRoot}/**/*",
      "sharedGlobals"
    ],
    "sharedGlobals": [
      "{workspaceRoot}/babel.config.json"
    ],
    "production": [
      "default",
      "!{projectRoot}/**/?(*.)+(spec|test).[jt]s?(x)?(.snap)",
      "!{projectRoot}/tsconfig.spec.json",
      "!{projectRoot}/jest.config.[jt]s",
      "!{projectRoot}/.eslintrc.json"
    ]
  }
}

Brody•2y ago

is the front-end a csr app?

guypyOP•2y ago

Nope, SSR

Brody•2y ago

could you send me whatever config file is used to setup nx serve?

guypyOP•2y ago

This is for one application, the others are pretty much the same

project.json

Brody•2y ago

it seems you have your app serving in dev mode https://nx.dev/packages/next#serving-next.js-applications

guypyOP•2y ago

I'm sure that the apps are running in production mode, in the screenshots you can see that on the development env the buildId is development and on production there's some random hash

guypyOP•2y ago

guypyOP•2y ago

Can you share maybe what is the meaning of a service getting SIGTERM on Railway? That might help me find the root cause here in case its applicative

Brody•2y ago

the app might be getting build as prod, but the server isn't running as prod, please check the link I sent and apply that flag you also have Doppler set to dev, that's unwanted too

guypyOP•2y ago

OK, I'll make sure that the apps are running in full prod mode. Thanks! What would cause a service to get a SIGTERM though?

Brody•2y ago

that sigterm likely isn't coming from railways container manager so my running guess is some unexpected side effect of running your server in dev mode

guypyOP•2y ago

Ok, I'll do some testing and see how it goes.. Hopefully you're right 🤞 Thanks for the help!

Brody•2y ago

happy to help, I hope you get it sorted also, generally a bonus side effect of not running a dev server is less ram usage

Gaming

Programming

NextJS Application Random Crash