Ian Woodfill
Ian Woodfill
RRailway
Created by Ian Woodfill on 9/20/2023 in #✋|help
Builds failing
Hello, our build are failing with: ERROR: failed to solve: process "/bin/bash -ol pipefail -c apt-get update && apt-get install -y --no-install-recommends libnss3 libatk1.0-0 libatk-bridge2.0-0 libcups2 libgbm1 libasound2 libpangocairo-1.0-0 libxss1 libgtk-3-0 libxshmfence1 libglu1 build-essential libcairo2-dev libpango1.0-dev libpangocairo-1.0-0 libjpeg-dev libgif-dev librsvg2-dev libuuid1" did not complete successfully: exit code: 100 Haven't changed nix config or anything and it repros continuously.
35 replies
RRailway
Created by Ian Woodfill on 9/17/2023 in #✋|help
RTT to nearby datacenter has gone way up in the past few days
Regions have been amazing and are super helpful for our use case. We host our database in AWS us-east-1, which is in the Washington DC area. We've moved our Railway servers to GCP us-east-4 as that is in the same local area. I was experimenting with this last week and was getting single digit to low double digit millisecond RTT/latency from Railway to AWS. This weekend I redeployed the same service and found that the latency has gone back up to >70 ms round-trip time. I'm scratching my head here, because when I can double check the IP of my Railway service and AWS database and see they are both in the Washington DC area, yet this latency is really bad for that proximity. My only thought is perhaps in your development work you all have added some configuration that adds some sort of an additional layer to this networking that adds additional latency. What could be going on here?
20 replies
RRailway
Created by Ian Woodfill on 9/10/2023 in #✋|help
Cannot have a TCP proxy and a domain
When trying to create a service with a proxy, I'm getting this error. Can I not have a TCP proxy service in the same environment as a service with a domain?
41 replies
RRailway
Created by Ian Woodfill on 9/8/2023 in #✋|help
Pin memory usage
I'm running a service that scales memory usage unpredictably and quickly, and it crashes in these cases. I believe strongly that this is due to Railway's autoscaling not provisioning enough memory quick enough (this is obviously hard when you do it all very suddenly). I would be happy to pay for the provisioned resources if it were possible, although I worry this isn't available. Any workarounds?
17 replies
RRailway
Created by Ian Woodfill on 9/4/2023 in #✋|help
Deploy restarted near the end
Just deployed a change (573e8198-6981-45c6-81ae-91d5210afc0c) and it got through the build, then started pushing the container layers, then all of a sudden it started again from the beginning. This has resulted in a 20 minute + build, when its normally around 10 minutes. Seems like a bug.
8 replies
RRailway
Created by Ian Woodfill on 6/19/2023 in #✋|help
Deployment DNS switchover behavior
We're running into issues setting up continuous deployment with our service. We have a webservice that has long running requests (take up to a couple minutes). We set up RAILWAY_DEPLOYMENT_OVERLAP_SECONDS to make sure the request finishes processing before the old server is removed. The behavior we're noticing, however, is that during the OVERLAP period, requests are still being routed to the old deployment, rather than beginning to be sent to the new server. Then when the server gets taken down, it cancels all the requests that are ongoing. Is this expected behavior? Our expectation is that when the RAILWAY_DEPLOYMENT_OVERLAP_SECONDS period begins, Railway would start routing new requests to the new deployment, and let the old deployment finish any outstanding requests in the system, but this seems to not be the case.
7 replies
RRailway
Created by Ian Woodfill on 6/5/2023 in #✋|help
Node graceful shutdowns
I have a node task manager process deployed to Railway that I'm trying to get to gracefully shutdown (finish current tasks before killing the service). Is this possible on railway? I've added logic to catch the SIGTERM and begin shutdown and this works in my local environment when I give it a SIGTERM, but in Railway it exits immediately with the following:
npm ERR! path /app
npm ERR! command failed
npm ERR! signal SIGTERM
npm ERR! command sh -c -- SKIP_ENV_VALIDATION=true node --es-module-specifier-resolution=node .wss-dist/src/server/scheduler/scheduler-entrypoint.js | npm run log-agent -- --service=scheduler | pino-pretty
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2023-06-05T15_35_34_524Z-debug-0.log
npm ERR! path /app
npm ERR! command failed
npm ERR! signal SIGTERM
npm ERR! command sh -c -- SKIP_ENV_VALIDATION=true node --es-module-specifier-resolution=node .wss-dist/src/server/scheduler/scheduler-entrypoint.js | npm run log-agent -- --service=scheduler | pino-pretty
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2023-06-05T15_35_34_524Z-debug-0.log
It doesn't even seem to hit my SIGTERM handler, defined below. Can't find any docs about this kind of thing, is this kind of delayed shutdown supported?
const gracefulShutdown = (code: string) => {
logger.info(`Worker received ${code}. Shutting down...`)
multiWorker.end().then(() => {
process.exit(0)
})
}

const signals: NodeJS.Signals[] = ["SIGTERM", "SIGINT", "SIGUSR2"]

signals.forEach((signal) => {
process.on(signal, () => gracefulShutdown(signal))
})
const gracefulShutdown = (code: string) => {
logger.info(`Worker received ${code}. Shutting down...`)
multiWorker.end().then(() => {
process.exit(0)
})
}

const signals: NodeJS.Signals[] = ["SIGTERM", "SIGINT", "SIGUSR2"]

signals.forEach((signal) => {
process.on(signal, () => gracefulShutdown(signal))
})
47 replies
RRailway
Created by Ian Woodfill on 3/24/2023 in #✋|help
Rollback causes ${{Redis.REDIS_URL}} variables to be missing from config
Love the rollback feature, its a lifesaver. although it seems to rollback my instance to a state where the redis url from the environment is missing, so my app breaks. When I rollback via deploy it seems to work just fine. Seems like a bug?
5 replies