Railway•3mo ago

Cron job deployment logs not showing up

Project id: 8eeaf105-a438-44c0-9215-78f268bcf4a9. I am trunning to run a cron job that has the configuration set in a railway.toml file I have configured it so that it run from a docker file. If I build the image and run it locally everything works as expected. Yet, when I deploy it on Railway, the cron services shows that it is about to run in x minutes but, aside from successfully building the Docker image, there are no deploy logs. How I translate that is that the container is not run in the first place.

42 Replies

Percy•3mo ago

Project ID: 8eeaf105-a438-44c0-9215-78f268bcf4a9

Raz | OogaOP•3mo ago

Hello @Brody ! Do you reckon there might be something that I might've misconfigured? If it's the case for a bug is there a workaround for it?

Brody•3mo ago

please do not tag team or mods - #🛂｜readme #5 anyway, what makes you say the container isn't being ran?

Raz | OogaOP•3mo ago

Thank you for your response. The fact that it is not displaying anything in the Deploy logs section. yet, it should since it thoretcally "has been run"

SCIENTIFIC•3mo ago

+1 – I am seeing the same. I waited up to an hour in the event there was some queue delay. I see several posts that seem to suggest crons have stopped working or there is a behavior change: https://help.railway.app/feedback/cron-reliability-improvements-c2f2c959

Railway Help Station

Cron Reliability Improvements

Hello,We've been barreling towards reliability improvements recently. In fact, it's our focus for the next 3 months.Crons, historically have been "Alright but not bulletproof". Sometimes they'd skip, sometimes they'd get stuck, etc. Not a great experience at all.We've rolled out a couple changes this week that should abate thisPlease let us know...

Brody•3mo ago

im not seeing any cron services in any of your environments?

Raz | OogaOP•3mo ago

Um, I think there should be. development-testnet environment Cron Jobs group. I think service id of one of the cron jobs is: 17c81b48-6409-4b66-ba85-5537f35e62aa

Raz | OogaOP•3mo ago

Raz | OogaOP•3mo ago

builder = "DOCKERFILE"
dockerfilePath = "./cron/<hidden-path>/Dockerfile"
watchPatterns = ["cron/**", "packages/**", "!/*.md", "package.json"]

[deploy]
cronSchedule = "0 0 * * *"
region = "europe-west4"

builder = "DOCKERFILE"
dockerfilePath = "./cron/<hidden-path>/Dockerfile"
watchPatterns = ["cron/**", "packages/**", "!/*.md", "package.json"]

[deploy]
cronSchedule = "0 0 * * *"
region = "europe-west4"

Brody•3mo ago

I must be blind then, my bad, I'll give another look

Raz | OogaOP•3mo ago

Please, it would be really helpful. Maybe there is something misconfigured on our end

Brody•3mo ago

I'll look when I'm back at my computer :salute: but in the mean time, please tell me more about this cron job. language? what does it achieve? are you using a logger library? etc update, found the service, something really strange is going on here for sure

Raz | OogaOP•3mo ago

Great news. The service is trying to run a Dockerfile that runs a binary file created after bun env build action. Other than that it is pretty standard. Locally (on my machine Macbook M1 series) everything works fine. For some reason on the Railway env nothing works Language is Typescript Run time environment is bun.js logger is used everywhere in our project so I doubt it's the case, we use pino The railway.toml is provided above. Here is the Dockerfile:

# oven/bun images dont work, have to have node underneath to build
FROM node:20.12.2 AS build

RUN npm install -g [email protected]
WORKDIR /app
COPY . .
RUN bun install --frozen-lockfile --prefer-offline

ARG TURBO_TEAM
ENV TURBO_TEAM=$TURBO_TEAM
ARG TURBO_TOKEN
ENV TURBO_TOKEN=$TURBO_TOKEN
RUN bun turbo build --filter secret-cron-name

FROM gcr.io/distroless/base-debian11
WORKDIR /app
COPY --from=build /app/cron/secret-cron-name/bin /app/bin

CMD [ "/app/bin" ]

# oven/bun images dont work, have to have node underneath to build
FROM node:20.12.2 AS build

RUN npm install -g [email protected]
WORKDIR /app
COPY . .
RUN bun install --frozen-lockfile --prefer-offline

ARG TURBO_TEAM
ENV TURBO_TEAM=$TURBO_TEAM
ARG TURBO_TOKEN
ENV TURBO_TOKEN=$TURBO_TOKEN
RUN bun turbo build --filter secret-cron-name

FROM gcr.io/distroless/base-debian11
WORKDIR /app
COPY --from=build /app/cron/secret-cron-name/bin /app/bin

CMD [ "/app/bin" ]

Brody•3mo ago

what are your logging settings? are you logging unbuffered to stdout / stderr?

Raz | OogaOP•3mo ago

Yes. It worked for a couple of days, now they are not being run again. Seems like the cron jobs are very unreliable

Brody•3mo ago

are you sure your application is exiting after it's done it's task?

Raz | OogaOP•3mo ago

Yes, it even has a process.exit(0) at the end when testing locally, it always ends

Brody•3mo ago

how long does it take?

Raz | OogaOP•3mo ago

1 should take under 30 seconds. The other should probably take 5 minutes initially. Aftewards it will take under 30 seconds as well

Brody•3mo ago

what is your cron schedule

Raz | OogaOP•3mo ago

0 * * * * For example, this is the cron service that does not run: b734ede7-3e09-4b94-8699-28d74a046e30

Brody•3mo ago

okay so every hour right?

Raz | OogaOP•3mo ago

yes tried running it manually and it still does not log anything

Brody•3mo ago

what logging library are you using?

Raz | OogaOP•3mo ago

pino - https://www.npmjs.com/package/pino

npm

pino

super fast, all natural json logger. Latest version: 9.4.0, last published: 13 days ago. Start using pino in your project by running npm i pino. There are 5247 other projects in the npm registry using pino.

Brody•3mo ago

are you the same person who reported that railway was overwriting the log level?

Raz | OogaOP•3mo ago

No, this is the only thread I have opened

Brody•3mo ago

okay, thank you. do you have a custom transport?

Raz | OogaOP•3mo ago

No, standard stdout/stderr

Brody•3mo ago

I'm fairly confident Pino is buffering logs, if Pino is buffering logs railway can't pick the logs up to send to the frontend

Raz | OogaOP•3mo ago

We are using pino for all of the services. They all log perfectly fine. The only service instances that don't print out the logs are the crons. Moreover, some of them should surely not fail, but I can't see their effects. This leads me to thinking that the cron jobs are not run in the first place

Brody•3mo ago

does railway's UI show you that your job was ran? or does it say skipped?

Raz | OogaOP•3mo ago

Brody•3mo ago

can you enable the beta cron UI in the service settings

Raz | OogaOP•3mo ago

Yes, I will attach a video in a bit

Raz | OogaOP•3mo ago

Actually, it would take too much to record a video. I will summarize it. Pressing on run now, redeploys the cron job Nothing else shows up. I will send you a screenshot in 3 minutes when the previous deployment shows that's supposed to run