Cron Missed Completely, No Logs
I worked on a cron application yesterday that fired at 6AM CST correctly on the first night, but did not do it again last night.
31 Replies
Project ID:
3c0425aa-9fe4-449a-a161-8a1efb3b53ee3c0425aa-9fe4-449a-a161-8a1efb3b53ee
The schedule is
0 11 * * *, which means everyday at 11:00 AM UTC.
Logs show the last invocation was on June 1st at 6:01 AM CST (correct, although 1 minute late?).
The expected last invocation would actually be June 2nd at 6:00AM CST, but there is no activity, logs, or even an invocation on the dashboard.
I only found out about this because my cron monitor raised an issue.The history below doesn't show anything interesting, if you're curious.
from my understanding, there are too many jobs being ran at 11am utc that some get skipped, until the team addresses this i would recommend switching to an in-code scheduler
are you serious? i literally just undid the node cron scheduler because it was consuming 50 MB memory constantly and i thought it'd be fun to move away from that
not mad at you, just... that's pretty sucky
i feel you, you could also try another time? 10:30am utc?
Yeah, I was thinking that some weird off-color time would be less likely to incur issues.
Something like XX:48
The whole job is done in 5 seconds usually.
yep you got the right idea
Gonna try 10:48 UTC and see what happens.
sounds good!
i have also sent this thread in a thread i have with cooper for gathering cron issues like yours
Alrighty; just my thought: a little detail about this being skipped, or likely to be skipped, or some transparency on the issues with cron would be nice.
I don't mind that Railway's platform is in need of improvement; but letting users explore until they hit a landmine isn't ideal.
i assume they had never thought they would be over scheduled so they never designed error handing and the ui around it
Kinda an interesting problem to think about in retrospect.
ideally the only issues that you could get out of a cron job would be an issue with the build or deploy
Maybe a check-mark that says "I'm okay with this being rescheduled slightly" would be good.
Crons that are more important could be charged at a higher rate, but they'll be prioritized on runners or whatever.
If you have 600 jobs every day at 11AM UTC, spinning up tons of machines to work on them is not exactly smart. Especially when most of them are tiny jobs.
Working in bursts would be better. And working early + late, like queueing. Start at 10:58 or even earlier to start executing.
im sure they have more than 600 at 11am utc, and if i recall correctly, its only a single schedular on their backplane
aha i really have no idea about the scale railway works at tbh
i dont really either, im just going off the crumbs they give us
i mean they do tell us a fair bit, but more info can never hurt in our position of community help
@Brody
This is pretty sucky as crons go.
I'm not sure what's going on, actually; I cannot tell if Railway is to fault here.
Nevermind, seems like something with Sentry is going wrong?
Error while running backup: AxiosError: connect EHOSTUNREACH 34.120.195.249:443host unreachable eh? you aren't the first person to see this error even after they resolved the incident
are you on the legacy or v2 runtime? check your service settings, if legacy, switch it to v2
Got it, switched it to V2. Didn't know that was a setting lol.
just for clarity, the v2 runtime has been confirmed to fix host unreachable, but it has no impact on cron being skipped since that's a completely different system
π
Working good!
No failures since.
they made changes to the cron scheduler too
the changes they made did not help π¦
but thank you for trying
i mean i guess at least it happened the same day?
lol
very odd
lol
back to in code