Railway•11mo ago

Cronjob code executing every 13 seconds instead of on the schedule 0 13 * * *

The process is the one that should send one daily notification to all the users. Even after I have deleted the ENTIRE cronjob service, there is some kind of zombie process that's in an infinite loop (since around last night / this morning). So its sending notifications to everyone, every 13 seconds. Project ID: 2bbc885c-7904-4d61-a9a6-3d1a117d28d2

37 Replies

Percy•11mo ago

Project ID: 2bbc885c-7904-4d61-a9a6-3d1a117d28d2

psgeorgeOP•11mo ago

Okay it might have died down a few minutes after deleting the entire service 🤞 But I will need to recreate the service now & I'm hoping the issue doesn't reoccur. Could you look into it and find a probable cause please? More info: I deployed new code (with different env variables), but in the logs the code that was running every few seconds had the old env variables. So the process somehow got detatched and stuck in a loop.

Brody•11mo ago

railway's cron scheduler has a minimum limit of 15 minutes last I checked, your code running in a loop seems more like a code issue?

psgeorgeOP•11mo ago

No, it wasn't doing this before today and there are no loops in the code It's very simple code

const API_KEY = process.env.API_KEY;
const API_URL = process.env.API_URL;
const DAILY_SCHEDULE = process.env.DAILY_SCHEDULE;

var fetch = (...args) =>
  import("node-fetch").then(({ default: fetch }) => fetch(...args));

console.log(
  "cron: Starting cron worker, API_URL:",
  API_URL,
  "Schedule:",
  DAILY_SCHEDULE,
);

console.log("cron: Running daily cronjob...");
const url = `${API_URL}/v1/notifications/daily`;
console.log("cron: pinging url", url);
const body = JSON.stringify({
  no_send: false
});
fetch(url, {
  body,
  headers: {
    "Content-Type": "application/json",
    Accept: "application/json",
    "x-api-key": API_KEY,
  },
  method: "POST",
})
  .then((response) => {
    if (!response.ok) {
      console.log("cron: error", response.status);
    }
    return response.text();
  })
  .then((data) => {
    console.log("cron: response data:", data);
  })
  .catch((error) => {
    console.log("cron: error parsing response", error);
  });

const API_KEY = process.env.API_KEY;
const API_URL = process.env.API_URL;
const DAILY_SCHEDULE = process.env.DAILY_SCHEDULE;

var fetch = (...args) =>
  import("node-fetch").then(({ default: fetch }) => fetch(...args));

console.log(
  "cron: Starting cron worker, API_URL:",
  API_URL,
  "Schedule:",
  DAILY_SCHEDULE,
);

console.log("cron: Running daily cronjob...");
const url = `${API_URL}/v1/notifications/daily`;
console.log("cron: pinging url", url);
const body = JSON.stringify({
  no_send: false
});
fetch(url, {
  body,
  headers: {
    "Content-Type": "application/json",
    Accept: "application/json",
    "x-api-key": API_KEY,
  },
  method: "POST",
})
  .then((response) => {
    if (!response.ok) {
      console.log("cron: error", response.status);
    }
    return response.text();
  })
  .then((data) => {
    console.log("cron: response data:", data);
  })
  .catch((error) => {
    console.log("cron: error parsing response", error);
  });

Brody•11mo ago

was railway's UI sending out a deployment every 13 seconds?

psgeorgeOP•11mo ago

Literally just does one request No; the deployment was invisible, but I could see the logs coming every 13 seconds

Brody•11mo ago

where could you see the logs?

psgeorgeOP•11mo ago

observability section

Brody•11mo ago

could you send a screenshot of that please?

psgeorgeOP•11mo ago

psgeorgeOP•10mo ago

Started today at 9:00am on the dot almost (UK time) It wasn't just logs, I was also getting a notification every 13 seconds 🥲 Hi, I haven't received a satisfactory response to this? Please advise Similar things have been happening with my database backups

psgeorgeOP•10mo ago

From July until November had no issues, it ran at 5am every day:

psgeorgeOP•10mo ago

And then it's getting worse and worse as time goes on:

psgeorgeOP•10mo ago

If this isn't resolved I'm going to have to exit because I can't have a cron scheduler that's running amock from time to time - it needs to be reliable

psgeorgeOP•10mo ago

Jan 4th was particularly bad:

psgeorgeOP•10mo ago

@Brody ☝️ I had to disable my notifications cron completely until I move it to a separate provider. DB backups are less harmful but it could still DDOS my database / cost me a ton of money if I had more users / data e.g. a 1GB backup that decides to run 10x a minute between 2am and 4am as above is not ok

Brody•10mo ago

this is very odd, no one else has yet to report anything even remotely similar can you share your repo and the exact cron expression you are using?

psgeorgeOP•10mo ago

Sure I can add you to the repo if you give me your GH name, and then link you to code I'm using It's all as basic as could be

cronSchedule="0 5 * * *"
startCommand = "node dist/index.js"
replicas = 1

cronSchedule="0 5 * * *"
startCommand = "node dist/index.js"
replicas = 1

import { PutObjectCommand, S3Client, S3ClientConfig } from "@aws-sdk/client-s3";
import { createReadStream, unlink } from "fs";

import { env } from "./env";

const uploadToS3 = async ({ name, path }: {name: string, path: string}) => {
  console.log("Uploading backup to S3...");

  const bucket = env.AWS_S3_BUCKET;

  const clientOptions: S3ClientConfig = {
    region: env.AWS_S3_REGION,
  }

  if (env.AWS_S3_ENDPOINT) {
    console.log(`Using custom endpoint: ${env.AWS_S3_ENDPOINT}`)
    clientOptions['endpoint'] = env.AWS_S3_ENDPOINT;
  }

  const client = new S3Client(clientOptions);

  await client.send(
    new PutObjectCommand({
      Bucket: bucket,
      Key: name,
      Body: createReadStream(path),
    })
  )

  console.log("Backup uploaded to S3...");
}

const dumpToFile = async (path: string) => {
  console.log("Dumping DB to file...");

  await new Promise((resolve, reject) => {
    exec(
      `pg_dump ${env.BACKUP_DATABASE_URL} -Fc > ${path}`,
      (error, stdout, stderr) => {
        if (error) {
          reject({ error: JSON.stringify(error), stderr });
          return;
        }

        resolve(undefined);
      }
    );
  });

  console.log("DB dumped to file...");
}

const deleteFile = async (path: string) => {
  console.log("Deleting file...");
  await new Promise((resolve, reject) => {
    unlink(path, (err) => {
      reject({ error: JSON.stringify(err) });
      return;
    });
    resolve(undefined);
  })
}

export const backup = async () => {
  console.log("Initiating DB backup...")

  const date = new Date().toISOString()
  const timestamp = date.replace(/[:.]+/g, '-')
  const filename = `backup-${timestamp}.dump`
  const filepath = `/tmp/${filename}`

  await dumpToFile(filepath)
  await uploadToS3({name: filename, path: filepath})
  await deleteFile(filepath)

  console.log("DB backup complete...")
}

import { PutObjectCommand, S3Client, S3ClientConfig } from "@aws-sdk/client-s3";
import { createReadStream, unlink } from "fs";

import { env } from "./env";

const uploadToS3 = async ({ name, path }: {name: string, path: string}) => {
  console.log("Uploading backup to S3...");

  const bucket = env.AWS_S3_BUCKET;

  const clientOptions: S3ClientConfig = {
    region: env.AWS_S3_REGION,
  }

  if (env.AWS_S3_ENDPOINT) {
    console.log(`Using custom endpoint: ${env.AWS_S3_ENDPOINT}`)
    clientOptions['endpoint'] = env.AWS_S3_ENDPOINT;
  }

  const client = new S3Client(clientOptions);

  await client.send(
    new PutObjectCommand({
      Bucket: bucket,
      Key: name,
      Body: createReadStream(path),
    })
  )

  console.log("Backup uploaded to S3...");
}

const dumpToFile = async (path: string) => {
  console.log("Dumping DB to file...");

  await new Promise((resolve, reject) => {
    exec(
      `pg_dump ${env.BACKUP_DATABASE_URL} -Fc > ${path}`,
      (error, stdout, stderr) => {
        if (error) {
          reject({ error: JSON.stringify(error), stderr });
          return;
        }

        resolve(undefined);
      }
    );
  });

  console.log("DB dumped to file...");
}

const deleteFile = async (path: string) => {
  console.log("Deleting file...");
  await new Promise((resolve, reject) => {
    unlink(path, (err) => {
      reject({ error: JSON.stringify(err) });
      return;
    });
    resolve(undefined);
  })
}

export const backup = async () => {
  console.log("Initiating DB backup...")

  const date = new Date().toISOString()
  const timestamp = date.replace(/[:.]+/g, '-')
  const filename = `backup-${timestamp}.dump`
  const filepath = `/tmp/${filename}`

  await dumpToFile(filepath)
  await uploadToS3({name: filename, path: filepath})
  await deleteFile(filepath)

  console.log("DB backup complete...")
}

import { backup } from "./backup";

const tryBackup = async () => {
  try {
    await backup();
  } catch (error) {
    console.error("Error while running backup: ", error)
  }
}

console.log("Running backup cron ...")
tryBackup();

import { backup } from "./backup";

const tryBackup = async () => {
  try {
    await backup();
  } catch (error) {
    console.error("Error while running backup: ", error)
  }
}

console.log("Running backup cron ...")
tryBackup();

🤔 I'm pretty sure its not an issue on my end, because this was literally working for months, before it started breaking (with me having made no changes). I can't tell you how much I would love to not have to move off railway, its such a great product aside this phantom behaviour!

Brody•10mo ago

it's been a while since you opened this thread and still no one has reported anything even remotely similar, so please add me to your repo brody192

psgeorgeOP•10mo ago

https://github.com/ps-george/hayfriends/tree/main/apps/cron Invited, link to the folder that has the cronjobs in it

psgeorgeOP•10mo ago

If you have access to my logs, you can see it plainly - the cronjob some days ends up triggering randomly and repeatedly, other days it acts normally

psgeorgeOP•10mo ago

https://railway.app/project/2bbc885c-7904-4d61-a9a6-3d1a117d28d2/logs?start=1707312600000&end=1707367725000

Brody•10mo ago

i dont work for railway so i dont have acess to your project

psgeorgeOP•10mo ago

Wasn't sure even railway would tbh

Brody•10mo ago

the team can see your logs yes

psgeorgeOP•10mo ago

In any case, in the past week every day is fine and then suddenly on the 7th Feb it goes haywire.. Looking at my repo I haven't even pushed unrelated code since 3rd Feb. And as you say, I reported this a while ago. This has been ongoing since around November

Brody•10mo ago

why not use railway's backup template?

psgeorgeOP•10mo ago

kind of irrelevent, I had to disable my daily notifications cronjob because overdoing backups is fine, overdoing sending users 100 notifications a day is not fine But when I set this up it was the recommended way of doing things

Brody•10mo ago

can you try with the restart policy set to never

psgeorgeOP•10mo ago

has been that all along

Brody•10mo ago

im not seeing anything that stands out to me, ill flag the team

psgeorgeOP•10mo ago

Thanks Brody

Duchess•10mo ago

Thread has been flagged to Railway team by @Brody.

Linear•10mo ago

Issue PRO-1865 created.

PRO-1865 - Cron Job running multiple times in quick succession

Cron job is scheduled to run 0 5 * * *. You can see in the screenshot in the thread that it ran multiple times back to back on Feb 8 (4:58 and then again at 5:00). This is one occurrence, but has happened several times in the past. See thread for more context going back to January.

Status

Triage

Product

Melissa•10mo ago

hi psgeorge, apologies for this experience. I have created a ticket for us to look into the issue

psgeorgeOP•10mo ago

Thanks @Melissa

Gaming

Programming

Cronjob code executing every 13 seconds instead of on the schedule 0 13 * * *