Wasp•4w ago

Presistent job in background

Is it a bad habbit to have a presistent job running in the background that gets some information from a third party endpoint for a user. This job would retrieve any new orders for each user and updates it in the dashboard such that it is up to date and realtime. If it is not a bad practise how would i best do this, also how can i see the impact of this on the server.

5 Replies

kapa.ai•4w ago

Hi @Sven. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave: ---------------------- Having a persistent job running in the background to fetch and update information is not inherently a bad practice, but there are some considerations to keep in mind when implementing such a solution in Wasp. Wasp supports background jobs, which can be useful for tasks like fetching data from third-party endpoints. However, the current implementation has some limitations: 1. Wasp uses pg-boss as its job executor, which runs alongside your server-side code. This means that CPU-intensive tasks should be avoided in these jobs Docs: Recurring Jobs (https://wasp-lang.dev/docs/advanced/jobs#pg-boss-considerations). 2. The jobs are not truly real-time. They run on a schedule or can be triggered manually, but they don't provide instant updates Docs: Recurring Jobs (https://wasp-lang.dev/docs/advanced/jobs#recurring-jobs). To implement a job that fetches new orders for users and updates the dashboard, you could create a recurring job in Wasp:

job fetchNewOrders {
  executor: PgBoss,
  perform: {
    fn: import { fetchOrders } from "@src/workers/orderFetcher"
  },
  schedule: {
    cron: "*/5 * * * *" // Runs every 5 minutes
  }
}

job fetchNewOrders {
  executor: PgBoss,
  perform: {
    fn: import { fetchOrders } from "@src/workers/orderFetcher"
  },
  schedule: {
    cron: "*/5 * * * *" // Runs every 5 minutes
  }
}

This job would run every 5 minutes to fetch new orders. However, this approach has limitations: 1. It's not real-time, as there's a delay between updates. 2. It might not be efficient if you have many users, as it would fetch data for all users every time it runs. For a more real-time approach, Wasp is considering adding support for real-time data updates in the future. This feature would use WebSockets to push updates to clients when data changes Github Issue: Real-time data updates (https://github.com/wasp-lang/wasp/issues/1454). Regarding the impact on the server, Wasp doesn't currently provide built-in tools for monitoring job performance. You would need to implement your own monitoring solution or use external tools to track the resource usage of your Wasp application. In conclusion, while it's possible to implement a background job for fetching and updating data in Wasp, the current implementation might not be ideal for truly real-time, high-frequency updates across many users. Consider the trade-offs and limitations when deciding if this approach is suitable for your specific use case. ---------------------- Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: ---------------------- Sorry @Sven, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist: Perhaps try again in 15mins?

SvenOP•4w ago

@kapa.ai for point "It might not be efficient if you have many users, as it would fetch data for all users every time it runs." what could i do to avoid this double fetching?

kapa.ai•4w ago

Let me check the knowledge base for you again. One moment! :books: ---------------------- To address the issue of potential inefficiency when fetching data for all users every time the job runs, you can implement a few strategies to optimize the process: 1. Batch processing: Instead of fetching data for all users in a single job run, you can implement a batching mechanism. This approach processes users in smaller groups, spreading the workload across multiple job executions. You can track which users have been processed and continue with the next batch in subsequent runs. 2. Incremental updates: Keep track of the last time data was fetched for each user. When the job runs, only fetch new data for users who haven't been updated recently or have pending changes. This can be implemented by storing a timestamp for each user's last update and using it as a filter when querying the third-party endpoint. 3. Webhooks: If the third-party service supports webhooks, you can set up a webhook endpoint in your Wasp application. This way, the third-party service can notify your application when there are new orders, allowing you to update only the relevant users' data. Wasp supports creating custom API endpoints that you can use for this purpose Wasp Docs: APIs (https://wasp-lang.dev/docs/advanced/apis). 4. Queue-based processing: Instead of fetching data for all users in a single job, you can implement a queue-based system. The job would add users who need updates to a queue, and a separate process would consume this queue, updating users one at a time or in small batches. Here's a simple example of how you might implement incremental updates in your Wasp job:

job fetchNewOrders {
  executor: PgBoss,
  perform: {
    fn: import { fetchOrdersIncrementally } from "@src/workers/orderFetcher"
  },
  schedule: {
    cron: "*/5 * * * *" // Runs every 5 minutes
  }
}

job fetchNewOrders {
  executor: PgBoss,
  perform: {
    fn: import { fetchOrdersIncrementally } from "@src/workers/orderFetcher"
  },
  schedule: {
    cron: "*/5 * * * *" // Runs every 5 minutes
  }
}

And in your orderFetcher.js:

export const fetchOrdersIncrementally = async () => {
  const users = await prisma.user.findMany({
    where: {
      lastOrderFetch: {
        lt: new Date(Date.now() - 5 * 60 * 1000) // Users not updated in the last 5 minutes
      }
    },
    take: 100 // Process 100 users at a time
  });

  for (const user of users) {
    const newOrders = await fetchNewOrdersForUser(user);
    await updateUserOrders(user, newOrders);
    await prisma.user.update({
      where: { id: user.id },
      data: { lastOrderFetch: new Date() }
    });
  }
};

export const fetchOrdersIncrementally = async () => {
  const users = await prisma.user.findMany({
    where: {
      lastOrderFetch: {
        lt: new Date(Date.now() - 5 * 60 * 1000) // Users not updated in the last 5 minutes
      }
    },
    take: 100 // Process 100 users at a time
  });

  for (const user of users) {
    const newOrders = await fetchNewOrdersForUser(user);
    await updateUserOrders(user, newOrders);
    await prisma.user.update({
      where: { id: user.id },
      data: { lastOrderFetch: new Date() }
    });
  }
};

This approach ensures that you're only fetching data for users who haven't been updated recently, and it processes users in batches to avoid overloading the system. Remember, the exact implementation will depend on your specific use case and the structure of your data model. You may need to adjust the approach based on your application's requirements and the capabilities of the third-party service you're integrating with. Github Issue: Background jobs (https://github.com/wasp-lang/wasp/issues/322) ---------------------- Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: ---------------------- Sorry @Sven, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist: Perhaps try again in 15mins?

Filip•4w ago

@Sven I see you're discussing it with the bot. Hope it's useful. Tag me and pls summarise if you need me to jump in at one point 🙂

SvenOP•4w ago

yessirr thanks

Gaming

Programming

Presistent job in background