Cloudflare Developers•4mo ago

Is Workers viable for running a function that takes minutes to complete?

I have a JavaScript function with Promise calls. It is responsible for running a Lighthouse-comparable feature called PageSpeed Insights API. It takes minutes before the function is executed because, just like a normal Lighthouse feature in the browser, it calculates metrics like performance and SEO for every website URL in the array I provide to the function. The problem is that it runs in the foreground of my Next.js app, preventing my users from clicking on any link, which would abort the whole process. The solution would be to ensure it runs in the background while users can still use my app. When the function is finished, a certain action should be performed on the Supabase database, and my Next.js app should know one way or another (perhaps using a webhook) that the function is finished. I am wondering if Workers might be a good fit to take that function? I remember that it is based on Edge so perhaps it might be troublesome for a worker to execute a function that takes couple of seconds to complete? I've tried to check the docs about Workers but I did not understand much.

6 Replies

Chaika•4mo ago

hmmm... the pagespeed insights api is just a normal rest api you call and await? Workers have no issues with waiting a long time for a fetch to return, it's not cpu time and duration is unlimited as long as the client stays connected:

Duration is a measurement of wall-clock time — the total amount of time from the start to end of an invocation of a Worker. There is no hard limit on the duration of a Worker. As long as the client that sent the request remains connected, the Worker can continue processing, making subrequests, and setting timeouts on behalf of that request. When the client disconnects, all tasks associated with that client request are canceled. Use event.waitUntil() to delay cancellation for another 30 seconds or until the promise passed to waitUntil() completes.

but if it's a fetch call, why are you having it block your next.js app? Just run it in a background function without holding up anything else and update when it finally returns/finishes

1chbinamin•4mo ago

Sorry, I also forgot to mention that the user can navigate from current page where the pagespeed api is executed to another page. So I suppose that the client (I think by that you meant the page where the worker is called) is not actually connected the moment when the user navigates to another page? And what do you mean with “having it block your next.js app)?

Chaika•4mo ago

when you say pagespeed api, which one do you mean? The REST API documented here: https://developers.google.com/speed/docs/insights/v5/get-started ? You said " The problem is that it runs in the foreground of my Next.js app, preventing my users from clicking on any link, which would abort the whole process", a normal REST API that you call via fetch should be as easy as just not having the result be awaited/block the action and instead doing it in the background

1chbinamin•4mo ago

Yes, it is that link you’ve sent here. I forgot to mention that I did put a code to block the foreground of the whole application while the API is running. I did not want my users to navigate to other pages and abort the API call. Now I want to prevent this by running the API call process somewhere in the background while the user can still use the app. What do you mean by "just not having the result be awaited"? If I understand it correctly, the API call fetch should be awaited, or else I get a Promise object as a result. How else can I get the real result?

Chaika•4mo ago

async function fetchData() {
  try {
    var fetch1 = await fetch("https://example.com/doAction");
...other logic
    var fetch2 = await fetch("https://example.com/postResults");
...
  } catch (error) {
    console.error("Error background fetching:", error);
  }
}

.. otherfunc
...
fetchData();
...
console.log("Continuing with other tasks...");

async function fetchData() {
  try {
    var fetch1 = await fetch("https://example.com/doAction");
...other logic
    var fetch2 = await fetch("https://example.com/postResults");
...
  } catch (error) {
    console.error("Error background fetching:", error);
  }
}

.. otherfunc
...
fetchData();
...
console.log("Continuing with other tasks...");

I see what you're saying though, if it's that important that they can't cancel the result, even if they lose connection/close browser/etc and it's always stored, Workers aren't really a great fit either as the second the client disconnects, the request gets killed, and even items in waitUntil() only get 30s to complete. Could maybe use a Worker pushing into a queue and have the queue handler deal with it, but it might be simpler to just do this on some other backend

1chbinamin•4mo ago

I suppose a cronjob perhaps or anything similar. Thanks btw

Gaming

Programming

Is Workers viable for running a function that takes minutes to complete?