Cloudflare Developers•7mo ago

Need help in setting up queue in workers

hey I am creating a simple worker to get html code of websites, I want to create a worker which gets urls from local json file, and send them in a queue this queue then invoke other worker which will do the rest, I read documentation but found that i have to do it in a single worker fucntion, I can undersatnd anything there, please help me how to do this?

1 Reply

DaniFoldi•7mo ago

Hi 👋 You'll need to add a queue producer and consumer binding to your wrangler.toml (with a queue already created):

[[queues.consumers]]
  queue = "<queue name here>"
  max_batch_timeout = 60

[[queues.producers]]
 queue = "<queue name here>"
 binding = "MY_QUEUE"

[[queues.consumers]]
  queue = "<queue name here>"
  max_batch_timeout = 60

[[queues.producers]]
 queue = "<queue name here>"
 binding = "MY_QUEUE"

You can add these two into different workers, you could even have multiple producers - just one consumer. You can then push a new message with env.MY_QUEUE.send({'your data': 'goes here'}) Which will call the queue handler of your worker. Here's a small example:

export interface Env {
  MY: Queue;
}

export default {
  async fetch(req, env): Promise<Response> {
    await env.MY_QUEUE.send({ url: await req.url });
    return new Response("Success!");
  },
  async queue(batch, env): Promise<void> {
    batch.ackAll();
  }
} satisfies ExportedHandler<Env>;

export interface Env {
  MY: Queue;
}

export default {
  async fetch(req, env): Promise<Response> {
    await env.MY_QUEUE.send({ url: await req.url });
    return new Response("Success!");
  },
  async queue(batch, env): Promise<void> {
    batch.ackAll();
  }
} satisfies ExportedHandler<Env>;

Batches of messages will be delivered to the queue() handler here, and you can acknowledge/retry them one by one or all at once. Here's a tutorial for a small crawler that's hopefully going to help you: https://developers.cloudflare.com/queues/tutorials/web-crawler-with-browser-rendering/#1-build-the-crawler-worker

Cloudflare Docs

Cloudflare Queues - Queues & Browser Rendering · Cloudflare Queues

This tutorial explains how to build and deploy a web crawler with Queues, Browser Rendering, and Puppeteer.

Gaming

Programming

Need help in setting up queue in workers