Hey all, wondering if theres a way to
Hey all, wondering if theres a way to set a limit on the number of requests to a worker over a given period of time before it starts rejecting new requests? If I can avoid implementing it myself that'd be great
3 Replies
there isn't! But you can use pull consumers if you want to control the rate at which messages are consumed: https://developers.cloudflare.com/queues/configuration/pull-consumers/
Cloudflare Docs
Cloudflare Queues - Pull consumers · Cloudflare Queues
A pull-based consumer allows you to pull from a queue over HTTP from any environment and/or programming language outside of Cloudflare Workers. A pull-based consumer can be useful when your message consumption rate is limited by upstream infrastructure or long-running tasks.
what are you using queues for btw?
I am planning to use a few queues to collect data from an api and save it to my DB. I don't need heaps of data to have a meaningful dataset but there is effectively (for my purposes) infinite data available. The intention is to control how much I ingest by limiting the amount of jobs submitted to the queue. The more important reason is to control how much I pay for these queues while I am in testing.
I'll need to rescrape every few weeks as a new patch comes out. It is a recursive process so maybe I'll just keep a count of ingested data and configure a limit per patch and reject any new jobs if that limit is hit