6 Replies
Just one queue!
but you can add multiple queues to increase concurrency ... that's what we've done ...
Multiple queue for the same task? Is there not a better way than that?
it's not ideal, but it's vastly better now than it used to be before they raised all of the limits recently (it previously took 10 queues to get the currently levels) ... you could shard on say
queue1, queue2, etc and the same worker can be consumer to all 10 or whatever number of those queuesHow do you manage know whether a queue is "at capacity" or do you just randomly distribute?
assume this will scale 250 consumer workers per queue? so 10 queues = up to 2500 workers
Did you also try using a cloudflare worker as a pull consumer instead of the default push consumer approach?
for pull consumers to work you'd have to manage your own invocations/scaling, as well as you might hit the Cloudflare API limit ... as for capacity, we just randomized the queues we were sending to, and that has worked fine for us ...