Queue "consumer concurrency" feature
Queue "consumer concurrency" feature does not seem to be suitable for scenarios that rely on sequential execution.
3 Replies
For example,
I have some subtasks that need to be executed in sequence, each task wraped into a queue message and will take a long time to finish:
1,2,3,4,5,6
max_batch_size = 2
max_concurrency = 2
Queue will create two consumer workers to handle those messages concurrently:
1,2 -> worker-1
3,4 -> worker-2
5,6 pending
so 3,4 may be excuted earlier than 1,2:
3,4,1,2,5,6
this breaks the original sequence.
Cloudflare Docs
Consumer concurrency | Cloudflare Queues
Consumer concurrency allows a consumer Worker processing messages from a queue to automatically scale out horizontally to keep up with the rate that messages are being written to a queue.
Queues today doesn't support strict ordering. But also, as Leo mentioned, concurrency doesn't really work with strict ordering anyway 🙂