guys i got over an hour of consumer
guys i got over an hour of consumer delay, is this normal?
6 Replies
depends on the rate of messages coming in. consumer delay = how old is the average message in my queue
So what this means is that your queue & consumer are working on old messages. This can happen if you write a burst of messages to the queue, if your consumer is running too slowly to keep up
@Pranshu Maheshwari Thanks for the explanation, but I'm a bit confused about the concurrency behavior. According to the docs, the queue should automatically scale up concurrency, but I'm seeing that it never exceeds 2 in the dashboard. Shouldn't the consumer be able to process messages in parallel to handle the backlog, especially when there's a burst of messages? Is there something specific I need to configure to enable higher concurrency, or could there be something limiting the parallel processing?
Hey Jacky, can you confirm that the
max_concurrency in the wrangler.toml file is not set to 2?Yep, can confirm we never set max_concurrency. Our consumer is just a worker API that can run for hours and shouldn't have any blocking. We're moving to a different queue service now, but I still want to figure out what caused this so we can make better use of Cloudflare's edge later.
@jacky could you send me your queue ID and account ID? Both are safe to share here
You don't need to do anything to enable concurrency. but here are a few reasons your consumers might not be auto-scaling:
https://developers.cloudflare.com/queues/configuration/consumer-concurrency/#why-are-my-consumers-not-autoscaling
Cloudflare Docs
Consumer concurrency · Cloudflare Queues
Consumer concurrency allows a consumer Worker processing messages from a queue to automatically scale out horizontally to keep up with the rate that messages are being written to a queue.