I really would like to use Cloudflare Queues instead of a third party provider, but consumer locatio
I really would like to use Cloudflare Queues instead of a third party provider, but consumer location not being affected by smart placement (round trips become a huge issue), concurrent consumers only scaling at the end of a batch and the 6 simultaneous open connections per worker instance result in concurrency autoscaling not working as expected and taking too long to scale up. My queue backlog becomes huge.
I get that the magic of autoscaling would be great but reading people complaining about the same thing shows that we are not there yet, or maybe we are just holding it wrong. I believe things would be way better if consumers would scale up as messages comes in, or if we had a min_concurrency setting (of course I don't know how viable it would be for us to have those). I'm really frustrated by the results of trying to use Queues again and again since the beta and still hitting the same problem.
4 Replies
Unknown User•3d ago
Message Not Public
Sign In & Join Server To View
Hi there! Looking for some help diagnosing a weird issue I've been running into. I am pulling data from an external API and then sending potentially 1k+ updates to Airtable. In order to circumvent the 1k subrequest limit, I'm caching the updates in KV storage, then sending the key to Queues to actually perform the updates. However, some messages seem to never be received by the consumer and are instead sent directly to the configured DLQ. I have not been able to reproduce this locally, it is only happening once deployed. Has anyone experienced something similar and/or have suggestions for debugging?
Amazing, keep it up
why queues isn't free
give at least 1000 free monthly messages so someone can test stuff