thanks for the feedback here! very detailed & agreed that this is a pain point with queues today 🙂
thanks for the feedback here! very detailed & agreed that this is a pain point with queues today 🙂
we're thinking about ways to improve this. our plan right now is to add the ability to connect multiple consumers to a single queue. you'll be able to specify partition keys to route specific messages to specific consumers. and ideally this can be dynamic, so you don't have to specify all the rules ahead of time. this project is still in the design phase though, so it's too soon to share timelines.
dynamically creating queues doesn't work well today, alas! rather than 1 queue per user, would you be able to setup a few queues as high-priority queues, and setup separate low priority queues? by default, messages get routed to the high priority queues. if a user uploads 10k files at once, you route those messages to the low priority queues so that they don't block each other?
17 Replies
Any plans to support pushing to Queues from an external platform direct via the rest API (not via an intermediate Worker)?
We published a deep dive on the Queues architecture today!
https://blog.cloudflare.com/how-we-built-cloudflare-queues/
The Cloudflare Blog
Durable Objects aren't just durable, they're fast: a 10x speedup fo...
Learn how we built Cloudflare Queues using our own Developer Platform and how it evolved to a geographically-distributed, horizontally-scalable architecture built on Durable Objects. Our new architecture supports over 10x more throughput and over 3x lower latency compared to the previous version.
I have a queue with batch of size 50 and timeout of 10s.
Few questions:
1- If I do
.send will msg be processed as soon as it arrives or it will wait for 10s before processing?
2- If a batch of 50 msgs is being processed by consumer and it takes longer than 10s, what happens next, if there are more messages waiting to be processed.
2.1- Will a new batch be delivered to the consumer? If so, will it make a new instance of the consumer or same isntance of consumer will process it.
2.2 Will new batch wait till the processing of previous batch is complete?
3- There is something related to concurrency. But it seems like it is never triggered. How can I verify if it is being triggered?
For now it seems like all processing is sequential. It becomes a bottleneck very quickly if mutliple users are using it.Unknown User•4w ago
Message Not Public
Sign In & Join Server To View
messages in my queue, even if they dont have a delay attached to them, are often not being delivered until 20-30 minutes after they are queues (sometimes more, the most ive seen so far is 1h 20m). any idea why that might be?
and ones that do have a delay attached to them are 20-30 minutes later than they are supposed to be
it looks like it started happening on october 23rd, around 9pm UTC
Can R2 Event Notifications to Queue be tested in local development?
Hello - I have a Queue with polling client running behind firewall. What is appropriate interval time to poll?
I also want to understand cost for this continuous polling.
Hello! I would be interested in FIFO ordering and also interested to know if you are going to support something like AWS SQS Message Group ID?
I tried to send an email to [email protected] as specified in the docs (https://developers.cloudflare.com/queues/configuration/javascript-apis/#messagebatch) but that bounced.
Thanks!
Hello, is there an issue with queues at the moment? I've got messages being reported as being in my deadletter, however I can't see them when listing messages and nothing else is reading from the queue 🤔
Is it possible to add multiple messages to a queue from the dashboard? Seems like no matter what JSON I put in there (even trying to replicate a message batch) it doesn't work
Small Question about Concurrency - if i publish a message with the same content twice, will only one be processed? Example: i have a worker that publishes an event to tell the queue to update an object in an database, if this step is triggered twice, and two queue workers will be invoke, this would possible create a race condition
Yes, each message is a message so both will be processed possibly concurrently if you have a busy queue. You can disable concurrency though if you need to process them more controlled. See: https://developers.cloudflare.com/queues/configuration/consumer-concurrency/
thx, yeah i have concurrency disabled currently, just hoped i overlooked something. we have a really busy queue and without concurrency it takes a while. i might add a second queue, for events where race conditions are a non-issue
Yeah either remove it being a problem that it runs concurrently (not sure about your use case obv. but maybe possible) or seperate the queues. You can even add more queues and "shard" them so they are still not concurrent but you have multiple queues processing. There are ways around it 😄
we handle cash register transactions async so if anything is triggered twice we really have a problem :KEKW:
You can even pipe your concurrency challenged jobs through a durable object which can work as a gate to only allow a single operation to run.
thats, actually a really good idea