Where do queue consumers execute? I've
Where do queue consumers execute? I've tried some searching here, but can't seem to find an authoritative answer... I mean you could have event fed into a queue from all over the world, but to ensure ordered handling, do consumers only execute in 1 geographic location? Meaning the latency from a producer queueing a message to a consumer processing a messaging will vary significantly and so queues should not be used for anything "real-time-ish" - is that correct?
4 Replies
Here's what we can publicly share about the architecture: https://blog.cloudflare.com/how-we-built-cloudflare-queues/
The Cloudflare Blog
Durable Objects aren't just durable, they're fast: a 10x speedup fo...
Learn how we built Cloudflare Queues using our own Developer Platform and how it evolved to a geographically-distributed, horizontally-scalable architecture built on Durable Objects. Our new architecture supports over 10x more throughput and over 3x lower latency compared to the previous version.
But yes, the latency to go from producer => consumer can vary a lot. Queues are meant to be when you don't need to do this work in realtime, and when you can do the work asynchronously 🙂 also note - we don't ensure ordered handling. events might appear out of order.
What's your use case for queues? What are you thinking about doing?
Thanks for your response @Pranshu Maheshwari! To date our entire backend is 1 cloudflare worker. I'm trying to separate it into multiple CF worker "microservices". I'm still learning various patterns and techniques and may do it all "in-process" with Worker RPC. But it did seem like cloudflare queues and cloudflare pubsub are two products that would be great for composable architecture with a bunch of services. However it seems like neither one is quite ready for primetime yet.
I've been doing a bunch of domain-driven-design (DDD) and event-driven-architecture (EDA) courses recently and trying to model my cloudflare workers using those patterns. It's going really well, but I haven't yet decided how to handle/broker domain events (application events that may trigger some other services/workers to do something). Domain events could be pushed onto various queues, they could be published to a pubsub style broker, and what I might actually do is an "application internal broker" which just routes the events within the workers via some static code for simplicity and so the consumers of those domain events will be executing in the same datacenter as the producer of those events (if I need it somewhere else then the service can delegate to a DO).
very cool. Queues is very robust - definitely give it a try. It should work really well here - with the caveat that each queue can only have a single consumer. If you need to fan out messages, you'd have to do it manually, by writing a consumer worker which writes to multiple queues. Let me know if you'd like help as you figure things out!