workers-help
pages-help
general-help
durable-objects
workers-discussions
pages-discussions
wrangler
coding-help
kv
🦀rust-on-workers
miniflare
stream
general-discussions
functions
zaraz
âš¡instant-logs
email-routing
r2
pubsub-beta
analytics-engine
d1-database
queues
workers-for-platforms
workerd
web3
🤖turnstile
radar
web-research
logs-engine
cloudflare-go
terraform-provider-cloudflare
workers-ai
browser-rendering-api
analytics
next-on-pages
cloudflare-ai
build-caching-beta
hyperdrive
vectorize
ai-gateway
python-workers-beta
vitest-integration-beta
workers-observability
workflows-beta
It would be great to be able to manually
It would be great to be able to manually pause/resume queues (collect from producers but pause dispatch to consumers) and also to purge queue messages. This help both in development and in production e.g. maintenance/migration, disaster recovery, or when dependent services are experiencing downtime. Queues are good for decoupling services for this very reason, and it's good to be able to intervene in emergencies!
is there message group like feature in
is there message group like feature in queue ? i.e sqs message group
@Pranshu Maheshwari Hi! Some questions
@Pranshu Maheshwari Hi! Some questions about Queues:
1. Do you plan to support message deduplication (refusing to publish the same message while it exists in the queue + support for 'force' mode to rewrite it)?
2. Do you have any plans to add an API method to be able to remove a message from the queue by its ID or impedance key?...
Hi! I received no response so far after
Hi! I received no response so far after filling out the form. How long does it usually take?
R2 Event Notifications Metadata
Hi all, I'm working with an R2 event notification queue and everything is working smoothly, love the feature and ease of use. I was wondering if there's any chance object metadata might be included in the upload notification event payload in the future?
Hi again! Is it possible to test pull
Hi again! Is it possible to test pull consumers locally? I.e. fetch() from some wrangler local dev host rather than api.cloudflare.com/.../queues/../messages/pull. Big thanks!
is there any kind of queue bindings-per-
is there any kind of queue bindings-per-worker limit? (sorry if I missed it anywhere in the docs). curious whether it's possible/sensible to have a queue-per-customer architecture all written to by the same producer worker. thanks!
I'm a big fan of CF Queues and
I'm a big fan of CF Queues and especially the Consumer Worker model! The biggest thing I wish for is easier way to publish items to a Queue from external systems (like external Kafka Topics).
For example I'm using Debezium for change data capture on an external Database.
This then leads me to having to use Kafka -> HTTP -> Worker -> Queue to get these messages to a CF Queue which is a lot of overhead and you have to deal with all of the usual Webhook headaches...
Appointments
Hi @everyone! Pranshu here. I’m a product manager at Cloudflare and I’ve recently taken on Queues. I’d love to learn more about how all of you are using Queues, and how you'd like to see it improve.
I’d especially love to hear from folks building startups. I’ve been a 2x founder myself, so I’ve been in your shoes!
If you’d like to grab some time to chat in the next two weeks, here’s my calendar: https://calendar.app.google/pdzfMFmwACg2PN8w8...
I am currently following this example
I am currently following this example https://github.com/cloudflare/workers-rs/blob/078d15bb60d2b05af0236c7b1370f598f5e17ada/examples/queue/src/lib.rs#L113
I am facing the following error message, similar to https://discord.com/channels/595317990191398933/1237042263150956667/1237062080117473280
```
Queue test-queue (5 messages) - Exceeded CPU Limit @ 30/05/2024, 19:52:42...
I am trying to deploy the basic queue
I am trying to deploy the basic queue template worker (which worked last week) and am getting this error.
✘ [ERROR] A request to the Cloudflare API (/accounts/[REDACTED]/queues/[REDACTED]/consumers) failed.
workers.api.error.unknown [code: 10013]
...
could i have some help understanding
could i have some help understanding what all of the stats mean? im a little confused.
`Conflict` I've only seen once. Still
Conflict Error: Queue send failed: Internal Server Error occasionally on .send, which calling-worker-side retrying does not seem to fixGiven you can't use a pages app and a
Given you can't use a pages app and a separate worker to consume a queue locally what is people's current workflows for handling this? I was thinking of deploying a queue and then just using that but I'm not sure how to have 1 binding be remote but the rest local
Queue processing stops locally
Stopped locally? Do you see errors? Can you open a bug report on GitHub (including the wrangler version & your code) - ?
This is something on the R2 side, but we
This is something on the R2 side, but we're also working on R2 Event Notifications - e.g. uploading to R2 via a Signed URL or the S3 API will publish an event to one of your Queues that you can consume.
Some updates for @Queues users:
Some updates for @Queues users:
- We're working on landing HTTP Pull, which will allow you to poll a queue and consume it from outside of a Worker (or use a Worker in a poll-based flow vs. push-based). This will likely land just before/just after the holidays.
- Queue throughput: our major milestone is about 3-5k msg/sec per queue (message size plays a role here), up from the ~400 msg/sec/queue currently
- We're also working on landing controls around setting message delays - e.g.
await env.MY_QUEUE.send(msg, { deliveryDelay: 600 }) on the producer side, and retry delays on the consumer side - e.g. messages.batch.retryAfter(300) or .retryAfter(seconds: number) on a per-message basis. API not final....No: batch timeout is on the producer
No: batch timeout is on the producer side. It does not determine how the consumer scales. Your volume/message rate is very low, and (right now) we prefer to avoid scaling too quickly on the consumer side.
Of course : )
Of course : )
So basically I want to locally debug the queue producer and queue consumer. The queue producer is a cloudflare pages app, built on Remix. The queue consumer is a worker with plain typescript, no framework.
I kept the worker that simple intentionally because it doesn't need database access or routing or whatever, it's basically a proxy for queue consumption (because cloudflare pages apps can't consume from a queue yet and other limitations)....
elithrar
@Matt !
Thanks for 1 and 3.
For 2: we finally got a reply from our Cloudflare TAM and yes, they stated we need to add a payment method on our non-prod account. I'm working on us setting that up....