I'm a big fan of CF Queues and especially the Consumer Worker model! The biggest thing I wish for is
I'm a big fan of CF Queues and especially the Consumer Worker model! The biggest thing I wish for is easier way to publish items to a Queue from external systems (like external Kafka Topics).
For example I'm using Debezium for change data capture on an external Database.
This then leads me to having to use Kafka -> HTTP -> Worker -> Queue to get these messages to a CF Queue which is a lot of overhead and you have to deal with all of the usual Webhook headaches
19 Replies
Thanks
What is it and how we can use it? what's the real life usecase of this?
i heavily agree with this. queues have a intended scenario of working on a per-customer basis, which i understand to be one queue per customer - each customer can then be a HTTP consumer and get+ack messages, but the issue is publishing messages to the queue. creating a queue is already possible with the CF api, the question only is if you can create enough access tokens for the queues
Since ~18:00 yesterday one of our queues has been erroring with
Queue send failed: Internal Server Error.
Trying to redeploy the worker gives workers.api.error.unknown [code: 10013], as does trying to delete the queue. Anyone able to help figure out what is going on here?
Queue ID: 18dbd75d6a1946b48f614689ed23bec9Which timezone and what is your account ID? (safe to post here)
Timezone: BST
Account ID: ef98901aa25fef025cee6651b601b491
Are you still seeing issues? (either sending messages or creating a Queue)
Was this sporadic?
This is ongoing, we are unable to interact with that queue in any way. Even trying to update the "Delivery delay" gives the same error
@Pranshu Maheshwari ^
We've deployed a new version of our worker under a new name, and a new queue so we can work around this, but it's ongoing for this queue
Thanks for highlighting this, we're looking into it
@FraznoFire ?
We use queues to aggregate calls to an external API, only thing that would make queues for that use-case better is that batches currently cannot be grouped by a (custom) identifier - say we scope all the data to a single customer and group it under a 30s queue timeframe, we could have more efficient calls to the external API (since they also operate under a per-customer/ID resource scope).
This is relevant because the external API has relatively strict account rate-limits (not scoped to the ID).
Right now it's hit or miss since possibly 100 different customers could join the same batch effectively resulting in groups of 1 and thus 100 API calls.
In this case we would prefer grouped batching over quick execution of a random batch if you understand what I mean
is there any kind of queue bindings-per-worker limit? (sorry if I missed it anywhere in the docs). curious whether it's possible/sensible to have a queue-per-customer architecture all written to by the same producer worker. thanks!
Unknown User•5mo ago
Message Not Public
Sign In & Join Server To View
Hmm I haven't been able to replicate this on my side. CLI arguments should override what you've specified in
wrangler.tomlHello,
I intend to use queues to save the data of each visit and then import them to have an analytic.
Is this the use of queues? I understand that yes, but if someone confirms me I would appreciate it.
What kind of analytics? Writing directly to WAE may be more effective: https://developers.cloudflare.com/analytics/analytics-engine/
Cloudflare Docs
Workers Analytics Engine · Cloudflare Analytics docs
Workers Analytics Engine provides unlimited-cardinality analytics at scale, via a built-in API to write data points from Workers, and a SQL API to …
a personalized one, I just wanted to know if it is useful for what I proposed.