Brett Willis
CDCloudflare Developers
•Created by Brett Willis on 2/12/2025 in #queues
Feature request: would be nice to be
I think it's been suggested before that it would be super cool to have a "sharded concurrency key" (for lack of better words), where for example we could use the user ID as the concurrency key so that any messages with a different key can run in parallel but those with the same key have limited concurrency.
But I digress, this would still require using the same queue anyway.
3 replies
CDCloudflare Developers
•Created by Brett Willis on 2/12/2025 in #queues
Feature request: would be nice to be
Nice!
Well... we're using one queue at the moment because we don't have a better solution for the following requirement: all of the messages in question are performing operations on users in our database (either a specific user or a batch of a whole bunch of users). If these batch jobs are all running concurrently, then there's basically going to be a whole lot of contention, potentially trying to operate on the same users at the same time.
So we use a single queue with a concurrency of 1 for basically everything that operates on users. It could be a bit of a bottleneck... but the throughput is fine at the moment and it just allows us to eliminate contention between all these batch jobs. Can't think of a better way of achieveing this?
Some of these batch jobs are triggered basically on a CRON, so they should just be discarded it unprocessed by the time the next CRON triggers another.
And some batch jobs are critical events that must be ingested eventually and therefore not discarded.
Hopefully that makes sense to clarify our use-case.
Right now, we just concede that the CRON batch jobs will keep accruing if unprocessed, which is not perfect, but it only happens when some part of the processing pipeline is down anyway, so not the biggest deal.
3 replies
CDCloudflare Developers
•Created by Brett Willis on 1/29/2025 in #workers-help
~5% of our workers requests never return from Cloudflare after the worker returns
The issue was related to promises that we cache on the global scope, and we relied on
waitUntil() and <30s timeouts to ensure they didn't get left dangling. But there must be something we don't understand going on with how the global scope is persisted and restored between requests. Long story short, some requests would get a global scope with promises that, one way or another, had been left dangling and therefore they would never return.
So we added timestamps and guards along with those global promises to detect or timeout if they are too old. Problem solved.33 replies
CDCloudflare Developers
•Created by Brett Willis on 1/29/2025 in #workers-help
~5% of our workers requests never return from Cloudflare after the worker returns
I think I have to eat my words here! Until I started the concerted effort to get some Ray IDs I could have sworn all the requests I caught had logs with
outcome: "ok"... which understandably lead me to look elsewhere than the code, hence the undocumented location hint we're using (because we've had outages related to smart placement in the past).
However subsequently all the requests I caught when trying to find Ray IDs had no logs!
So I tried a few things in the code and have given it a few days to observe, and it looks like the issue has been resolved that way.
So my apologies @Walshy for wasting your time here, but I do thank you for your great support here as always. It's great to know you're around to help!33 replies
CDCloudflare Developers
•Created by Brett Willis on 1/29/2025 in #workers-help
~5% of our workers requests never return from Cloudflare after the worker returns
Thanks @Walshy | Workers/Pages I haven't managed to catch a ray yet. It's still happening though... Working on it.
33 replies
CDCloudflare Developers
•Created by Brett Willis on 1/29/2025 in #workers-help
~5% of our workers requests never return from Cloudflare after the worker returns
Perfect, thank you so much!!
33 replies
CDCloudflare Developers
•Created by Brett Willis on 1/29/2025 in #workers-help
~5% of our workers requests never return from Cloudflare after the worker returns
Also it's been happening for a week or two so it's not something recent. We just recently managed to capture it.
33 replies