pat
CDCloudflare Developers
•Created by pat on 2/2/2025 in #queues
Hey guys, is there any problem with
Overall goal is to ingest data from an external service that has rate limiting
More specifically, I want to ingest League of Legends games into my db. As I can logically break it by match, I am sending a single message on a CF queue for each match and have a consumer that ingests the match.
I can create jobs much quicker than I can consume them and I want it to happen in the background, so I'm just chucking as many matches as I need onto the queue and letting it figure itself out.
Will have a CRON based worker see how many matches have been ingest + how many jobs are queued/in progress to tell how many more jobs I should push.
10 replies
CDCloudflare Developers
•Created by pat on 2/2/2025 in #queues
Hey guys, is there any problem with
Maybe my definition of recursion and the one in that thread are different. By recursive I dont mean a retry, I mean the consumption of message A leads to message B being published. IE that the consumer produces to its own queue because we uncover more jobs by completing a job.
10 replies
CDCloudflare Developers
•Created by pat on 2/2/2025 in #queues
Hey guys, is there any problem with
For anyone that happens upon this, it seems like its solved by acking the message or batch in code.
batch.ackAll() or msg.ack(). This is some phantom knowledge I now hold10 replies
CDCloudflare Developers
•Created by pat on 2/2/2025 in #queues
Hey guys, is there any problem with
I had a look around and found this thread that says recursion is not recommended. Can anyone please give me some idea as to why?
10 replies
CDCloudflare Developers
•Created by pat on 2/2/2025 in #queues
Hey guys, is there any problem with
For now I am going to try and fix this by separating it into 2 separate workers that publish to each others queue because I need a way to tell more accurately what my CPU time is. The amount of data I can ingest is limited by it.
10 replies
CDCloudflare Developers
•Created by pat on 2/2/2025 in #queues
Hey guys, is there any problem with
10 replies
CDCloudflare Developers
•Created by pat on 2/2/2025 in #queues
Hey guys, is there any problem with
10 replies
CDCloudflare Developers
•Created by pat on 2/2/2025 in #workers-help
Help debugging high CPU time in worker
Those 10 requests respond with 1MB of json each. Parsing them is taking a long time holy moly
3 replies
CDCloudflare Developers
•Created by pat on 2/2/2025 in #workers-help
Help debugging high CPU time in worker
Omg I think I understand
3 replies
CDCloudflare Developers
•Created by pat on 1/26/2025 in #queues
Hey all, wondering if theres a way to
I'll need to rescrape every few weeks as a new patch comes out. It is a recursive process so maybe I'll just keep a count of ingested data and configure a limit per patch and reject any new jobs if that limit is hit
4 replies
CDCloudflare Developers
•Created by pat on 1/26/2025 in #queues
Hey all, wondering if theres a way to
I am planning to use a few queues to collect data from an api and save it to my DB. I don't need heaps of data to have a meaningful dataset but there is effectively (for my purposes) infinite data available. The intention is to control how much I ingest by limiting the amount of jobs submitted to the queue. The more important reason is to control how much I pay for these queues while I am in testing.
4 replies
CDCloudflare Developers
•Created by pat on 9/30/2024 in #queues
Hey fellows, I've got a really simple
I'm not sure if this is a bug btw, the interface seems to allow async or sync - maybe this is just a quirk of cloudflare I'm not aware of?
2 replies
CDCloudflare Developers
•Created by pat on 9/30/2024 in #queues
Hey fellows, I've got a really simple
This is so painful to come back after literally 3 minutes with a solution... I just chucked an
async on the queue function and bam, it works. Pain2 replies