Workers KV bulk reads
Is there a way that a bulk get can be implemented for cloudflare KVs? Like in a way that can keep the limit but only require one call to limit the number of requests that are required to be sent for large data querying? For example, I want to create a bulk endpoint for a large number of lookups. Currently, my client is held up by making thousands of requests which are held up by browser limitations. If I can bundle them in say 200-300 requests to a bulk endpoint it would save my client very precious time
3 Replies
This is interesting, what are you storing in KV and whats the use case for a bulk read? Are you using the Workers KV API?
I can’t say exactly on a public forum, but essentially it is a client in-browser that fires up a request to a worker to provide the processing power that the browser cannot provide. It uses 6 different KV stores that store simple information used to determine the action of the worker. The kv stores are split up for other workers and are fed data via a kv api call from local data centers.
I’m working on a way to optimize it to help prevent the chaining up problem that I’m having with the browser only allowing between 4-6 parallel requests up at a time by bundling the information and sending it in one request.
The problem is that a worker can only send 6 simultaneous requests at once. By my testing, a ramped up KV get request on these resources requires about a 50ms transmit delay. Very impressive, but I’m wanting to send batches of 200-300 items to be processed. The chaining of sending so many get requests to the KV stores makes the round trip a 2.5 second delay per batch leaving my tool to have up to a 12 second delay on a time sensitive process.
Does this help?
Following up to see if this example was helpful or if you need further information
This is helpful, and totally understand you can't share more on a public forum. I want to think about if there's a better patter that could work here, any chance we can meet? here's my link https://calendar.app.google/ynRoYiJudGqyGPRN9