Support for assets > 25 MiB in KV
Support for assets > 25 MiB in KV
1ļøā£ OMG YES I NEED THIS
2ļøā£ Neat
3ļøā£ I need you to fix something else (please specify)
Poll by beelzabub#0369
16 Replies
Not committing to anything but as part of KV 2.0 one of the things I've been noodling on is whether this kind of feature is useful.
If the limit wasn't 25 MiB, then what would it be? Any thoughts on that?
What would make it interesting to you?
Unknown Userā¢3y ago
Message Not Public
Sign In & Join Server To View
Hmm, most of my use for KV is more with storing web assets and such, maybe 100 MiB?
To me, it seems right now there's a pretty big cliff where you go from KV's 25 MiB Limit to R2's 5 TB, losing KV's benefits.
Yeah. I don't know if the R2 team will pick this up and integrate or if this might become a standalone product, but the architectural improvements we're landing for KV are reusable both internally and potentially externally as a standalone product. Not happening any time soon though for sure.
Hmm, I agree the 25MB limit is sometimes just a bit too small for web assets, although what I'm most sad about is that KV is more like an edge "cache" than a storage solution/database. Being pull-based I find it hard to justify $5/m list/write operations with reading potentially stale data for up to a minute (or longer if explicitly cached, yet still billed for reads). If it were push based, something like quicksilver, I'd give up size - say 8k would be more than enough for some kv-like configuration data storage - we have AE now 
so it would be so perfect for saas's to have config stored in KV, but eventually be pushed to all colos for low latency cold reads.
so it would be so perfect for saas's to have config stored in KV, but eventually be pushed to all colos for low latency cold reads.Have you signed up for KV 2.0 Beta?
I have 
Kk. I hope to start admitting people in soon. What does "something like silverlight" mean?
*quicksilver, I got the name confused š
That would be expensive lol, can't imagine that being cheap at all. Does KV's latency/pull-basedness effect you that much? If you get a enough traffic to get stuff in cache, and it revalidates in the background, should be set.
Ah ok. KV 2.0 is hybrid push+pull and should be somewhat competitive with QS in terms of how quickly writes propagate and more than competitive in terms of how many write/s we can support. In theory we can make the eyeball see a write within seconds but thereās logistical and scaling challenges with making it actually happen, so that capability may have more expensive pricing. New API capabilities and their pricing isnāt something weāll have clarity on for a few more months I think
Again, it doesn't necessarily have to be KV, and it doesn't have to support 25MB values, I'd be happy if keys+metadata were pushed, and the value pulled when read for example
I see. I was curious what exact use case you had that KV wouldn't cover but Quicksilver would, but rereading it again I guess your reasoning is more along not wanting stale data & wanting faster lists and other operations then just simple fetching from a key.
Super interesting that Vitali said it is push+pull though, sounds like it may be somewhat like you are saying. Hopefully we get beta access sometime 
Iām happy to give anyone beta access. Thereās a signup list. I think we lit up a bunch of test namespaces last week. Weāre in the middle of fixing a bad constituency bug but I think we have it narrowed down so weāre going to go broader on the rollout sometime this week. Iām also going to challenge this assumption that the only way to get it is through push. I think we can get low consistency delay reads another way. As for āglorified cacheā vs ādatabaseā, I think thatās a very vague comparison and depends on how you define ādatabaseā. Anything that stores structured data is technically a database. A file system is a non-distributed hierarchical database. A CDN is a distributed eventually consistent database. QS is a push replicated KV store. KV2 is a hybrid push + edge cache w/ opportunistic refresh. I have ideas on how to explore making it fully push but the main challenge is one of scale. For example, KV scales better than QS both in terms of total amount of data stored, write throughput, and the sizes of the objects. QS performs better for read latency but thatās hard to compare because thereās a number of bottlenecks in our way (that weāre working to remove) that arenāt for QS. The only thing left I think is the consistency delay and I think we can actually be just as good (maybe better depending on how you measure) without needing to push all the way to the edge. YMMV of course but if you havenāt signed up for KV2 I would recommend doing it. Itās not yet possible to self control the refresh rate but if itās helpful for you to experiment, we can discuss.