Is best practice for supporting more complicated query semantics than just filtering to upsert vecto
Is best practice for supporting more complicated query semantics than just filtering to upsert vectors with a metadata field containing foreign keys to D1 records? Then implement your own pre/post filtering between the two?
12 Replies
as an aside it seems like if the reasoning behind launching vectorize as a separate product from D1 is that either sqlite did not support vector search at the time of launch, or that it was not considered an ideal substrate for vector search, it could be nice to integrate vectorize within D1 under the hood (with restrictions to what is possible within vectorize databases) so that users could write simpler queries/have easier setup/less to manage. Because at the end of the day a lot of people (like me) will want to combine topK with joins and it'd be nice to not have to implement some of that myself, assuming I'm not missing something re: the current impl
Given the limitations of the meta-filtering operation, I have to leverage a nested key structure as documented:
{ "pandas.nice": 42 }
// looks for { "pandas": { "nice": 42 } }
However, when I attempt to upsert metadata where the key contains a JSON object, as specified in the documentation, I encounter the following error:
VECTOR_UPSERT_ERROR (code = 40019): invalid metadata for vector id=“jhbqpwft0qogbr1fs6re-0”; the metadata JSON object cannot contain JSON objects.
I’m not sure what to do next. Does anyone have any suggestions? Thanks!
When do you expect to update the Typescript API to use V2? I'm running into a problem where Wrangler doesn't support the
--deprecated-v1 flag on certain calls. I just created the index today so I'm very happy to use V2 but don't want to roll my own client.npm
cloudflare
The official TypeScript library for the Cloudflare API. Latest version: 3.5.0, last published: 3 months ago. Start using cloudflare in your project by running
npm i cloudflare. There are 93 other projects in the npm registry using cloudflare.Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
We posted a deep dive into how we are building Vectorize on Cloudflare's blog. https://blog.cloudflare.com/building-vectorize-a-distributed-vector-database-on-cloudflare-developer-platform/ Check it out 🙂
The Cloudflare Blog
Building Vectorize, a distributed vector database, on Cloudflare’s ...
Vectorize was recently upgraded and made generally available, now supporting indexes of up to 5 million vectors, delivering faster responses, with lower pricing and a free tier. This post dives deep into how we built Vectorize to enable these improvements.
thanks. Seems not null is not functional, for example, for any not indexed properties , if i use {"$ne","null"}, it always return everything
Are you quoting
null like that?
Because that's a string "null", not actual null.
Whereas your initial question wasn't quoted, meaning it would be actual nullsorry for the typo i did {"path": {"$ne": null}, } and the path prop is not indexed
Quick question
1. Why do we need to create metadata index and even if we have to create, why only 1 key at a time and upto 10 index is the limit? While upstash allows to filter on metadata normally on all metadata
2. Can we also expect glob and more type of searches operators along with eq and neq
Like : contains , IN, not in, glob, pattern match, and , or etc
3. can we also have hybrid search like a cosine similarity search with Full text search as well for better search in future?
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
Hey! Whats the limit on insert/upsert.
for some reasons, my vector inserts/upsers have stoped at 128.04k vectors.
In dev, in another vectorize database, everything is working fine. Is there any hidden limit we dont know about???
@yevgen
Is there a way to dynamically add Vectorize bindings programatically?
ah i found it https://developers.cloudflare.com/vectorize/best-practices/create-indexes/
Hmmm, but these don't seem to be enough number for my use-case
Indexes per account 50,000 (Workers Paid)