do we need to fill this form to test V2 or need to subscribe a paid plan?
do we need to fill this form to test V2 or need to subscribe a paid plan?
10 Replies
Vectorize V2 is in public beta. As long as you have access to Vectorize and use the latest version of Wrangler https://github.com/cloudflare/workers-sdk/releases/tag/wrangler%403.71.0 you should have access to Vectorize V2.
GitHub
Release [email protected] · cloudflare/workers-sdk
Minor Changes
#6464 da9106c Thanks @AnantharamanSI! - feat: rename --count to --limit in wrangler d1 insights
This PR renames wrangler d1 insight's --count flag to --limit to improve clarity ...
Unknown User•7mo ago
Message Not Public
Sign In & Join Server To View
Hi @BlockTxn. Could you please confirm that the API Token you are using includes the Vectorize:Edit permission for the account in which this index is present? That permission would be needed to create or delete Vectorize indexes.
Question about metadata indexing behavior: does it apply to existing vectors or only to those created after the metadata index is created? otherwise, does it backfill the index async?
FYI, submitted API site feedback, https://developers.cloudflare.com/api/operations/vectorize-list-metadata-indexes should be GET not POST; also fails in ⛅️ wrangler 3.71.0 when running
$ wrangler vectorize list-metadata-index <index-name>there is something bad going on with vectorize indexes, I tried three times to create a text embeddings index, I defined them with metric cosine and 1536 dimensions. It appears that after inserting a certain amout of vectors my index breaks and my queries starting returning absolutely nonsense "scores" - for example right now my queries to this cosine index are returning scores like 28.955 or something like that (and that is breaking search results)
I don't see my vectorize v2 index in cloudflare dashboard. I only see v1 vector index. listing it with wrangler cli works
Unknown User•7mo ago
Message Not Public
Sign In & Join Server To View
Hey Cloudflare team and community, I have a large dataset of
500M vectors, each with 256 dimensions. I've recently seen the changelog about Vectorize v2 being in public beta, which mentions support for up to 5 million vector dimensions per index but there is not vector dimension limit mentioned on limits page? I'd like some clarification and advice on how to best use this with my dataset.
Limits: https://developers.cloudflare.com/vectorize/platform/limits/
Changelog: https://developers.cloudflare.com/vectorize/platform/changelog/
Given my vector dimensions (256) and the new limit of 5 million vector dimensions per index, my understanding is that I could potentially store up to 19,531 vectors per index (5,000,000 / 256 = 19,531.25). Is this correct?
If so, I would need approximately 25,600 indexes to store all 500M vectors (500,000,000 / 19,531 ≈ 25,600). However, this seems impractical given the current limit of 100 indexes per account.
My questions are:
1. Is my understanding of the "5 million vector dimensions" limit correct? Or does this mean something different? I wish to insert 5M vector for each index based on Limits page statement: Maximum vectors per index= 5M
2. If my understanding is correct, what would be the best approach to handle such a large dataset with Vectorize? Given the current beta limit of 5 million vectors per index (on limits page, not changelog) I am proposing to distribute your data across 100 indexes, each containing 5 million vectors. My insertion strategy involves using a modulo operation to determine which index a vector should be inserted into. For querying, I plan to search all 100 indexes in parallel and then aggregate and rank the results.
3. Are there plans to increase the number of indexes allowed per account? If so wha tis maximum?
Any advice or insights would be greatly appreciated. Thank you!I'm getting a similar error to this when trying to create meta-indexes
this is the command I ran
ERROR] A request to the Cloudflare API Failed
Expected request with
Content-Type: application/json [code: 40026]
If you think this is a bug, please open an issue at:
https://github.com/cloudflare/workers-sdk/issues/new/choose
This is the output I'm getting
additionally the docs have this command with the name property and type property being wrapped in quotes on one page and not being wrapped in quotes on another
Both commands return the same error (I'm guessing its expecting a post request and using a get request)
Happens on both wrangler version: 3.72.0 and 3.72.2(latest)Hey @Joey. Thank you for reporting this! We are aware of this issue (https://github.com/cloudflare/workers-sdk/issues/6516) and we have pushed a fix. It should be available with the next release of Wrangler (expected early next week).
If you'd like to unblock yourself today, you can use a Wrangler pre-released version associated with this fix: https://github.com/cloudflare/workers-sdk/pull/6548#issuecomment-2302185968.
GitHub
🐛 BUG: Error when wrangler vectorize create-metadata-index on Vecto...
Which Cloudflare product(s) does this pertain to? Wrangler What version(s) of the tool(s) are you using? Wrangler 3.72.0 What version of Node are you using? 22.3.0 What operating system and version...
GitHub
fix: Fix Vectorize JSON payloads by garvit-gupta · Pull Request #65...
What this PR solves / how to test
Add content-type header to Vectorize POST operations
Fix Vectorize getVcetors, deleteVectors payload in Wrangler Client
Fixes #6516/VS-269 and VS-271.
Author has a...