Hey guys, I'm try to insert a vector via REST API, but I'm getting ```json { "result": null, "

Hey guys, I'm try to insert a vector via REST API, but I'm getting

{
  "result": null,
  "success": false,
  "errors": [
    {
      "code": 4000,
      "message": "vectorize.vector.missing_part"
    }
  ],
  "messages": []
}

{
  "result": null,
  "success": false,
  "errors": [
    {
      "code": 4000,
      "message": "vectorize.vector.missing_part"
    }
  ],
  "messages": []
}

I'm creating a blob to simulate as a file and send it via formdata

const blob = new Blob([JSON.stringify({ id, values }) + '\n'], { type: 'application/x-ndjson' });
console.log('blob :', blob);
const formData = new FormData();
formData.append('file', blob, 'data.ndjson');

const response = await fetch(`${c.env.CLOUDFLARE_API_URL}/accounts/${c.env.CLOUDFLARE_ACCOUNT_ID}/vectorize/indexes/${vectorize}/insert`, {
   method: 'POST',
   headers: {
     Authorization: `Bearer ${c.env.CLOUDFLARE_API_TOKEN}`,
   },
   body: formData,
 });

const blob = new Blob([JSON.stringify({ id, values }) + '\n'], { type: 'application/x-ndjson' });
console.log('blob :', blob);
const formData = new FormData();
formData.append('file', blob, 'data.ndjson');

const response = await fetch(`${c.env.CLOUDFLARE_API_URL}/accounts/${c.env.CLOUDFLARE_ACCOUNT_ID}/vectorize/indexes/${vectorize}/insert`, {
   method: 'POST',
   headers: {
     Authorization: `Bearer ${c.env.CLOUDFLARE_API_TOKEN}`,
   },
   body: formData,
 });

Am I doing something wrong? Any ideas?

13 Replies

schkovich•8mo ago

Bad Request / Internal Server Error

import http.client

conn = http.client.HTTPSConnection("api.cloudflare.com")

headers = {
    'Content-Type': "application/json",
    'Authorization': f"Bearer {vectorize_token}"
    }

conn.request("GET", f"/client/v4/accounts/{account_id}/vectorize/indexes", headers=headers)

res = conn.getresponse()
data = res.read()

print(data.decode("utf-8"))

import http.client

conn = http.client.HTTPSConnection("api.cloudflare.com")

headers = {
    'Content-Type': "application/json",
    'Authorization': f"Bearer {vectorize_token}"
    }

conn.request("GET", f"/client/v4/accounts/{account_id}/vectorize/indexes", headers=headers)

res = conn.getresponse()
data = res.read()

print(data.decode("utf-8"))

returns correctly list of indexes.

import http.client

conn = http.client.HTTPSConnection("api.cloudflare.com")

payload = "/home/ubuntu/jupy/notebooks/test_vector.ndjson"

headers = {
    'Content-Type': "application/x-ndjson",
    'Authorization': f"Bearer {vectorize_token}"
    }

conn.request("POST", f"client/v4/accounts/{account_id}/vectorize/indexes/hey-hi-index/upsert", payload, headers)

res = conn.getresponse()
data = res.read()

print(data.decode("utf-8"))

import http.client

conn = http.client.HTTPSConnection("api.cloudflare.com")

payload = "/home/ubuntu/jupy/notebooks/test_vector.ndjson"

headers = {
    'Content-Type': "application/x-ndjson",
    'Authorization': f"Bearer {vectorize_token}"
    }

conn.request("POST", f"client/v4/accounts/{account_id}/vectorize/indexes/hey-hi-index/upsert", payload, headers)

res = conn.getresponse()
data = res.read()

print(data.decode("utf-8"))

unfortunately upsert is returning 400 Bad Request path to ndjson file is correct. there is only a single 768 vector.

!ls -lh test_vector.ndjson
-rw-r--r-- 1 ubuntu ubuntu 17K Apr 12 15:49 test_vector.ndjson

!ls -lh test_vector.ndjson
-rw-r--r-- 1 ubuntu ubuntu 17K Apr 12 15:49 test_vector.ndjson

I am getting the same results with curl. curl response differs slightly:

curl --request POST   --url https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/vectorize/indexes/hey-hi-index/upsert   --header "Authorization: Bearer ${VECTORIZE_API_TOKEN}"   --header 'Content-Type: application/x-ndjson'   --data test_vector.ndjson
{
  "result": null,
  "success": false,
  "errors": [
    {
      "message": "unexpected - Internal server error"
    }
  ],
  "messages": []
}

curl --request POST   --url https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/vectorize/indexes/hey-hi-index/upsert   --header "Authorization: Bearer ${VECTORIZE_API_TOKEN}"   --header 'Content-Type: application/x-ndjson'   --data test_vector.ndjson
{
  "result": null,
  "success": false,
  "errors": [
    {
      "message": "unexpected - Internal server error"
    }
  ],
  "messages": []
}

My best guess that something might be wrong with ndjson format

[0.032991889864206314, 0.0012868141056969762, ,,,]

[0.032991889864206314, 0.0012868141056969762, ,,,]

Unknown User•8mo ago

Message Not Public

schkovich•8mo ago

Same response.

import requests
from requests_toolbelt import MultipartEncoder

def upsert(url, file_name, content_type, authorization_token):
  """Sends a POST request.
  :param url: URL for the new :class:`Request` object.
  :param file_name: File to be sent/attached in the body of the request.
  :content_type: File content type
  :authorization_token: Cloudflare authorization token
  :return: :class:`Response <Response>` object.
  """
  payload = MultipartEncoder(
    fields = {'field1': ('filename', open(file_name, 'rb'), content_type)}
  )
  headers = {
    'Content-Type': payload.content_type,
    'Authorization': f"Bearer {authorization_token}"
    }
  return requests.post(url, data = payload, headers = headers)
 
upsert(
    f"https://api.cloudflare.com/client/v4/accounts/{account_id}/vectorize/indexes/hey-hi-index/upsert",
    "test_vector.ndjson",
    "application/x-ndjson",
    vectorize_token
)
<Response [400]>

import requests
from requests_toolbelt import MultipartEncoder

def upsert(url, file_name, content_type, authorization_token):
  """Sends a POST request.
  :param url: URL for the new :class:`Request` object.
  :param file_name: File to be sent/attached in the body of the request.
  :content_type: File content type
  :authorization_token: Cloudflare authorization token
  :return: :class:`Response <Response>` object.
  """
  payload = MultipartEncoder(
    fields = {'field1': ('filename', open(file_name, 'rb'), content_type)}
  )
  headers = {
    'Content-Type': payload.content_type,
    'Authorization': f"Bearer {authorization_token}"
    }
  return requests.post(url, data = payload, headers = headers)
 
upsert(
    f"https://api.cloudflare.com/client/v4/accounts/{account_id}/vectorize/indexes/hey-hi-index/upsert",
    "test_vector.ndjson",
    "application/x-ndjson",
    vectorize_token
)
<Response [400]>

Let me know if I can provide further details.

Unknown User•8mo ago

Message Not Public

schkovich•8mo ago

same. let me try to insert that vector using wrangler. just to be sure that the file is in the correct format. @cfnathan Okay, it appears that file format is wrong.

$ npx wrangler vectorize insert hey-hi-index --file test_vector.ndjson 
✨ Uploading vector batch (1 vectors)

✘ [ERROR] A request to the Cloudflare API (/accounts/account_id/vectorize/indexes/hey-hi-index/insert) failed.

  vectorize.invalid_body_payload [code: 1006]

$ npx wrangler vectorize insert hey-hi-index --file test_vector.ndjson 
✨ Uploading vector batch (1 vectors)

✘ [ERROR] A request to the Cloudflare API (/accounts/account_id/vectorize/indexes/hey-hi-index/insert) failed.

  vectorize.invalid_body_payload [code: 1006]

How it should it be formatted?

[0.032991889864206314, 0.0012868141056969762, ,,,]

[0.032991889864206314, 0.0012868141056969762, ,,,]

@cfnathan validator says OK

Unknown User•8mo ago

Message Not Public

schkovich•8mo ago

perfect! that's it. wrangler works as well as multipart request. however, the first example is still returning 400. you might have a bug there. i will review the code in quesion once again. 😃 curl is giving the same error.

curl --request POST   --url https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/vectorize/indexes/hey-hi-index/upsert   --header "Authorization: Bearer ${VECTORIZE_API_TOKEN}"   --header 'Content-Type: application/x-ndjson'   --data test_vector.ndjson
{
  "result": null,
  "success": false,
  "errors": [
    {
      "message": "unexpected - Internal server error"
    }
  ],
  "messages": []
}

curl --request POST   --url https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/vectorize/indexes/hey-hi-index/upsert   --header "Authorization: Bearer ${VECTORIZE_API_TOKEN}"   --header 'Content-Type: application/x-ndjson'   --data test_vector.ndjson
{
  "result": null,
  "success": false,
  "errors": [
    {
      "message": "unexpected - Internal server error"
    }
  ],
  "messages": []
}

Jiamin Wang•8mo ago

Hi, Cloudflare team I encountered a 502 error when accessing the Vectorize Query API using a Singapore IP, but it works fine with a China IP. How can I resolve this? This is the API being used. const url = https://api.cloudflare.com/client/v4/accounts/${ACCOUNT_IDENTIFIER}/vectorize/indexes/${indexName}/query;

Unknown User•8mo ago

Message Not Public

notorious_tom•8mo ago

Does anyone know where I can find OpenAPI spec files for vectorize db?

DaniFoldi•8mo ago

https://developers.cloudflare.com/api/operations/vectorize-list-vectorize-indexes if you go back to https://developers.cloudflare.com/api/ you can export the whole cloudflare openapi spec, if you need that

zensin•8mo ago

All the examples show populating a wrangler.toml with:

[[vectorize]]
binding = "VECTOR_INDEX"
index_name = "vector-index"

[[vectorize]]
binding = "VECTOR_INDEX"
index_name = "vector-index"

and then they show using it in a worker script. Do I have to declare the index in the toml file? What I would instead like to do is dynamically create indexes and let the client specify which one it wants to use (or create) in the request. That way each user can have their own index of whatever. I know I can create indexes using the REST API. However, idk how to then dynamically load them by name in a worker to use them for search.

Unknown User•8mo ago

Message Not Public

Gaming

Programming

Hey guys, I'm try to insert a vector via REST API, but I'm getting ```json { "result": null, "