Cloudflare Developers•13mo ago

``` curl --location 'https://api.cloudflare.com/client/v4/accounts/{my-account}/ai/run/@cf/mistral/m

curl --location 'https://api.cloudflare.com/client/v4/accounts/{my-account}/ai/run/@cf/mistral/mistral-7b-instruct-v0.1' \
--header 'Authorization: Bearer {mytoken}' \
--header 'Content-Type: application/json' \
--data '{ 
    "prompt": "write me a poem about a dog"
}'

curl --location 'https://api.cloudflare.com/client/v4/accounts/{my-account}/ai/run/@cf/mistral/mistral-7b-instruct-v0.1' \
--header 'Authorization: Bearer {mytoken}' \
--header 'Content-Type: application/json' \
--data '{ 
    "prompt": "write me a poem about a dog"
}'

16 Replies

Apollo•13mo ago

Hey there, I would love to use Cloudflare Workers to do NSFW image checks. Stable Diffusion has the model CompVis/stable-diffusion-safety-checker[1] which is apart of their text 2 img pipeline. I have it stood up on SageMaker, but I really love to use it in CF workers. 1: https://huggingface.co/CompVis/stable-diffusion-safety-checker

CompVis/stable-diffusion-safety-checker · Hugging Face

Swell•13mo ago

Hi all, where can I find information about pricing for using workers AI, and different LLMs? Can't find it anywhere in cloudflare's docs, or my dashboard

Erisa•13mo ago

Workers AI is free during beta with the limits listed here https://developers.cloudflare.com/workers-ai/platform/limits/ Pricing is just an early plan at the moment, mentioned on https://blog.cloudflare.com/workers-ai/

Swell•13mo ago

Okay, thank you.

chand1012•13mo ago

Is this code incorrect? I'm having issues with errors being spit out. Here is the code I'm using to call Workers AI.

import { Ai } from '@cloudflare/ai';

export const testHandler = async (request, env) => {
    const messages = [
        {
            role: 'system',
            content:
                'You are a helpful assistant who identifies languages. Only output one of the following and nothing else: english, chinese, french, spanish, arabic, russian, german, japanese, portuguese, hindi',
        },
        { role: 'user', content: 'Hola mi nombre es brian y el tuyo?' },
        { role: 'assistant', content: 'spanish' },
        { role: 'user', content: 'Was für ein schönes Baby!' },
        { role: 'assistant', content: 'german' },
        { role: 'user', content: 'Ma voiture est en panne.' },
    ];

    const model = '@cf/meta/llama-2-7b-chat-int8';

    const ai = new Ai(env.AI);

    const stream = await ai.run(model, { messages, stream: true });

    return new Response(stream, {
        headers: {
            'content-type': 'text/event-stream',
        },
    });
};

import { Ai } from '@cloudflare/ai';

export const testHandler = async (request, env) => {
    const messages = [
        {
            role: 'system',
            content:
                'You are a helpful assistant who identifies languages. Only output one of the following and nothing else: english, chinese, french, spanish, arabic, russian, german, japanese, portuguese, hindi',
        },
        { role: 'user', content: 'Hola mi nombre es brian y el tuyo?' },
        { role: 'assistant', content: 'spanish' },
        { role: 'user', content: 'Was für ein schönes Baby!' },
        { role: 'assistant', content: 'german' },
        { role: 'user', content: 'Ma voiture est en panne.' },
    ];

    const model = '@cf/meta/llama-2-7b-chat-int8';

    const ai = new Ai(env.AI);

    const stream = await ai.run(model, { messages, stream: true });

    return new Response(stream, {
        headers: {
            'content-type': 'text/event-stream',
        },
    });
};

I get the following error in the logs:

GET https://my.account.workers.dev/test - Exception Thrown @ 12/6/2023, 12:40:16 PM
✘ [ERROR]   TypeError: Inter-TransformStream ReadableStream.pipeTo() is not implemented.


POST http://workers-binding.ai/run - Canceled @ 12/6/2023, 12:40:36 PM

GET https://my.account.workers.dev/test - Exception Thrown @ 12/6/2023, 12:40:16 PM
✘ [ERROR]   TypeError: Inter-TransformStream ReadableStream.pipeTo() is not implemented.


POST http://workers-binding.ai/run - Canceled @ 12/6/2023, 12:40:36 PM

Is there something I'm missing? I thought this was pretty close to what was in the docs. https://developers.cloudflare.com/workers-ai/models/text-generation/#using-streaming

Text Generation · Cloudflare Workers AI docs

Family of generative text models, such as large language models (LLM), that can be adapted for a variety of natural language tasks.

Rob M.•13mo ago

Code looks fine to me. But it doesn't look like a regular worker. I would try to update deps, create a basic worker with your AI code and deploy it to the prod. Then adapt to your needs. I used this guide and it worked great: https://developers.cloudflare.com/workers-ai/get-started/workers-wrangler/

chand1012•13mo ago

I should have mentioned, its using Itty-Router. The actual worker index can be found here: https://github.com/chand1012/openai-cf-workers-ai/blob/assistants/index.js

GitHub

openai-cf-workers-ai/index.js at assistants · chand1012/openai-cf-w...

Replacing OpenAI's API with Cloudflare AI. Contribute to chand1012/openai-cf-workers-ai development by creating an account on GitHub.

Rob M.•13mo ago

TinyRouter or itty-router?

chand1012•13mo ago

Itty-router, fixed the original message

Rob M.•13mo ago

Does non-streaming flow work fine?

chand1012•13mo ago

Yeah non streaming is fine, but I want to be able to support streaming on the API as well as take advantage of the higher token limits that streaming allows

Rob M.•13mo ago

Looks like some of your middleware is using ReadableStream.pipeTo() which is not implemented in workers runtime, according to this post: https://community.cloudflare.com/t/running-into-unimplemented-functionality/77343/3

chand1012•13mo ago

Which middleware you referring to? I just pushed the above test endpoint to my assistants branch so its all there. Is there something within itty-router that's doing something its not supposed to?

chand1012•13mo ago

Test file for reference https://github.com/chand1012/openai-cf-workers-ai/blob/assistants/routes/test.js

GitHub

openai-cf-workers-ai/routes/test.js at assistants · chand1012/opena...

Replacing OpenAI's API with Cloudflare AI. Contribute to chand1012/openai-cf-workers-ai development by creating an account on GitHub.

Rob M.•13mo ago

I think so. I've tested your Test snippet with my basic worker and it works just fine. I see that your only additional dep. is itty-router, so I would try to comment it out and test. also, check your "@Cloudflare/ai" version. It's stale in your git repo. "@Cloudflare/ai": "^1.0.15"

chand1012•13mo ago

On the assistants branch its updated Haven't merged to main since the features aren't done yet

Gaming

Programming

``` curl --location 'https://api.cloudflare.com/client/v4/accounts/{my-account}/ai/run/@cf/mistral/m