Thread on Workers AI onboarding for a

Thread on Workers AI onboarding for a new user 🧵

3 Replies

JamesOP•8mo ago

I just put together this worker after following the docs:

import { Ai } from '@cloudflare/ai'

interface Env {
    AI: Fetcher;
}

export default {
    async fetch(request: Request, env: Env, ctx: ExecutionContext) {
        const ai = new Ai(env.AI, { sessionOptions: { ctx }});

        const number = Math.random();
        const response = await ai.run('@cf/mistral/mistral-7b-instruct-v0.1', {
            prompt: `I have ${number} apples. How many apples do I have?"`,
            stream: true
        });

        return new Response(response);
    },
};

import { Ai } from '@cloudflare/ai'

interface Env {
    AI: Fetcher;
}

export default {
    async fetch(request: Request, env: Env, ctx: ExecutionContext) {
        const ai = new Ai(env.AI, { sessionOptions: { ctx }});

        const number = Math.random();
        const response = await ai.run('@cf/mistral/mistral-7b-instruct-v0.1', {
            prompt: `I have ${number} apples. How many apples do I have?"`,
            stream: true
        });

        return new Response(response);
    },
};

From the docs here: https://developers.cloudflare.com/workers-ai/configuration/workers-ai-sdk/#async-airun Immediately, I am hit with two type errors: -

Argument of type 'AiTextGenerationOutput | ReadableStream<any>' is not assignable to parameter of type 'BodyInit | null | undefined'.  Type '{ response?: string | undefined; }' is not assignable to type 'BodyInit | null | undefined'.

- Object literal may only specify known properties, and 'sessionOptions' does not exist in type 'AiOptions'. sessionOptions appears to not be a thing anymore, since it looks like 1.1.0, released almost a month ago. And the other error seems to just be bad types/code in the docs. My code "works" (https://hello-ai.jross.workers.dev/), but my DX here is super rough. I then also saw a few random 503s when developing, which went away after a refresh. Frankly, at this point, as a new user, I'm kinda done. Type errors, random response failures, docs not matching reality. This does not feel like it deserves a "GA" title right now. cc @celso @michelle @Logan Grasby @Craig

Victor•8mo ago

I also want to chime in that the types for image-gen specify a UInt8Array being returned, but in some cases, it seems to be the raw ReadableStream<UInt8Array> from the underlying Fetcher that is used. The docs show it being returned immediately inside a Response object which can take either without any issue or error. But it is important if you want to do any other work that directly re-stream it back out the worker. My 2 cents would be to carry over the stream option from text-gen for instances where you return directly a ReadableStream to remove the buffering (if desired)?

Craig•8mo ago

Thank you for the feedback @James , looking into these! And you as well @Victor @DemosJarco | Chainfuse

Gaming

Programming

Thread on Workers AI onboarding for a