jeromemarshall Posts - Answer Overflow

jeromemarshall

•Created by jeromemarshall on 5/15/2023 in #questions

How to Stream responses when using Langchain in the new route handlers

Im trying to build a ChatGPT app using langchain, but I cant figure out how to stream the responses to the client. This is how I have initialized openai with langchain

import { ChatOpenAI } from "langchain/chat_models/openai";
import { HumanChatMessage } from "langchain/schema";

const chat = new ChatOpenAI({
  streaming: true,
  callbacks: [
    {
      handleLLMNewToken(token: string) {
        process.stdout.write(token); // how to stream this response??
      },
    },
  ],
});

await chat.call([
  new HumanChatMessage("Write me a song about sparkling water."),
]);

import { ChatOpenAI } from "langchain/chat_models/openai";
import { HumanChatMessage } from "langchain/schema";

const chat = new ChatOpenAI({
  streaming: true,
  callbacks: [
    {
      handleLLMNewToken(token: string) {
        process.stdout.write(token); // how to stream this response??
      },
    },
  ],
});

await chat.call([
  new HumanChatMessage("Write me a song about sparkling water."),
]);

I can get the stream from the handleLLMNewToken callback. handleLLMNewToken returns a string for every token(word) generated. Can someone please help me on how to stream this to the client using the new route handlers?

1 replies

TTCTheo's Typesafe Cult

•Created by jeromemarshall on 5/8/2023 in #questions

Acessing a route api in app dir causes re-render loop.

okay, I'm just trying out the next js app routes. I'm getting a render loop when using it like this in /app/page.tsx:

"use client"

const getData = async () => {
  const response = await fetch("http://localhost:3000/api/generate", {
    method: "POST",
  })
  const data = await response.json()
  return data
}

export default async function Home() {
  const data = await getData()

  return (
    <main>
      page
    </main>
  )
}

"use client"

const getData = async () => {
  const response = await fetch("http://localhost:3000/api/generate", {
    method: "POST",
  })
  const data = await response.json()
  return data
}

export default async function Home() {
  const data = await getData()

  return (
    <main>
      page
    </main>
  )
}

Can someone please explain why this is happening and how to fix this? Thanks

4 replies

TTCTheo's Typesafe Cult

•Created by jeromemarshall on 3/23/2023 in #questions

How to skip batching for a single trpc query in trpc v10?

I want to skip batching for only one query. But I am not able to figure out how to do that. As in the trpc v 10 docs I can see that we can completely skip batching, but I can't figure out how to skip batching for a single query. In trpc v9 we can do it like this const postsQuery = trpc.useQuery(['posts'], { context: { skipBatch: true, }, }); Can someone please guide me in the right direction please?

3 replies

TTCTheo's Typesafe Cult

•Created by jeromemarshall on 3/21/2023 in #questions

Next-auth useSession throws an exception in prod. Please help

7 replies

TTCTheo's Typesafe Cult

•Created by jeromemarshall on 3/8/2023 in #questions

How to use react toastify to show a promise toast when using a mutation in trpc?

I'm trying to show a promise toast but cant figure out how to do it. Can someone please help. Thanks

26 replies

Gaming

Programming