Next.js Discord

Discord Forum

How to stream openai chat response from backend to frontend.

Answered
Chartreux posted this in #help-forum
Open in Discord
ChartreuxOP
This SSE works in local development but when I am publishing to vercel it gives whole response after the openai SSE is completed. Can someone please help me.

This codes are developed in nextjs
Please help me resolve this

I can provide with the code if needed
Answered by joulev
You may want to use the vercel ai sdk https://sdk.vercel.ai/docs, it helps a lot in simplifying these boilerplate code
View full answer

9 Replies

ChartreuxOP
backend code:
import OpenAI from "openai";

export default async function handler(req, res) {
  if (req.method === "POST") {
    /// parse the request object
    const body = JSON.parse(req.body);

    const { similaritySearchResults, messages, userQuery } = body;

    // Set response headers for SSE
    res.setHeader("Content-Type", "text/event-stream");
    res.setHeader("Cache-Control", "no-cache");
    res.setHeader("Connection", "keep-alive");
    res.setHeader("Content-Encoding", "none");

    // Fetch the response from the OpenAI API
    const openai = new OpenAI({
      apiKey: process.env.NEXT_PUBLIC_OPENAI_KEY,
    });
    const stream = await openai.chat.completions.create({
      model: "gpt-3.5-turbo-16k",
      temperature: 0,
      top_p: 1,
      messages: [
        {
          role: "system",
          content: `Use the following pieces of context to answer the users question.
                  If you don't know the answer, just say that you don't know, don't try to make up an answer.
                  ----------------
                  context:
                  ${similaritySearchResults}
    
                  Answer user query and include images write respect to each line if available`,
        },
        ...messages,
        {
          role: "user",
          content: `
                    Strictly write all the response in html format with only raw text and img tags.
                    Answer user query and include images in response if available in the given context
    
                    query: ${userQuery}`,
        },
      ],
      stream: true,
    });
    for await (const part of stream) {
      res.write(part.choices[0]?.delta?.content || "");
    }

    res.end();
  }
}
front-end code:
 const responseFromBackend: any = await fetch(
            `${process.env.NEXT_PUBLIC_WEBSITE_URL}api/chat`,
            {
              method: "POST",
              body: JSON.stringify({
                similaritySearchResults,
                messages,
                userQuery,
              }),
              headers: {
                "Content-Type": "text/event-stream",
              },
            }
          );
          let resptext = "";

          const reader = responseFromBackend.body
            .pipeThrough(new TextDecoderStream())
            .getReader();
          while (true) {
            const { value, done } = await reader.read();
            if (done) {
              /// setting the response when completed
              setMessages((prev: any) => [
                ...prev,
                { role: "assistant", content: resptext },
              ]);
              /// store the chathistory
              setResponse("");
              setLoading(false);
              break;
            }

            resptext += value;
            setResponse(resptext);
          }
Answer
No, it’s free and open source
It’s just a normal js library
@joulev It’s just a normal js library
ChartreuxOP
Okay thank you will let you know once I resolve the issue through this
@joulev No, it’s free and open source
ChartreuxOP
Thank you it is done via ai sdk
Californian
Good