How to stream openai chat response from backend to frontend.
Answered
Chartreux posted this in #help-forum
ChartreuxOP
This SSE works in local development but when I am publishing to vercel it gives whole response after the openai SSE is completed. Can someone please help me.
This codes are developed in nextjs
Please help me resolve this
I can provide with the code if needed
This codes are developed in nextjs
Please help me resolve this
I can provide with the code if needed
Answered by joulev
You may want to use the vercel ai sdk https://sdk.vercel.ai/docs, it helps a lot in simplifying these boilerplate code
9 Replies
ChartreuxOP
backend code:
import OpenAI from "openai";
export default async function handler(req, res) {
if (req.method === "POST") {
/// parse the request object
const body = JSON.parse(req.body);
const { similaritySearchResults, messages, userQuery } = body;
// Set response headers for SSE
res.setHeader("Content-Type", "text/event-stream");
res.setHeader("Cache-Control", "no-cache");
res.setHeader("Connection", "keep-alive");
res.setHeader("Content-Encoding", "none");
// Fetch the response from the OpenAI API
const openai = new OpenAI({
apiKey: process.env.NEXT_PUBLIC_OPENAI_KEY,
});
const stream = await openai.chat.completions.create({
model: "gpt-3.5-turbo-16k",
temperature: 0,
top_p: 1,
messages: [
{
role: "system",
content: `Use the following pieces of context to answer the users question.
If you don't know the answer, just say that you don't know, don't try to make up an answer.
----------------
context:
${similaritySearchResults}
Answer user query and include images write respect to each line if available`,
},
...messages,
{
role: "user",
content: `
Strictly write all the response in html format with only raw text and img tags.
Answer user query and include images in response if available in the given context
query: ${userQuery}`,
},
],
stream: true,
});
for await (const part of stream) {
res.write(part.choices[0]?.delta?.content || "");
}
res.end();
}
}front-end code:
const responseFromBackend: any = await fetch(
`${process.env.NEXT_PUBLIC_WEBSITE_URL}api/chat`,
{
method: "POST",
body: JSON.stringify({
similaritySearchResults,
messages,
userQuery,
}),
headers: {
"Content-Type": "text/event-stream",
},
}
);
let resptext = "";
const reader = responseFromBackend.body
.pipeThrough(new TextDecoderStream())
.getReader();
while (true) {
const { value, done } = await reader.read();
if (done) {
/// setting the response when completed
setMessages((prev: any) => [
...prev,
{ role: "assistant", content: resptext },
]);
/// store the chathistory
setResponse("");
setLoading(false);
break;
}
resptext += value;
setResponse(resptext);
}@Chartreux This SSE works in local development but when I am publishing to vercel it gives whole response after the openai SSE is completed. Can someone please help me.
**This codes are developed in nextjs**
Please help me resolve this
I can provide with the code if needed
You may want to use the vercel ai sdk https://sdk.vercel.ai/docs, it helps a lot in simplifying these boilerplate code
Answer
@joulev You may want to use the vercel ai sdk <https://sdk.vercel.ai/docs>, it helps a *lot* in simplifying these boilerplate code
ChartreuxOP
does it take any extra cost ?
No, it’s free and open source
It’s just a normal js library
@joulev It’s just a normal js library
ChartreuxOP
Okay thank you will let you know once I resolve the issue through this
@joulev No, it’s free and open source
ChartreuxOP
Thank you it is done via ai sdk
Californian
Good