Streaming data back to frontend with res.write() not working in Vercel deployment.
Unanswered
Pacific anchoveta posted this in #help-forum
Pacific anchovetaOP
I'm trying to stream GPT-4 chat completions from my chat api file to the frontend component, this works as expected when deployed locally. I'm guessing this has to do with Vercel's serverless architecture but I'm not sure how I should refactor my code without a total rewrite using Vercel dependancies. Below is a link to the file. Response does come through once res.end() is called but I'd like the response be written to the front end as the stream comes in.
https://github.com/jearthman/homeworker/blob/24b65f38aac0234ac64263de2a548113c838ec59/pages/api/chat.js
https://github.com/jearthman/homeworker/blob/24b65f38aac0234ac64263de2a548113c838ec59/pages/api/chat.js
11 Replies
Vercel AI SDK on Edge Runtime(Cloudflare Workers) is easy-breezy. see my poc project.
https://github.com/tfutada/zenn-vercel-ai-sdk/blob/main/app/api-openai/route.ts
https://github.com/tfutada/zenn-vercel-ai-sdk/blob/main/app/api-openai/route.ts
or it is worth noting that Cloudflare just released AI solutions. I have yet to try it.
Pyramid ant
Edge not compatible w/ Firebase or Pinecone or many other node/non-edge based packages for example auth, db, exec. There must be a better solution.
American Crow
pretty sure Vercel AI SDK works on both edge and node runtime
@American Crow pretty sure Vercel AI SDK works on both edge and node runtime
Pyramid ant
you have to return the stream, cant do any post processing or extract stream or anything else, correct me if im wrong, but its kind of useless except for small demo applications
@Pyramid ant you have to return the stream, cant do any post processing or extract stream or anything else, correct me if im wrong, but its kind of useless except for small demo applications
American Crow
I just did my first project with it. I feel like i can do everything by streaming ui. I'd think i am doing a little post processing. But it's a bit vague i don't know what you are trying to achieve
@American Crow I just did my first project with it. I feel like i can do everything by streaming ui. I'd think i am doing a little post processing. But it's a bit vague i don't know what you are trying to achieve
Pyramid ant
How did you do it? After streaming did you continue processing extra code?
@Pyramid ant How did you do it? After streaming did you continue processing extra code?
American Crow
Okay now i see our misunderstanding:
I am post processing after the LLM response but before the UI is streamed in.
You are saying after the ui is streamt.
I am not doing any processing after that point you are right
I am post processing after the LLM response but before the UI is streamed in.
You are saying after the ui is streamt.
I am not doing any processing after that point you are right
@American Crow Okay now i see our misunderstanding:
I am post processing after the LLM response but before the UI is streamed in.
You are saying after the ui is streamt.
I am not doing any processing after that point you are right
Pyramid ant
How about server side? any post processing code serverside? or using vercel ai sdk serverside but custom code client side?
American Crow
I am doing nothing special just working with tools and doing something like:
.then((response) => {
const tool = response.toolCalls.find(
(toolCall) => toolCall.toolName === "printNames"
)
if (!tool) {
history.done([])
throw new Error("Tool printNames was not called")
}
// ******* Tool was successfully called **********
const { names } = tool.args
// post processing possible here
// Add the LLM response to the history
history.done((currentHistory: AIState[]) => [
...currentHistory,
{
role: "assistant",
content: names.join(", "),
},
])
// Update the UI Stream
uiStream.update(<Results names={names} />)
})
.catch((error) => {
console.error("🚀 ~ error:", error)
})
.finally(() => {
uiStream.done()
})