VERY slow api route in production Vercel.
Answered
King Rail posted this in #help-forum
King RailOP
I have an api route that in development usually takes a few seconds to complete its function. It first calls an api, and then calls the openAI api, that is why it takes a bit to complete. But when the app is built and hosted on Vercel, the route takes 10-20x longer to complete. There is no difference in code, env variables or anything like that.
Using APP directory.
I have tried redeploying the app a few times on Vercel. Does the app have to "boot up" the first few times the route is called, before the route starts to have the same performance as development?
Does anyone know what could be going on?
Using APP directory.
I have tried redeploying the app a few times on Vercel. Does the app have to "boot up" the first few times the route is called, before the route starts to have the same performance as development?
Does anyone know what could be going on?
Answered by tafutada777
@King Rail you might want to use Vercel AI SDK with Edge Runtime, which is no cold start time, V8 Isolate technology from Cloudflare Worker.
https://vercel.com/blog/introducing-the-vercel-ai-sdk
Serverless, AWS lambda, use containers, so there are down sides like cold start time, wall time(elapsed time) measure, which is not good for I/O bound services like OpenAI API.
https://vercel.com/blog/introducing-the-vercel-ai-sdk
Serverless, AWS lambda, use containers, so there are down sides like cold start time, wall time(elapsed time) measure, which is not good for I/O bound services like OpenAI API.
32 Replies
@King Rail you might want to use Vercel AI SDK with Edge Runtime, which is no cold start time, V8 Isolate technology from Cloudflare Worker.
https://vercel.com/blog/introducing-the-vercel-ai-sdk
Serverless, AWS lambda, use containers, so there are down sides like cold start time, wall time(elapsed time) measure, which is not good for I/O bound services like OpenAI API.
https://vercel.com/blog/introducing-the-vercel-ai-sdk
Serverless, AWS lambda, use containers, so there are down sides like cold start time, wall time(elapsed time) measure, which is not good for I/O bound services like OpenAI API.
Answer
@tafutada777 <@347790860521570305> you might want to use Vercel AI SDK with Edge Runtime, which is no cold start time, V8 Isolate technology from Cloudflare Worker.
https://vercel.com/blog/introducing-the-vercel-ai-sdk
Serverless, AWS lambda, use containers, so there are down sides like cold start time, wall time(elapsed time) measure, which is not good for I/O bound services like OpenAI API.
King RailOP
I will take a look.
But i don't understand why this would make any difference.
Since i am only calling the api to open AI i am not really using any computing power to do anything, i am just waiting for a response.
Could this fix it anyways?
But i don't understand why this would make any difference.
Since i am only calling the api to open AI i am not really using any computing power to do anything, i am just waiting for a response.
Could this fix it anyways?
are u really sure it is faster when running local on ur mac/win compared to running on Vercel Serverless? if so it might have something to do with AWS lambda streaming stuff. Vercel recommends to run on Cloudflare edge runtime, so worth giving it try. plus visit OpenAI Discord server, then ask there.
King RailOP
I am very sure it is slower on Vercel.
I will try the sdk.
Give me a bit then I will report back.
I will try the sdk.
Give me a bit then I will report back.
Giant panda
Serverless API routes on Vercel are affected by cold starts as suggested by somebody earlier.
@Giant panda Serverless API routes on Vercel are affected by cold starts as suggested by somebody earlier.
King RailOP
Makes sense.
I am trying to convert to an edge function but I am running into some problems.
In my original serverless route i trigger a function right before the response is sent. This function is the one that calls the first api, and then openAI.
It works fine with serverless, but it seems that the function isn't run in the edge function.
This is the code:
Is there something that needs to be changed?
I am trying to convert to an edge function but I am running into some problems.
In my original serverless route i trigger a function right before the response is sent. This function is the one that calls the first api, and then openAI.
It works fine with serverless, but it seems that the function isn't run in the edge function.
This is the code:
generate(user, itemsToGenerate, saveName, saveId)
return NextResponse.json(
{
message:
"Success.",
saveId: saveId
},
{
status: 200,
headers: {
"content-type": "application/json",
},
}
)Is there something that needs to be changed?
Giant panda
Is
generate returning a promise?The thing is that there is no guarantee how the execution of code behaves once you return a response
So if there is something async still running it may finish or not
King RailOP
No, it does not return anything.
I have done it in this way because depending on the user input the function might take a few minutes to complete (when running "fast" on localhost). The first thing the function does is to save something to the database, but that is not being done currently in edge when hosted on Vercel. Ones the openAI calls are done, it will update the original database save with the text from the ai.
As mentioned it has worked fine before changing to edge, but when testing on vercel it doesn't seem to work.
I have done it in this way because depending on the user input the function might take a few minutes to complete (when running "fast" on localhost). The first thing the function does is to save something to the database, but that is not being done currently in edge when hosted on Vercel. Ones the openAI calls are done, it will update the original database save with the text from the ai.
As mentioned it has worked fine before changing to edge, but when testing on vercel it doesn't seem to work.
Giant panda
Your expectation about the behavior is wrong - Vercel has no guarantees for long running tasks
Especially when you exit early out of execution
(In addition to the time limit of 60s per function invocation)
King RailOP
What would be the ideal way of doing it? A lambda function?
Giant panda
A separate long running function on a different host, yes
Either a stateful version on Fly or Railway or something more common on AWS that doesn't interrupt
Which you trigger via an API call for example
King RailOP
Cloudflare was mentioned, would that be a better choice for this use or is it irrelevant?
Vercel uses Clouldflare worker under the hood for edge runtime.
Vercel has two runtimes: Edge Runtime and Serverless. The former is Cloudflare worker, the latter is AWS lambda behind the scenes.
plus OpenAI SDK does not work in Edge Runtime as it uses axios.
Thus Vercel created Vercel AI SDK to work in Edge Runtime. just git it a try.
Thus Vercel created Vercel AI SDK to work in Edge Runtime. just git it a try.
Edge Runtime implements subset of Node.js, and it works like tabs in a Chrome. so no cold start.
and Edge Runtime is measured by CPU time, while AWS lambda is walltime(elapsed time), so I/O bound tasks can run so long in Edge Runtime as it won't consume CPU.
plus edge is located in closely your home, while AWS lambda is a specifi region like US east.
King RailOP
But if vercel has the 60s limit on functions, I would have to move the aws, cloudflare or something like that? Or am i misunderstanding.
As the function might exceed 60s depending on user input
60s is cap on AWS, wall time(elapased time.
Edge is 50ms CPU time cap.
Edge is 50ms CPU time cap.
so it could run more than 60s in Edge. im not sure though.
King RailOP
Idk, I have already had functions on AWS Lambda with +60s run time, so I will just move the code to that.
Hopefully that will fix the slow execution.
Hopefully that will fix the slow execution.
sure
King RailOP
Thank you for the massive amount of help to both of you!
actually it is a brand new tech. and AWS lambda supports streaming recently, so Vercel could still be working on integrating AWS and Cloudflare streaming.