Running out of vercel resources
Unanswered
Shiny Cowbird posted this in #help-forum
Shiny CowbirdOP
So I have an open source website and its being used by a lot of students from my college and my vercel resources are depleting fast, any other hosting option I can host at for nextjs which is fast and serverless? I dont really wanna spend money on something that doesnt make money
77 Replies
is your nextjs website already cached properly?
what is your website's route setup?
build log?
@alfonsüs ardani is your nextjs website already cached properly?
Shiny CowbirdOP
I get a lot of users actually like 1.5k daily visitors so I guess it’s expected?
It’s a PWA app with only 1 page and 7 endpoints which the user can click
I am running out of fluid CPU so fast because of that 7 endpoints
I scrape and process a lot of data so maybe
\
Route (app)
┌ ○ /
├ ○ /_not-found
├ ○ /~offline
├ ƒ /api/fetchAllGrades
├ ƒ /api/fetchAttendance
├ ƒ /api/fetchExamSchedule
├ ƒ /api/fetchGrades
├ ƒ /api/fetchHostelDetails
├ ƒ /api/fetchMarks
├ ƒ /api/getCaptcha
├ ƒ /api/login
└ ƒ /api/parseSemTT
┌ ○ /
├ ○ /_not-found
├ ○ /~offline
├ ƒ /api/fetchAllGrades
├ ƒ /api/fetchAttendance
├ ƒ /api/fetchExamSchedule
├ ƒ /api/fetchGrades
├ ƒ /api/fetchHostelDetails
├ ƒ /api/fetchMarks
├ ƒ /api/getCaptcha
├ ƒ /api/login
└ ƒ /api/parseSemTT
@Shiny Cowbird I get a lot of users actually like 1.5k daily visitors so I guess it’s expected?
yea but have you cached the functions?
@alfonsüs ardani yea but have you cached the functions?
Shiny CowbirdOP
not sure
Traced Next.js server files in: 43.875ms
Created all serverless functions in: 168.902ms
Collected static files (public/, static/, .next/static): 10.873ms
Build Completed in /vercel/output [22s]
Deploying outputs...
Deployment completed
Creating build cache...
Created build cache: 26.254s
Uploading build cache [267.15 MB]
Build cache uploaded: 5.737sthese are the logs
do I have to manually cache the functions?
@Shiny Cowbird do I have to manually cache the functions?
i guess so? does your data changes very actively? it would be a waste if it scrapes the same data if it doesn't change for few seconds/mimutes/hour i think?
do you have an estimate on how often the data changes?
@alfonsüs ardani i guess so? does your data changes very actively? it would be a waste if it scrapes the same data if it doesn't change for few seconds/mimutes/hour i think?
Shiny CowbirdOP
the button to sync the data is manual an the user presses it on their own, the data prolly only changes once a day but because its manual I expect a user to only click it once in a few days but again I have a big amount of users using it daily
sometimes lets say results/marks came so a lot of users will reload together
sometimes lets say results/marks came so a lot of users will reload together
its all manual and depending on the user
@Shiny Cowbird the button to sync the data is manual an the user presses it on their own, the data prolly only changes once a day but because its manual I expect a user to only click it once in a few days but again I have a big amount of users using it daily
sometimes lets say results/marks came so a lot of users will reload together
yeah so what you can do is just strictly cache the function to only rescrape once per day.
and cache results/marks once every 30 minute or an hour
or 10 minutes even
Shiny CowbirdOP
but I dont know when the data updates, different teachers upload stuff differently so the data for everyone doesnt update together, I save everything in localhost aswell but still thats why I give the manual button
@alfonsüs ardani and cache results/marks once every 30 minute or an hour
Shiny CowbirdOP
I save everything in localhost anyways as they are scraped
then what is the function doing in vercel :/
@alfonsüs ardani then what is the function doing in vercel :/
Shiny CowbirdOP
the manual button calls those API's to scrape different endpoints of a website and get the data
there are 14-15 endpoints I scrape all with different data
so the scraping isn't being done in vercel?
Shiny CowbirdOP
its being done inside the functions
you said you save everything in localhost yet you have users clicking a manual button that hits those endpoint? i am confused sorry
Shiny CowbirdOP
mb localstorage*
i said localhost
had to say localstorage
i dont know how or why thats relevant
try caching it for 15 - 30 mins and see if the usage decreases
Shiny CowbirdOP
I have cached every 30 mins, will see if the usage decreases
@Shiny Cowbird I have cached every 30 mins, will see if the usage decreases
make sure refreshing the endpoint in the browser shows instant result
even at and after the 30th minute
Shiny CowbirdOP
Hmm it isn’t happening instantly
const nextConfig = {
reactStrictMode: true,
devIndicators: false,
images: {
remotePatterns: [
{
protocol: 'https',
hostname: 'avatars.githubusercontent.com',
},
{
protocol: 'https',
hostname: 'cdn.discordapp.com',
},
{
protocol: 'https',
hostname: 'lh3.googleusercontent.com',
},
],
},
async headers() {
return [
{
source: "/api/(.*)",
headers: [
{
key: "Cache-Control",
value: "public, s-maxage=1800, stale-while-revalidate=60",
},
],
},
{
source: "/:path*",
headers: [
{
key: "Cache-Control",
value: "public, s-maxage=1800, stale-while-revalidate=60",
},
],
},
];
},
webpack: (config, { isServer }) => {
if (isServer) {
config.externals.push("@napi-rs/canvas");
}
return config;
},
};This is the only thing we have to add right?
you have to wrap your api with unstable_cache or "use cache" (if next16 with cachedComponent: true)
i am eager to show you how fast it will be. hopefully it will decrease your usage
Shiny CowbirdOP
Ok
I am using next 16 yes
that case you can just write "use cache" on file-top-level of the API endpoint and add
https://nextjs.org/docs/app/api-reference/functions/cacheLife
cacheLife("minutes") (or hours, or custom time with custom profiles)https://nextjs.org/docs/app/api-reference/functions/cacheLife
Shiny CowbirdOP
Kk
Shiny CowbirdOP
Cache Error: Cannot access json on the server. You cannot dot into a temporary client reference from a server component. You can only pass the value through to the client.
at POST (src\app\api\fetchMarks\route.js:13:62)
11 | try {
12 | cacheLife('halfHour');
> 13 | const { cookies, dashboardHtml, campus } = await req.json();
| ^
14 |
15 | const $ = cheerio.load(dashboardHtml);
16 | const cookieHeader = Array.isArray(cookies) ? cookies.join("; ") : cookies; {was getting this, what's the way around this? because well i cannot remove it
@Shiny Cowbird Cache Error: Cannot access json on the server. You cannot dot into a temporary client reference from a server component. You can only pass the value through to the client.
at POST (src\app\api\fetchMarks\route.js:13:62)
11 | try {
12 | cacheLife('halfHour');
> 13 | const { cookies, dashboardHtml, campus } = await req.json();
| ^
14 |
15 | const $ = cheerio.load(dashboardHtml);
16 | const cookieHeader = Array.isArray(cookies) ? cookies.join("; ") : cookies; {
was getting this, what's the way around this? because well i cannot remove it
interesting. there is an API that depends on inocming request?
i thought its just a GET without request data
that way you need to cache only the scraping function not the whole route.
something like
also make sure that cacheLife profile halfHour exists
something like
export async function scrapeSomething() {
"use cache"
cacheLife('halfHour')
const $ = cheerio.load(...)
...
// do not do check auth here. check auth before or after this function.
return data
}also make sure that cacheLife profile halfHour exists
Shiny CowbirdOP
Ya I did that I did define halfHour
I will cache the function ok
@Shiny Cowbird I will cache the function ok
did it work? sorry for pinging
@alfonsüs ardani did it work? sorry for pinging
Shiny CowbirdOP
I did try it and was having some errors, earlier I had all the functions bunched up into 1 button so the things that only changed maybe once per semester were being synced as much as attendance and marks, so now I have only attatched attednance and marks to the reload button and all the other stuff spaced out in their own pages so the user reloads the thing only when they think it might have changed so as to not so much ( everything does reload once on login )
I will do the caching now but this splitting of sync buttons so the useless functions dont sync so often should decrease the usage a lot
was working on the splitting yesterday all day, it just finished now
@Shiny Cowbird So I have an open source website and its being used by a lot of students from my college and my vercel resources are depleting fast, any other hosting option I can host at for nextjs which is fast and serverless? I dont really wanna spend money on something that doesnt make money
Cassin's Kingbird
cloudflare workers should do the job
unless ur hitting 100k requests per day
then the paid tier is just $5 / month
@Cassin's Kingbird cloudflare workers should do the job
Shiny CowbirdOP
I have tried Cloudflare workers but was getting certificate errors which is most prolly because my college website gives invalid certificate sometimes ( idk why don’t ask )
Have opened a ticket in cloudflare’s server regarding if I could bypass that error but no response till now
Cassin's Kingbird
Wdym certificate errors
Like ssl certificates?
@Cassin's Kingbird Wdym certificate errors
Shiny CowbirdOP
failed: TLS peer's certificate is not trusted; reason = unable to get local issuer certificate
I am assuming its because of my college website
Cassin's Kingbird
Well there are many other hosting providers too
Shiny CowbirdOP
otherwise I am not sure
Cassin's Kingbird
Do you have GitHub student pack?
@Cassin's Kingbird Well there are many other hosting providers too
Shiny CowbirdOP
could host in multiple websites and have a loadbalancer accross them ( my friend suggested this )
@Cassin's Kingbird Do you have GitHub student pack?
Shiny CowbirdOP
yes
Cassin's Kingbird
Free for github students
@Shiny Cowbird could host in multiple websites and have a loadbalancer accross them ( my friend suggested this )
Cassin's Kingbird
no u don't need a loadbalancer if your hosting on an edge network
@Cassin's Kingbird Free for github students
Shiny CowbirdOP
kk
will check that out