Avoid Initial Load on ISR pages
Unanswered
British Shorthair posted this in #help-forum
British ShorthairOP
Hello, I have a NextJS project hosted on Vercel with 600k+ pages that uses ISR.
Once the page has been visited by a user, the next page load is quite fast and great. However, each time I create a new deployment, my ISR pages are no longer cached and hit a fairly abysmal build time. (4-5seconds). How can I avoid this build time?
From what I've read 600k pages will be too many to generate at build time with getStaticPaths due to 45min timeout in Vercel but I have not tried.
Possible solutions I've pondered:
1) Is it possible to serve up the page from that last deployment when the page is in "fallback: true" until the page on new deployment has been built? That way I do not hit that initial build time.
2) Can I build all 600k pages statically then upload them to Vercel somehow? I have Vercel linked to my GitHub repo which triggers a Deployment when new code is pushed. I'm quite a noob in how this works to be honest.
3) Can I build a bot of some type that would visit each page to force the page to be built after a deployment?
Please let me know if this should be forwarded to a different help section like Vercel support.
Thanks.
Once the page has been visited by a user, the next page load is quite fast and great. However, each time I create a new deployment, my ISR pages are no longer cached and hit a fairly abysmal build time. (4-5seconds). How can I avoid this build time?
From what I've read 600k pages will be too many to generate at build time with getStaticPaths due to 45min timeout in Vercel but I have not tried.
Possible solutions I've pondered:
1) Is it possible to serve up the page from that last deployment when the page is in "fallback: true" until the page on new deployment has been built? That way I do not hit that initial build time.
2) Can I build all 600k pages statically then upload them to Vercel somehow? I have Vercel linked to my GitHub repo which triggers a Deployment when new code is pushed. I'm quite a noob in how this works to be honest.
3) Can I build a bot of some type that would visit each page to force the page to be built after a deployment?
Please let me know if this should be forwarded to a different help section like Vercel support.
Thanks.
39 Replies
British ShorthairOP
bump
2 is totally feasible
Use the vercel cli
Use it create a build and then upload the artifacts to vercel
But how successful this approach will be
Depends on how much power your machine has that the pages are building built on
British ShorthairOP
Thanks for feedback. I'll give that a go!
British ShorthairOP
I was able to get the pages to build however I ran into new problem.
Note this is trying to prebuild 700k pages.
The .next/prerender-manifest.js and .json files are too large.
Note this is trying to prebuild 700k pages.
The .next/prerender-manifest.js and .json files are too large.
Warning: Max serverless function size of 52.4 MB compressed or 262 MB uncompressed reached
Serverless Function's pages:
Large Dependencies Uncompressed size Compressed size
.next/prerender-manifest.js 131.03 MB 11.27 MB
.next/prerender-manifest.json 123.16 MB 11.15 MB
node_modules/next/dist 8.21 MB 2.32 MB
node_modules/react-dom/cjs 1.64 MB 404.02 KB
node_modules/caniuse-lite/data 939.22 KB 333.99 KB
All dependencies 265.58 MB 25.65 MB
Error: Required files read using Node.js fs library and node_modules exceed max lambda size of 52428800 bytes
Learn More: https://vercel.com/docs/platform/limits#serverless-function-size@linesofcode Any ideas?
Wow
I’ve seen that error before but never because the json files are too big
@British Shorthair
1. open an issue in the nextjs repo and start a discussion in the vercel repo. This is highly suspcious. I understand that loading large npm packages will cause lambda sizes to exceed in size, but the prender manifests should ideally not cause this. I'm not familiar with the internals of the build system to say for certain so ask a professional
2. 700k pages is A LOT. Do all of those need to be prebuild or can some them be built on demand?
1. open an issue in the nextjs repo and start a discussion in the vercel repo. This is highly suspcious. I understand that loading large npm packages will cause lambda sizes to exceed in size, but the prender manifests should ideally not cause this. I'm not familiar with the internals of the build system to say for certain so ask a professional
2. 700k pages is A LOT. Do all of those need to be prebuild or can some them be built on demand?
a common pattern is to only build the most popular pages and let the rest be built as the user navigates to them
British ShorthairOP
Thanks for your response.
Not all 700k pages need to be prebuilt but we were running into some Core Web Vitals issues which I believe is due to ISR page's first load (core web vitals perfect after). So I wanted to just prebuild all the pages to avoid this issue entirely.
I'll open an issue on nextjs repo to see if anyone has any ideas.
Not all 700k pages need to be prebuilt but we were running into some Core Web Vitals issues which I believe is due to ISR page's first load (core web vitals perfect after). So I wanted to just prebuild all the pages to avoid this issue entirely.
I'll open an issue on nextjs repo to see if anyone has any ideas.
Do you mind linking me the issue after you make it so I could follow it?
I'm curious in this as well
"no longer cached " < what happend?
solution 3) in your post is roughly equivalent to a static render
that's how you would do in frameworks that do not have built-in static render, using cache-control headers on the response
but not very "Nextish"
2) seems to hit Node limitations, but that's worth opening an issue
building 700K page may mean something is off, but I would expect a bigger limit like at least a million or 2 before it crashes
after all it's your right to want many pages and it's not really an absurd amount
Vercel (or whatever host you pick) can choose to accept that or not though, I would expect the build to timeout instead of a size issue
so I feel like ISR is the right approach for you but you have issues with it?
3 would also incur bandwidth costs
If the bot visits all 700k pages
@Eric Burel "no longer cached " < what happend?
British ShorthairOP
"no longer cached" due to new deployments invalidating cache.
@Eric Burel so I feel like ISR is the right approach for you but you have issues with it?
British ShorthairOP
ISR seems good but my Google Search Console has almost half the pages failing on Core Web Vitals.
When testing these pages with LightHouse, Core Web Vitals Extension, https://www.webpagetest.org/, they all come back with greens across the board.
So I'm assuming that the first page load before ISR page is cached is causing this issue.
When testing these pages with LightHouse, Core Web Vitals Extension, https://www.webpagetest.org/, they all come back with greens across the board.
So I'm assuming that the first page load before ISR page is cached is causing this issue.
@British Shorthair "no longer cached" due to new deployments invalidating cache.
Ok I get your issue, I think you'd want to explore ways to reduce the prerender-manifest size
I don't know this file, what does it contain? Is it really needed (you could try locally to see what happens)?
You could open a ticket about that, probably more on Next side than Vercel side
maybe its size could be limited or somth
you are not too above the limit, so you could try at least to increase the number of pages that are SSRed
do you have an approximation of the traffic for each page?
you could also reach out to Vercel to see if you can increase this limit
if you have a good reason to prebuild each and everyu of the 700k page eg access is truly random and you want the best perfs on every case (say you sell lot of products on a very competitive market)
the bot trick would work too as a last resort but indeed it might not be free if you run it too often