Note: That I really want to build once and run everywhere with docker - so ideally I wouldn't be recompiling the app at all after its PR stage. I'm really looking for a runtime solution.

Unfortunately that may against Next.js’s cache. Normally your app will be bundled and pre-rendered during build, including some fetch requests are cached too. If you want to dynamically change these environment variables, you can use some methods (fs read file, fetch from local host, etc) + revalidate to achieve this.
I will recommend Vercel Edge config if you are hosting the project on Vercel, it’s great

Notice that it may turn your SSG page into SSR (rendered at request time), since it cannot be built statically due to environment variables

Thanks. But yes exactly we want the benefits of SSG and can't find a way to achieve this without a complete rebuild per deploy - which is madness.

Does Vercel Edge config use some magic that's not available to us self-hosting mere mortals do you think? Or are they rebuilding the whole app every time a single config var changes?

No, edge config isn’t a magic. It’s a global data store available for edge/serverless runtime, which replaces the need for an external database.
Pages can only generated during build time, so you might have to switch to ISR/SSR.

Normally ISR is the solution, it’s flexible enough for your use case

With on-demand revalidation, you can revalidate specific pages and routes.

Vercel uses a CI/CD pipeline, so it actually does rebuild for each "deployment". I suspect you want something similar, like a CI/CD that knows your production environment variables and ships the build artifacts to your docker containers

that way it would still only run the build a minimum number of times '

I think the OP is trying to change environment variables dynamically, in this case, though I don’t recommend skipping the build process, but it’s theoretically possible to achieve this via ISR

That is not what I am reading from the question

In any case, to emulate what Vercel does, you would just ship a project folder with the .next folder from a next build that had the production environment variables. https://nextjs.org/docs/app/building-your-application/deploying

Thanks. I'm really just trying to avoid rebuilding the app for every environment as it goes against the whole build-an-image once and ship to multiple environments with different environment variables that's standard for just about every other kind of app

So variables won't change in real time per environment, but do vary per environment. At the moment I have to wait 5 minutes for my containers to come up while the build completes.

Additionally if a container gets killed by Kubernetes for any reason instead of a fast restart i'm left with 5 minutes of downtime while the container starts and the app rebuilds

So pre-building sounds like the way to go, but this is a hell of a lot of work and I wish Next just supported what I see as an industry standard here

Next.js generates pages in build time, one of the drawbacks is that it doesn't allow changing environment variables without re-building the app.
It creates a .next folder which also includes some cache that can speed up builds, however, it will be around ~1 minute anyway.
it's clear that it's impossible without using ISR with on-demand revalidation. (changing environment variables in real-time)

If your container is killed for an unknown reason, you can run next start without re-building it. (This uses .next folder directly)

Generating pages in build time does help with performance and initial load speed, I don't think it will be supported anytime soon

That'd require some kind of persistence storage on each container right? Again not great on immutable infrastructure

I'm looking into https://github.com/andrewmclagan/react-env but not sure how it'll work with SSG

You have to combine it with ISR, such as on-demand revalidation since Next.js generated the page already.

Next.js itself also persists a cache, it's safe to re-use cache from .next as this can decrease build time.
It doesn't make sense to drop build cache everytime

I agree it doesn't make sense when you have to rebuild on every deploy 🤦🏼‍♂️

Why anyone would build next that way though still boggles my mind

I think you are still missing the third option of shipping prebuilt next folders with the .next part still there. Docker instances just run next start

Then your production env variables belong to jenkins or github actions, or whatever

Usually people strip the .git folder too when publishing

I get it I think, it still means rebuilding the image for every environment I want to deploy to - so dev, integration, test, staging, pre-prod, prod - just doing it in build pipelines instead of in container startup - it's still a big hassle/waste.

But I agree better than delaying container start / using a PV on the container.

It is a little bit stinky up front but at least the dockers start and run fast

Also you can bundle the node_modules to skip npm install

True, build cache should be shared between deployments as they are cache. Next.js needs to re-build anyway, it's the design rather than a bug or lack of feature

shared build cache sounds like a recipe for confusing production bugs when cache bugs happen

I guess I'll look into building for every environment deployment - the amount of build time it's going to waste is obscene. It might be by design but you can tell I'm really annoyed by that design choice!

But otherwise, if you think of at least some part of next being an SSG the build step for each environment actually makes a lot more sense

Since you are running your own dockers that do have a file system, you can kind of bend the rules a little bit as compared to completely stateless stuff in lambdas. For example, if you actually 100% do not use any of the data in the environment variables for pre-rendered stuff (ie. no CMS change, or database data in SSR) maybe ship the environment configuration as a config file to the containers...