Deployments failing using NextJS and next-on-pages
Hi,
We have been deploying our nextjs application for quite some time, and we have been seeing several errors on building that we cannot really diagnose:
one is:
And other occurrences are out of heap memory limit (and we tried with the NODE_OPTIONS --max-old-space-size of 8192 (even when the console says it only accepts 7GB of mem).
We can build in local, with no issues (yes, we have good hardware...).
Is there a way to at least output some more information in the logs?
Thanks, team!
62 Replies
Were there any dependency updates in the last time? There are for example some issues with certain version of the vercel package with next-on-pages. We're currently on ^30.
This specific build contained some dependency updates, but the previous builds were also failing, so..
Anyway, it all started when moving and migrating to the /app folder, and without visibility on the logs we are quite clueless.
It is using Vercel CLI 32.5.0, but anyway at the end is running our
yarn run build, which runs next buildHave you tried building the app locally with next-on-pages? You can run yarn next-on-pages (if your have @ cloudflare/next-on-pages as a dev dependency), it will first run next build and then bundle for pages.
Yes, I did try it. I can also run with npx
And this is successful?
That's our build command in the project in Cloudflare.
"Works in my local" (tm).
mh, I think you're on the latest node LTS (18.x)? Thats what CFs build pipeline uses
yes, node18, (node20 entered LTS 2 weeks ago, but vercel CLI would break the build if you use node20 as they need AWS Lambda (https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html) to support node20)
Lambda runtimes - AWS Lambda
Learn about the languages that Lambda supports through runtimes.
Interesting, didn't see there is a new LTS version yet. But there is no hurry to update anyways
It depends on AWS and Vercel for that part. Many GitHub actions were upgraded to node20, but that's another topic not relevant now 😉
But otherwise I don't really have an idea to debug this. Never encountered such an issue. If it builds fine with next-on-pages locally it worked in CFs build pipeline too
Looks like something is going wrong in Next.js' build process that leads to the Vercel CLI catching an error (129) when it tries to run the build command, although you probably already deduced that much.
Maybe using
next build --debug would provide more info that might help. Not sure. If it was exceeding the memory limit, I would have expected to see the memory error logged. Maybe segfaulting perhaps, but nothing about that is surfaced :/I did run with next build --debug and the output is quite short:
(
yarn build runs next build --debug):/
We have run the same build with the same commitId, same hash, with 10 different branches, and 1 of them failed. Is it possible that the machines that run the build are different on every run?
Are there limits on the build size or anything like that?
We've been doing more tests, our build works with 13.5.4, but it does not with any following NextJs versions, 13.5.5, 13.5.6, 14.0.1 😦
Likely a newly introduced issue in nextjs from the sounds of things then
Yes... That's what we thought... but we don't dare right now to check 462 commits and 2136 files.
Go through the canary releases first 😛
It seems that someone else has the same issue as we have https://github.com/vercel/next.js/issues/51298
We set the flag to not minimize the build and it is going through... So it really does seem like a memory issue...
GitHub
Edge runtime - JavaScript heap out of memory · Issue #51298 · verce...
Verify canary release I verified that the issue exists in the latest Next.js canary release Provide environment information Operating System: Platform: darwin Arch: arm64 Version: Darwin Kernel Ver...
Intriguing, thanks for sharing your findings - I didn't expect that to be quite so resource intensive
Does you see any difference in final worker size when that is enabled vs disabled?
I will need to get into it and come back later.
If you are asking for:
That's with the flag disabled.
And surprisingly with the flag enabled:
But yeah, same old errors...
I thought we tamed it.
The size of the next-on-pages dist folder
Hope is this one. Not minimized.
This one is minimized:
And, yes we have set the NODE_OPTIONS to --max-old-space-size=8192, and the CI warns about that the max size is 7620
Some more background, we have been moving from the pages folder to the app folder in NextJS.
Failed: build exceeded memory limit and was terminated 😢No difference, nice, that means our minifying still takes care of everything regardless of how much swcMinify does
Still happening during the Next.js build part?
Yes, still happening after "Creating an optimized production build ..." and before "Compilation successful"
😭
That's also the thing, build in preview succeeds, then hit merge, and the integration tries it in develop (our prod branch), and it fails.
If you are going to escalate it somehow I could provide the accountId or anything.
Oh, this one is new
More tomorrow...
Hi there
?pages-deployment-id
The Pages deployment ID is a unique build identifier.
It's the UUID in the browser bar (for example, a URL would be
dash.cloudflare.com/ACCOUNT_ID/pages/view/PROJECT/DEPLOYMENT_ID where the deployment ID looks something like a398d794-7322-4c97-96d9-40b5140a8d9b).
This ID can help troubleshoot some issues with Pages builds so if you have a failing build make sure you grab that ID for the Pages team to use.Could you share a deployment ID?
Sure,
7feed5cc-d39a-4bf3-af44-99393feaa616
This one failed with an internal error: 69ec6c4f-28c2-4171-936b-7bd263c1afda
This one was OOM killed for running out of memory
Checking the other
Yes, we are trying to know why it goes over the limit when compiling the NextJS application and its app folder...
Thanks for checking those builds!
This seems to have genuinely been an internal error but feels transient, does it work or give another error on a retry?
May also have been an OOM that wasn't caught
I did retry the build.
Yes, all my last builds about it are raising OOM.
Alright
This one went through! 83013e77-ebd5-435f-958f-716cdc4ae35f
I'll keep an eye on the following builds we do (well, the integration with GitHub and our CI takes care of it 😉 ).
This has been an ongoing problem for us during the last 2 weeks 😦
I'll continue tomorrow with more builds
Morning (here)! There are other people who did some memory analysis and commented on the thread in GitHub about it https://github.com/vercel/next.js/issues/54708#issuecomment-1771802059
GitHub
Next.js development high memory usage · Issue #54708 · vercel/next....
Before posting a comment on this issue please read this entire post. Previous work The past few weeks we've been investigating / optimizing various memory usage issues. Specifically geared towa...
Sharing it so you may be aware that this can happen to other teams.
We are back at this again, moved 3 pages from the pages to the app folder, and started the memory issue because of compiling for the edge.
😭 😭
No progress today. All of them OOM.
I saw that there is a new version of next-on-pages that was solving some issues to make it compatible with 14.0.2. I tried a deployment, it went well, but tried again, and it failed, again. For the same reasons.
Yeah that fix won't improve your issue sadly, was just to patch a bug that they shipped in nextjs that broke rendering on Workers
I made it!
Full page with build errors!
I don't know what to check or do. Will see if 14.0.3 fixes something from Mon.
Otherwise will try publishing the pages from GitHub Actions.
We will lose the preview then... which is not good.
Uhm, it is possible to publish to preview...
You can publish to preview still. You can tell wrangler which branch to publish as. Not sure if the pages-action does that automatically or not
wrangler does it automatically based on current checkout which pages-action will also end up doing since it uses wrangler
We were deploying to our "demo" and "prod" environments using GitHub Actions, so it was a matter of copying and pasting for us.
When setting the branch, it always displays ¨HEAD¨ in Cloudflare´s deployment list, if the name of the branch I tell it in the parameter ¨branch¨ of cloudflare/pages-action is different from ´develop´.
Like
We delete the branches after we merge the PRs, I'll have a look next time we just push a PR.
I dont think it should be showing HEAD, are you able to share more than just that crop or a deployment ID?
DeploymentId a5d11c9e-fce3-4749-9d33-562e82acfdbb
I'm thinking that as I suspended the access to the Cloudflare App in GitHub, then you don't have access to our account anymore.
I didn't want to disconnect the whole apps, but I may do it soon.
next 14