Pages upload limits
Hey there! I can see the stated file limit for Pages/Workers is currently 20k.
We're leaning hard into a fully static build using 11ty for an Australian business news site, and are already hovering around the 15k mark. We're also looking to implement full site search via Pagefind, which introduces another new file for each searchable item in the build.
Basically, we're going to hit that 20k limit real fast! We previously had a split build, some static, some on-demand -- but the full site search we're hoping to use requires everything to be built to make the index.
My colleague has an open case for this:
01456574, but I just wanted to provide some more context and try to get a definitive answer around if this is a hard limit, or just a sensible-fits-most-normal-cases limit, which can be bumped when required 🙂 Ty!11 Replies
It’s a technical limit. Depending on your architecture, you might be able to break it up into separate Workers?
Ah bummer. We probably could split collection types into their own workers (one for articles, one for newsletters, etc), but given how news works (everything happens so much), splitting things up will just delay the inevitable. I'll hit up the whiteboard, but keen to hear if you have any other news/large static users that might be happy to share strats and tactics!
You might also be able to move it out onto an R2 bucket? It wouldn’t be quite as clean, but you could scale a lot higher
I haven't messed with bindings yet, but sounds like a fun experiment!
Something along the lines of:
1. Build site locally (a bunch of html files)
2. Push to R2 bucket
3. Set up Worker with binding to R2 bucket
4. All requests to worker on a known path will grab html from R2 and use it as the response
I'm assuming it'll be fast enough, cause it's all in Cloudflare?
Sure! You would probably want to add some caching to reduce costs and increase speed a bit
One thing I love love love about the pages upload is the in-built check to see what has actually changed:
Does R2 have a similar capability?
Not by default? The reason Pages has a limit to the number of files is because it uses a manifest to map file names to hashes, so if a hash has already been uploaded, it doesn’t need to be uploaded again. You could build your own manifest, but you would have to manage it yourself
Gotcha. If it's not secret sauce, I'd love to know more about the technical limitations -- if we're gonna make our own manifest, it'd be nice to not have to discover issues you're already across. Fully understand if that's not possible.
Well, the big thing is that the Pages Manifest has to fit inside a Worker. If you have somewhere else to store the manifest, like a SQLite/Postgres/bring-your-own-flavor DB, you can grow it however large you want
Ah yep, that makes sense!
Might be a terrible idea, but I was just thinking a .json file, either locally or in the R2 bucket? Will do some more reading and get into it.
Thanks heaps for taking the time to get me up to speed, really appreciate it!
Yeah, that could work! Then, your limit would just be the Worker memory limit, but at 128 MB, you have a bit of wiggleroom