R2 file processing inconsistent behaviour
Hi,
I have a worker that fetches a file from R2 and then processes it.
I'm looking to find the biggest file size that I can process in reasonable time on workers platform.
The worker takes r2 path as parameter in the url. The files I'm testing with are:
- 1, 10, 15, 20, 25, 50 MB (each file is tested in separate request)
When I start testing using increasing size order it works up to 25 MB then it fails on 50MB with CPU time exceeded (that's ok).
If I go back to test 25MB or even 1MB I'm still getting Error 1102 Worker exceeded resource limits. Why? It was working before.
The environment didn't change and I'm processing the file exactly the same way. Currently I'm on Workers Free plan - does it matter?
Just to be clear how to reproduce - requests order:
https://xxx.workers.dev/?f=file10mb (OK)
https://xxx.workers.dev/?f=file25mb (OK)
https://xxx.workers.dev/?f=file50mb (FAILS Error 1102 Worker exceeded resource limits)
https://xxx.workers.dev/?f=file25mb (FAILS Error 1102 Worker exceeded resource limits)
https://xxx.workers.dev/?f=file10mb (FAILS Error 1102 Worker exceeded resource limits)
4 Replies
What's your Worker code?
Pastebin
export interface Env { MY_BUCKET: R2Bucket;}function objectNotFoun...
Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.
Looks like you're doing lots of processing so that's expected
You've got more data to process so it'll use more CPU
CPU time isn't deterministic, sometime it'll use more and sometimes it'll use less.
Now, I also tested the same file (10MB) between 10-20 times and after around 10th try it exceeded resource limits.
Thank you, that makes it clearer - I'll test with a paid plan.