duch
CDCloudflare Developers
•Created by duch on 10/2/2023 in #workers-help
Workers not scaling well - io getting slower
Hi,
I'm working on an application that needs to process (usually) a few thousand files and wanted to use Workers combined with R2 for that purpose.
However I'm experiencing slow file downloads and wanted to ask for help.
My setup:
- WorkerA -> receives array of N filenames (N=6 to stick to the open connections limit) and executes WorkerB N times
- WorkerB -> receives filename, downloads it from R2 (service binding) and sends timings to external webhook.
- Each file is almost the same size around 25 MB. For now files are only downloaded (no additional processing)
Testing:
I'm using 600 files for testing (so 15GB). My local script executes 100/200/300 requests to WorkerA passing 6 filenames in each request. The requests are made in less than 2 seconds.
The webhook receives data from WorkerB with times it took to download the file and I can see degradation in performance with growing number of files
- on 600 files 89% of all requests complete under 5s
- on 1200 files (each file requested 2 times, so 200 requests to WorkerA) 62% of all requests complete under 5s
- on 1800 files (each file requested 3 times, so 300 requests to WorkerA) 51% of all requests complete under 5s
The requests that take longer than 5 seconds easily reach 8...15 to even 20+ seconds in some cases.
The same happens when I'm downloading the same files with fetch api from different provider - so this doesn't look like R2 issue.
I'm on Workers Paid plan and the workers are in Unbound mode. "SmartPlacement" doesn't change anything.
The actions will be triggered by the end user so for obvious reasons this needs to be as fast as possible - I was hoping to download each file in less than 4 seconds.
Can you explain why it happens? Is it a known limitation?
Is the bandwidth throttled / some limit per account?
Is there anything I can do to have consistent performance that doesn't depend on the number of files I want to process?
Kind regards
8 replies
CDCloudflare Developers
•Created by duch on 6/2/2023 in #workers-help
R2 file processing inconsistent behaviour
Hi,
I have a worker that fetches a file from R2 and then processes it.
I'm looking to find the biggest file size that I can process in reasonable time on workers platform.
The worker takes r2 path as parameter in the url. The files I'm testing with are:
- 1, 10, 15, 20, 25, 50 MB (each file is tested in separate request)
When I start testing using increasing size order it works up to 25 MB then it fails on 50MB with CPU time exceeded (that's ok).
If I go back to test 25MB or even 1MB I'm still getting Error 1102 Worker exceeded resource limits. Why? It was working before.
The environment didn't change and I'm processing the file exactly the same way. Currently I'm on Workers Free plan - does it matter?
Just to be clear how to reproduce - requests order:
https://xxx.workers.dev/?f=file10mb (OK)
https://xxx.workers.dev/?f=file25mb (OK)
https://xxx.workers.dev/?f=file50mb (FAILS Error 1102 Worker exceeded resource limits)
https://xxx.workers.dev/?f=file25mb (FAILS Error 1102 Worker exceeded resource limits)
https://xxx.workers.dev/?f=file10mb (FAILS Error 1102 Worker exceeded resource limits)
7 replies