Hello R2, I am trying to track a hang/
Hello R2, I am trying to track a hang/slow down when I download a lot of files very fast. I use R2 to hold data/artifacts for ML training, and before training download a few hundred GB per worker from 16-64 workers.
What I see if things seem to work fine downloading on one worker, but when I ramp it up some files hang or download very slowly (O(kbps)).
I use S5cmd with parallelism, so total download can be ~300 Gbit/s and 300+ class b requests per second. Is that something that's likely to get throttled?
1 Reply
I messaged support and they pointed me here
This is a bummer
yeah, but wouldn't it be cool