Poor Worker<->R2 performance even within same region
We’re trying to serve assets stored in R2 via a Worker. The assets are distributed in 4 R2 buckets, each in a different region, and we even tried issuing 2 read requests to the 2 nearest regions in parallel to smooth out the latency.
None of this seems to help much. We’re still seeing R2 TTFB latencies to the nearest region of up to 400-600ms in about 10% of requests, often after a long period of no requests (even though the worker boots up quickly - almost as if the storage itself is “cold”). Since it’d be highly unlikely that both R2 regions we’re reading from would take a slow path or get congested at the same time, it’s likely that there is an issue on the workers side that stalls our R2 requests.
Are these numbers expected? They seem really high. In comparison, S3 operations from an AWS Lambda in the same region have 10x lower TTFB (based on my crude eyeballing).
These assets are on the critical path for page loads, and with client RTT + CF cache lookups sporadically taking close to 100ms, we often see end-user response times over 1s. This is obviously really high.
Has anyone had any experience with this behavior with R2 when accessed from a Worker and has some advice on how to mitigate? Relying only on the Cloudflare cache for performance has proven tricky since it evicts quickly and very often doesn’t even seem to commit new data into the cache, causing a miss on a subsequent request, for example.
0 Replies