Long-running worker performance
I've implemented a background sync worker, triggered by a queue. It calls an external, paginated API (dozens of sub-requests) and performs 100s of SQL queries using the socket API.
This works wonderfully, but it's very slow. Where a local execution might take 2-3 seconds, in production it takes over 60. Some syncs are ~5 minutes.
Any tips on how to diagnose the slowdown in production, or where to look to optimize? Off the top of my head, the source could be:
1. Roundtrips to/from the DB server (which is local in dev, and supabase.co in prod)
2. Worker CPU speed (don't think I'm doing anything too CPU intensive, but there is some list processing)
3. Sub-request (or socket) latencies
4 Replies
Some 99th percentile stats:
- CPU time: 3,053.7 ms
- Execution duration: 41.5 GB-sec
- Subrequests: ~10 per request (at least, according to the metrics reported)
For RTT to an external origin, look at https://developers.cloudflare.com/workers/platform/smart-placement/
Smart Placement (beta) · Cloudflare Workers docs
By default, Workers and Pages Functions are invoked in a data center closest to where the request was received. If you are running back-end logic in a …
I don’t think it’ll pick a location based on the TCP requests though
only on the fetch subrequests
Yeah, I've enabled Smart Placement, but it had no noticeable effect on execution duration.
Running the worker locally against the production DB leads to durations about 50% as long as production workers. Seems to suggest the primary issue is DB performance. I'll dig into DB query optimization first. I'm also going to try deploying the sync task as a Supabase edge function instead of a Cloudflare worker to see if that performs better.