Latency issues
How many Workflows do you have running?
We are aware of the some performance issue under heavy load (per account) and we are working on fixes for it
9 Replies
So what happened was I was trying to queue all of my workflows (I had 900,000 to do)
And what happened was initially I got response times of 200ms or so then as more were added my response time got slower and slower (up to 40 seconds to queue a workflow instance)
The best I could get was 2 requests per second
So I moved to cloudflare queues in the end as it wasn’t viable for me to leave this script running (from my local machine sending the workflow start instructions) for 5 days which is what I would have needed to do with being capped at this rps
(Each workflow takes about 3 seconds)
Thank you for reporting Sam! I are looking into it
When I moved to queues i got the workload done but it did take 10 hours
But at least I could queue all the requests up then switch my laptop off
So I’m also curious how worker concurrency can scale beyond 250 workers in queues as a side note if you have a connection to queues team also @Diogo Ferreira
Workflows is the ideal product for me (for this use case not queues) but I do understand it’s in beta. So I’m open minded to using either.
Sorry Sam, I do not get your question
I queued 900,000 requests (to cf queues) last night because of the issues with workflows
It took 10 hours to finish even with the worker concurrency maxed out at 250 workers
Oh, I see.
You can trigger/create multiple workflows from the same worker. This is a way to scale it.
So effectively create multiple workflows that do the same thing then randomly distribute to a pool of workflows @Diogo Ferreira ?
Do i need to care about worker concurrency also or is that handled by the workflow itself?
Basically what Im asking is if I do what you just said will it scale the worker compute as much as is needed automaticaly (providing I make sure my non cloudflare resources are setup for it e.g. postgres db)
and this approach basically means if I have 5 workflows for example i can have 500 requests running at once instead of 100
There is a limit on the number of concurrent Workflow instances (which will grow over the beta)
So you’re likely running into one or more things here
Is the solution I proposed the way to deal with this problem today?