Latency issues

How many Workflows do you have running? We are aware of the some performance issue under heavy load (per account) and we are working on fixes for it
9 Replies
Sam
Sam2w ago
So what happened was I was trying to queue all of my workflows (I had 900,000 to do) And what happened was initially I got response times of 200ms or so then as more were added my response time got slower and slower (up to 40 seconds to queue a workflow instance) The best I could get was 2 requests per second So I moved to cloudflare queues in the end as it wasn’t viable for me to leave this script running (from my local machine sending the workflow start instructions) for 5 days which is what I would have needed to do with being capped at this rps (Each workflow takes about 3 seconds)
Diogo Ferreira
Diogo FerreiraOP2w ago
Thank you for reporting Sam! I are looking into it
Sam
Sam2w ago
When I moved to queues i got the workload done but it did take 10 hours But at least I could queue all the requests up then switch my laptop off So I’m also curious how worker concurrency can scale beyond 250 workers in queues as a side note if you have a connection to queues team also @Diogo Ferreira Workflows is the ideal product for me (for this use case not queues) but I do understand it’s in beta. So I’m open minded to using either.
Diogo Ferreira
Diogo FerreiraOP2w ago
Sorry Sam, I do not get your question
Sam
Sam2w ago
I queued 900,000 requests (to cf queues) last night because of the issues with workflows It took 10 hours to finish even with the worker concurrency maxed out at 250 workers
Diogo Ferreira
Diogo FerreiraOP2w ago
Oh, I see. You can trigger/create multiple workflows from the same worker. This is a way to scale it.
Sam
Sam2w ago
So effectively create multiple workflows that do the same thing then randomly distribute to a pool of workflows @Diogo Ferreira ? Do i need to care about worker concurrency also or is that handled by the workflow itself? Basically what Im asking is if I do what you just said will it scale the worker compute as much as is needed automaticaly (providing I make sure my non cloudflare resources are setup for it e.g. postgres db) and this approach basically means if I have 5 workflows for example i can have 500 requests running at once instead of 100
elithrar
elithrar2w ago
There is a limit on the number of concurrent Workflow instances (which will grow over the beta) So you’re likely running into one or more things here
Sam
Sam2w ago
Is the solution I proposed the way to deal with this problem today?
Want results from more Discord servers?
Add your server