Performance: Sequential queue processing instead of parallel (horizontal scaling)
I recently converted a piece of logic into multiple workers, hoping to improve performance drastically by separating the logic in simple repeatable workers.
Im processing a list of data and passing this to the next worker using Queue [binding].
What i'm seeing as that the queue is being processed message per message instead of in parallel => concurrent => horizontal scaling.
I'm having a hard time figuring this out.
I've tried:
- seperate messages
- batches
- batch sizes
Can anyone help me find the problem?
6 Replies
Im confused what you mean by this, can you explain what you mean by actually explaining the flow you want for messages
Sure.. my brain is a bit fried. But I'll give it a go.
To simplify it, let's say a have 2 workers: A and B.
1. Im sending a API request to worker A
2. Worker A processes the request preprocesses the data and splits it into 200 objects.
3. Worker A (producer) sends the objects to the queue, asynchronously using Promise.All
4. Worker B (consumer) processes the messages.
My expectation would be that worker B would automatically scale to process the messages.
I feel like the queue is blocking somehow and resulting in the messages being processed sequentially.
It will scale up but theres limits to how many concurrent invocations it will start. The maximum concurrent invocations is currently 20. And if you worker is throwing errors it can also reduce the amount of concurrency it will use
I saw that limit in the docs. But as I mentioned the messages are processed 1 at a time.
I am also confused as to how batches effect scaling.
My messages can be processed separately.
But i see no difference between batches of 10 messages and sending single messages
Is that just what it feel like or do the queue stats actually say theres only 1 concurrent invocation
The stats do show some concurrency. But im updating my DB and streaming the status to my webapp. And the really seem to be updated one at a time
The average wall time of my worker is 10 seconds.
So i would expect the total processing time to be significantly lower then the amount of messages x 10.