❔ How to do faster HttpClient requests?
I've done a classic HttpClient initialize once and for-loop it and it takes about 0.7s per request, which is fine-ish but due to the size of my array I would really love to get it faster.
I'm aware that I may be API-limited, but what would be the best way going about proving that?
I've tried having the responses be saved in a concurrent bag, and parallel.foreach-ing my array while creating an httpclient in each iteration.
This was far from the best, and I've understood that. My current implementation has a ConcurrentQueue of HttpClients, with a maximum of 100 parallelisations at once.
Each thread would dequeue an httpClient, do the request and then enqueue it back. But I'm getting worse results like that. It's approximately 50-60 requests in 40s, which is still roughly 0.7-0.8s per requests.
Would this indicate an API limitation, or is my way of optimizing this wrong?
Thanks a ton!
(It's currently being done in a console-app with .NET 6, just to get proof it works)
5 Replies
Little update:
I've managed to take it down to approx. 0.5s per request if I'm running multithreaded 8 clients only..
Would this indicate a network limit or an API limit to not be able to process the 100 clients.
I'd assume it's both. Either way, what would be the best way to fine-tune the maximum number of parallelism to get requests taking the least time possible.
Have you tried using HttpClientFactory? It handles all the management of the HttpClient instances for you: https://www.stevejgordon.co.uk/introduction-to-httpclientfactory-aspnetcore
Steve Gordon
Steve Gordon - Code with Steve
HttpClientFactory in ASP.NET Core 2.1 (Part 1) - Steve Gordon - Cod...
A basic introduction to the new HttpClientFactory feature available in ASP.NET Core 2.1 which helps with management and usage of HttpClient.
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
I’ll post a code snippet here later today 🙂
Was this issue resolved? If so, run
/close - otherwise I will mark this as stale and this post will be archived until there is new activity.