C#•2y ago

❔ Multithreading an async function

I have an async function that I want to multithread with n number of threads, but I get convert error if I initialize the thread with new Thread(methodName) From my understanding, I need to use the .Wait() method every time I call the request method, because it has a Task object.

int threadNumber = 5;
Thread[] threads = new Thread[threadNumber];

for (int i = 0; i < threadNumber; i++)
{
  threads[i] = new Thread(request);
  threads[i].Start();
}

for (int i = 0; i < threadNumber; i++) threads[i].Join();

int threadNumber = 5;
Thread[] threads = new Thread[threadNumber];

for (int i = 0; i < threadNumber; i++)
{
  threads[i] = new Thread(request);
  threads[i].Start();
}

for (int i = 0; i < threadNumber; i++) threads[i].Join();

The method request():

static async Task request()
{
  using var client = new HttpClient();
  client.Timeout = TimeSpan.FromSeconds(5);
  var res = await client.GetAsync(URI);
}

static async Task request()
{
  using var client = new HttpClient();
  client.Timeout = TimeSpan.FromSeconds(5);
  var res = await client.GetAsync(URI);
}

73 Replies

KnucelesOP•2y ago

This is the error I get from initializing thread with the method

KnucelesOP•2y ago

KnucelesOP•2y ago

If I change the return type from Task to void, then I can initialize it, but the method doesn't complete at the await key

anita•2y ago

don't manually create new threads, use the threadpool with Task.Run or the Parallel class dont create a new httpclient on each request, you should only use one across your application but you shouldn't run this method on a new thread, because it isn't cpu bound work you could do something like this:

var tasks = Enumerable.Range(0, 5).Select(x => request());
await Task.WhenAll(tasks);

var tasks = Enumerable.Range(0, 5).Select(x => request());
await Task.WhenAll(tasks);

KnucelesOP•2y ago

But this isn't making threads Doesn't WhenAll wait until all the code is executed, but doesn't it do it synchronously, rather than creating multiple threads

anita•2y ago

Why do you want to create multiple threads? If you want to run the requests in parallel, then this works An if you really want multiple threads then just use the methods i mentioned before and NOT new Thread()

Pobiega•2y ago

your request method should work just fine in a single thread its just waiting for IO thats literally the ideal usecase for async :p

KnucelesOP•2y ago

I'm trying to send multiple http requests at once, and some requests would get stuck in a timeout, so I think using multiple thread is necessary unless I'm missing something

x0rld 👻 🎃•2y ago

the new HttpClient() in each request catsweat

Pobiega•2y ago

absolutely not.

KnucelesOP•2y ago

I'm setting a new proxy address for each request. dang it, i forgot that i can just override those address lol

Pobiega•2y ago

you can make multiple requests at once without having multiple threads. as long as you don't use any blocking code, you're fine

KnucelesOP•2y ago

Thank! The Task.Run works perfectly fine with this. but for educational purpose, is there a way to do it with multi threads?

Pobiega•2y ago

Task.Run will use multiple threads.

KnucelesOP•2y ago

Do you have an example?

anita•2y ago

don't use Task.Run for this!!!! you don't need multiple threads to send the request at the same time

Pobiega•2y ago

var client = new HttpClient();
var listOfUrls = new List<string>
{
    "https://www.google.com",
    "https://www.microsoft.com",
    "https://www.bing.com",
};
var tasks = listOfUrls.Select(url => client.GetAsync(url)).ToArray();
var results = await Task.WhenAll(tasks);

var client = new HttpClient();
var listOfUrls = new List<string>
{
    "https://www.google.com",
    "https://www.microsoft.com",
    "https://www.bing.com",
};
var tasks = listOfUrls.Select(url => client.GetAsync(url)).ToArray();
var results = await Task.WhenAll(tasks);

anita•2y ago

KnucelesOP•2y ago

oh this is fascinating I need to learn this separately thank you guys !close

Accord•2y ago

Closed!

KnucelesOP•2y ago

new problem

static async void Test()
{
  var client = new HttpClient();

  Task[] tasks = Enumerable.Range(0, 5).Select(async x =>
  {
    var res = await client.GetAsync(URL);
    Console.WriteLine(res.StatusCode);
    }).ToArray();
    await Task.WhenAll(tasks);
  }
}

static async void Test()
{
  var client = new HttpClient();

  Task[] tasks = Enumerable.Range(0, 5).Select(async x =>
  {
    var res = await client.GetAsync(URL);
    Console.WriteLine(res.StatusCode);
    }).ToArray();
    await Task.WhenAll(tasks);
  }
}

When I execute Test(), it seems like the code doesn't wait for the await part in the function but if i replace await client.getasync(url) to something that doesnt require async, the code runs

cap5lut•2y ago

if just call Test(); instead of await Test(); its basically fire and forget

Omnissiah•2y ago

await Task.WhenAll doesn't seem correct there it should be outside the lambda

cap5lut•2y ago

instead of thinking about multi threading u should first think about how async works

KnucelesOP•2y ago

🤦 totally forgot

cap5lut•2y ago

also u should not use async void but async Task

Pobiega•2y ago

public static async Task Main()
{
    var client = new HttpClient();
    var listOfUrls = new List<string>
    {
        "https://www.google.com",
        "https://www.microsoft.com",
        "https://www.bing.com",
    };
    var tasks = listOfUrls.Select(url => GetStatusCode(client, url)).ToArray();
    var results = await Task.WhenAll(tasks);

    foreach (var statusCode in results)
    {
        Console.WriteLine(statusCode);
    }

}

private static async Task<HttpStatusCode> GetStatusCode(HttpClient client, string url)
{
    var resp = await client.GetAsync(url);
    return resp.StatusCode;
}

public static async Task Main()
{
    var client = new HttpClient();
    var listOfUrls = new List<string>
    {
        "https://www.google.com",
        "https://www.microsoft.com",
        "https://www.bing.com",
    };
    var tasks = listOfUrls.Select(url => GetStatusCode(client, url)).ToArray();
    var results = await Task.WhenAll(tasks);

    foreach (var statusCode in results)
    {
        Console.WriteLine(statusCode);
    }

}

private static async Task<HttpStatusCode> GetStatusCode(HttpClient client, string url)
{
    var resp = await client.GetAsync(url);
    return resp.StatusCode;
}

Florian Voß•2y ago

just for my understanding, doesn't your solution also use multiple threads? this one

Pobiega•2y ago

no only one thread there.

cap5lut•2y ago

arent the tasks executed on the thread pool and thus its simply not defined on which threads its running?

Pobiega•2y ago

I highly recommend reading $nothread

MODiX•2y ago

https://blog.stephencleary.com/2013/11/there-is-no-thread.html

There Is No Thread

This is an essential truth of async in its purest form: There is no thread.

Pobiega•2y ago

the tl;dr is that no, not every task/await results in work being done on the threadpool

cap5lut•2y ago

basically each task gets scheduled, runs on a non busy thread til its execution is paused, once it can resume it will continue execution on one of the threads again? eg, the requests will be built on one or more threads of the thread pool, etc. then the task execution will be suspended until the request is sent and response comes in, then the task execution continues on a non busy thread again

KnucelesOP•2y ago

With this code, how would I add dynamic handler (Proxy address changes per request) to HttpClient? I don't see a way to add different handler values unless I initialize HttpClient in the method, because you can't add handlers after HttpClient is initialized

Pobiega•2y ago

Use IHttpClientFactory, imho

Florian Voß•2y ago

Great article, thx

Pobiega•2y ago

well, this works fine. granted its only three urls and all three of those are usually quick to respond, but I'm 100% sure it started all three before the first one replied.

anita•2y ago

what makes you think this?

Pobiega•2y ago

yeah the whole "proxy addr changes per request" thing has me thrown off I've never ever had to do anything like that before.

anita•2y ago

what repo?

Pobiega•2y ago

its @Knuceles thread

Unknown User•2y ago

Message Not Public

Pobiega•2y ago

yeah actually, in all the samples they shown so far the URL seemed to be the same only the proxy changed.. so its hittiing the same url with multiple proxies

Unknown User•2y ago

Message Not Public

Pobiega•2y ago

Agreed

Unknown User•2y ago

Message Not Public

KnucelesOP•2y ago

What’s this about?

Unknown User•2y ago

Message Not Public

KnucelesOP•2y ago

What repository? The code related to proxies?

Unknown User•2y ago

Message Not Public

KnucelesOP•2y ago

Which thread

Unknown User•2y ago

Message Not Public

KnucelesOP•2y ago

KnucelesOP•2y ago

Unknown User•2y ago

Message Not Public

KnucelesOP•2y ago

This is the thread related code

Unknown User•2y ago

Message Not Public

KnucelesOP•2y ago

I'm just making a proxy checker no i dont know what that means Regarding the google analytics cookies, i know that those cookies are generated from client-side javascript file, so I wanted to see how I can retrieve them with C# the intent is to see how I can get client-side js generated cookies

Unknown User•2y ago

Message Not Public

KnucelesOP•2y ago

Wdym?

Unknown User•2y ago

Message Not Public

KnucelesOP•2y ago

Yes, but there must be some way for me to mimic the browser behavior in C#, right?

Unknown User•2y ago

Message Not Public

KnucelesOP•2y ago

Obtaining the cookie somehow

Unknown User•2y ago

Message Not Public

cap5lut•2y ago

seems more like they want to bypass the openai rate limits, im out here

Unknown User•2y ago

Message Not Public

KnucelesOP•2y ago

Too meticulous

KnucelesOP•2y ago

I'm just playing around with the OpenAI API if it's not too obvious, im clearly new to C# I created bunch of C# projects, and it's becoming too hard to manage, so I'm just putting everything in this one project

Unknown User•2y ago

Message Not Public

KnucelesOP•2y ago

which includes this proxy project

Accord•2y ago

Was this issue resolved? If so, run /close - otherwise I will mark this as stale and this post will be archived until there is new activity.

Gaming

Programming

❔ Multithreading an async function