C
C#2y ago
Meistro

❔ multithreading with Tasks

I have a question about C# multithreading or async methods. I'm currently trying to upload multiple files to a server via an API (with login data, etc.). My problem is that while waiting for all the tasks to complete, I want to query how many tasks have already finished, as long as the tasks are not yet completed. Specifically, I am uploading 30k files. Each file is started as a task. While the tasks are running, a loop should run during the await (await Task.WhenAll(uploadTasks);) to check how many tasks have already completed, and every 10k files, there should be a message like "Hey, I have uploaded 10k files and it's still running."
//<--- there is a code for the task above that runs all task

await Task.WhenAll(uploadTasks);

//decides how often should the system give a feedback every 10000 Files
runnerctr = 0;

//10.000 steps
int runner = 0;

// Loop that checks out hwo is process doing
while (runner < int.Parse(File_Upload_Count_Simu.Text))
{
//wait for some time before updating
await Task.Delay(TimeSpan.FromSeconds(5));

// loop through all tasks and update the counter for all done uploads(tasks)
foreach (var task in uploadTasks)
{
if (task.IsCompleted)
{
runner++;
}
}
// get progress every 10000 tasks which are done
if (runner >= runnerctr + 10000)
{
runnerctr += 10000;
UpdateRichTextBox("do something");
}
}
//<--- there is a code for the task above that runs all task

await Task.WhenAll(uploadTasks);

//decides how often should the system give a feedback every 10000 Files
runnerctr = 0;

//10.000 steps
int runner = 0;

// Loop that checks out hwo is process doing
while (runner < int.Parse(File_Upload_Count_Simu.Text))
{
//wait for some time before updating
await Task.Delay(TimeSpan.FromSeconds(5));

// loop through all tasks and update the counter for all done uploads(tasks)
foreach (var task in uploadTasks)
{
if (task.IsCompleted)
{
runner++;
}
}
// get progress every 10000 tasks which are done
if (runner >= runnerctr + 10000)
{
runnerctr += 10000;
UpdateRichTextBox("do something");
}
}
it is working by the way ...but I am not sure whether it is realy counting while awating or not.
77 Replies
sibber
sibber2y ago
async await isnt multithreading btw anyway youre waiting for all the tasks to finish, then running the loop
Meistro
MeistroOP2y ago
Oh, thank you for correction 🙏🏽
sibber
sibber2y ago
in other words, the loop wont run until all the tasks have finished so youre not counting anything
Meistro
MeistroOP2y ago
Is there a way to do the loop while waiting ?
sibber
sibber2y ago
await after the loop
DaVinki
DaVinki2y ago
Out of curiosity are you uploading each file individually
Meistro
MeistroOP2y ago
Sounds legit 😅 Every file is a task and they are all starting within a while loop. After the while loop the code above runs
sibber
sibber2y ago
also this could be simplified
Meistro
MeistroOP2y ago
The file upload is a HTTP POST request btw. And the task is done when the http gives a ok back I really want to be open to any idea. I'm still a beginner and need your help
sibber
sibber2y ago
you want to send a message every 10k tasks right?
Meistro
MeistroOP2y ago
Maybe I missunderstood your question Yes right
sibber
sibber2y ago
you probably dont need anything more complicated than this then:
Task aggregateTask = Task.WhenAll(uploadTasks);

int lastCompletedTaskCount = 0;
while (!aggregateTask.IsCompleted)
{
await Task.Delay(TimeSpan.FromSeconds(5));

int completedTaskCount = uploadTasks.Count(x => x.IsCompleted);
if (completedTaskCount - lastCompletedTaskCount >= 10_000)
{
SendMessage();
lastCompletedTaskCount = (completedTaskCount / 10_000) * 10_000;
}
}

await aggregateTask;
Task aggregateTask = Task.WhenAll(uploadTasks);

int lastCompletedTaskCount = 0;
while (!aggregateTask.IsCompleted)
{
await Task.Delay(TimeSpan.FromSeconds(5));

int completedTaskCount = uploadTasks.Count(x => x.IsCompleted);
if (completedTaskCount - lastCompletedTaskCount >= 10_000)
{
SendMessage();
lastCompletedTaskCount = (completedTaskCount / 10_000) * 10_000;
}
}

await aggregateTask;
like we said, await after the loop
Meistro
MeistroOP2y ago
Oh wow !!! Thank you very much gor the effort !!! 🙏🏽 This looks interessting xD but will it give a message every 10k file ? Example I want to upload 50k files I need the programm to give me every 10k a message (how long did it took in everage etc.)
DaVinki
DaVinki2y ago
I never realized Task.WhenAll is a task itself, that's useful lol
Meistro
MeistroOP2y ago
But ! I can work with this ! Thank you so much !
sibber
sibber2y ago
sure np
Meistro
MeistroOP2y ago
How usefull ? 😅
sibber
sibber2y ago
there was a small bug, fixed it it should but you should test it i havent
Meistro
MeistroOP2y ago
Thank you … ah btw. Do I need the Task.delay(….? Yes I will
sibber
sibber2y ago
well not really, but you probably should have some delay you had a delay of 5 secs in your original code, so i kept it
Meistro
MeistroOP2y ago
its for my student Job you know… if I dont finish the project .. they might kick me from company Ah ok thank you :))) happy you answered so fast !!!! Great job !!!!
sibber
sibber2y ago
actually theres another bug lol
Meistro
MeistroOP2y ago
which one ?
sibber
sibber2y ago
alright fixed it before, it wouldnt exactly send a message every 10k because imagine the first iteration it did 10,001
Meistro
MeistroOP2y ago
yes
sibber
sibber2y ago
that means the second would need to at have 20,001 tasks done for it to send a message not 20,000
DaVinki
DaVinki2y ago
Instead of spinning, what if you used a countdown event?
sibber
sibber2y ago
i.e. it used to send a message when 10k tasks were done since the last iteration, not every 10k you could probably clean this up a bit
Meistro
MeistroOP2y ago
hmmm
sibber
sibber2y ago
but it works should work
Meistro
MeistroOP2y ago
you tryed it ?
sibber
sibber2y ago
wdym no i tried it in my head which isnt reliable
Meistro
MeistroOP2y ago
well dumb question 😄
sibber
sibber2y ago
no, good question
DaVinki
DaVinki2y ago
Remove the while loop and replace it with a blocking CountdownEvent with an initial count of 10,000, decremented at the end of every successful task
sibber
sibber2y ago
i shouldve said "it should work"
Meistro
MeistroOP2y ago
I am confused 😄 I am a beginner I can show you the whole method guys .. What I have right now if you want
sibber
sibber2y ago
oh TIL this exists
Meistro
MeistroOP2y ago
should I use this countdown event ? My main goal is that the task counting should be live I mean not after when all tasks are done but while the tasks arent still done you know what I mean ? Like when I download something... it shows something like 10% of 100% my problem before was, it showed me 10% of 100% done but actualy its done for long time haha sorry for my bad english
sibber
sibber2y ago
actually thats not exactly what we need here this signals when it reaches zero, we dont need that it would also require each task to have a reference to the cde
DaVinki
DaVinki2y ago
But you can get rid of Task.Delay and the spinning
sibber
sibber2y ago
not really? how does this replace that
Meistro
MeistroOP2y ago
this is the method guys 😄
DaVinki
DaVinki2y ago
It's blocking so you can wait it three times Preferably in a for loop in case maybe there are 20k or 40k requests
Meistro
MeistroOP2y ago
there are some german comments in it.. sorry for that you mean I should use for loop instead of while ?
DaVinki
DaVinki2y ago
For a beginner you should probably stick with the while version
Meistro
MeistroOP2y ago
ok ok
Florian Voß
Florian Voß2y ago
@Meistro is it normal in your company to write the code in english but the comments in german? Cuz I'm german too, we have projects where we write both in german or both in english but that is incosistent what you have
Meistro
MeistroOP2y ago
hei Florian 😄 no its not normal tho... I didnt push the code to Git yet. But I will change all comments into english as soon as I have a stable version 😄 sometimes its in german becauz I dint have time to translate it and I wanted a quick note for myself 😄
Florian Voß
Florian Voß2y ago
sure, just make sure you dont forget adjusting that before pushing
Meistro
MeistroOP2y ago
Yes I will! thanks for the advice 🙂 did this helped a bit ? or is it too much ? I know its confusing but I thought it might clarify things 😄 I think it worked
Meistro
MeistroOP2y ago
i mean "72,09682680000013 seconds" is not correct but I will fix this anyways
sibber
sibber2y ago
so it worked?
Meistro
MeistroOP2y ago
yeah ! 😄 realy Happy thank you !
sibber
sibber2y ago
nice np :)
Meistro
MeistroOP2y ago
this dumb method took me a whole day and I did not fix anything alone 😄 realy ! thank you very much !!!!
sibber
sibber2y ago
happy to help
Meistro
MeistroOP2y ago
🙂 is it ok if I delete the code I posted above ?
sibber
sibber2y ago
ofc
Meistro
MeistroOP2y ago
ok thx
sibber
sibber2y ago
$close
MODiX
MODiX2y ago
Use the /close command to mark a forum thread as answered
Meistro
MeistroOP2y ago
thanks again and have a good week !
sibber
sibber2y ago
you too :)
Meistro
MeistroOP2y ago
thx thx $close
MODiX
MODiX2y ago
Use the /close command to mark a forum thread as answered
Meistro
MeistroOP2y ago
the close doesnt work 😄 I tryed /close but it doesnt work xD
sibber
sibber2y ago
oh looks like the bot is down oops
Meistro
MeistroOP2y ago
another/next mission for you ! 😛
sibber
sibber2y ago
its fine just leave it lol its not our bot
Meistro
MeistroOP2y ago
ah kk
DaVinki
DaVinki2y ago
@Cyberrex I think we may have been overcomplicating it, you could just go through every task and ContinueWith a check on an atomic integer or similar state object and run a function dependent on the state, then just Task.WaitAll
sibber
sibber2y ago
that would be overcomplicating it and would also be a lot more inefficient especially when you have tens of thousands of files
DaVinki
DaVinki2y ago
It would be simpler but sure, less efficient
Jester
Jester2y ago
im just wondering if its fine to have 50k tasks
Accord
Accord2y ago
Was this issue resolved? If so, run /close - otherwise I will mark this as stale and this post will be archived until there is new activity.
Want results from more Discord servers?
Add your server