❔ Task.WhenAll for Tests, and other async questions
In xUnit, there is no support for testing async or in parallel within the same collection, where a collection is basically tests that share a single startup and teardown
So I extended some xUnit classes, found where the tests are ran, and instead of awaiting each one in a loop, I start them all and then Task.WhenAll. This makes the tests really fast, and when they pass, verifies there are no concurrency issues
Why would this be a bad idea? I assume there must be a reason they did not do this by default
And, would this be considered Parallel? I always thought async and Parallel are different, even opposite things, but if not awaiting them immediately, then it is parallel, isn't it?
And lastly, should I do some sort of batching, or degrees of parallelism, when doing a large number of Task.WhenAll (in general)? Or is it just going to handle that performantly on its own?
2 Replies
Then you may not be using xUnit, it's a pretty explicit limitation
All methods in the same class are by default in the same collection, and can't be parallel tested
In my case, I built a specific 'Collection' that spins up all our services, and I need all my tests to share this same set of services, so I have to make them all use the same Collection - so then they all get stuck not parallel without these modifications
Not sure whether to call it async or parallel testing, what I'm doing
For one, that's just verbose and messy boilerplate for no reason 😛 and for two, if I want xUnit to do startup once before all tests, and cleanup once after all tests, they have to share a Collection even if they're in separate classes
I don't want a new instance at every test, half the point is to test for concurrency problems
(I don't particularly like xUnit, especially for the boilerplate required. lol)
But that's part of the question sort of, is Task.WhenAll appropriate. I know it works well and I've used it before. But is it better to explicitly call Parallel with a DegreesOfParallelism, or to let the threadpool manage itself?
Or basically, if I do a Parallel on 16-thread machine, I can have 16 threads all awaiting a Task - but there are only 16 Tasks running. Each one has nothing else to do while it awaits
Whereas a Task.WhenAll should allow all 16 threads to run 16 tasks, then pick up 16 new ones, etc, before going back to finish the first ones
(But I don't think that's really how it works, because in past experimentation, I've gotten much more speed out of Parallel+await than Task.WhenAll. Or maybe it was just starting the tasks in parallel then still doing a WhenAll... can't remember... which sounds trivial, but everything up to the first await is otherwise blocking, and I can clearly see that one test spins for a while, then all of them start, so there is apparently some significant setup)
ForEachAsync might actually be the solution to the weird problem I'm now imagining, where ideally you'd want to start all the tasks then await them in parallel
well, not always ideally. There are likely good reasons to choose WhenAll over that. Probably WhenAll would be better if you are trying to minimize the impact, and Parallel would be for getting it done ASAP at the cost of everything (probably more appropriate for tests)
Though my original question is more along the lines of ... why does xUnit say this is impossible, when it seems really rather easy. I think I'm doing something fundamentally wrong, or basically (for some reason) you're not supposed to want to test the same collection in parallel because they share too much (even though that's exactly why I want to do it... because it simulates prod to have one instance processing many concurrent requests)
Maybe it's just that not all code is thread-safe and there's no way to opt-out of concurrent testing, but that'd be a simple attribute to add
Time to fork it, add my stuff, and release xPUnit
right, you want to be able to not have it in some cases. But if everything is async, you really ought to test them asynchronously
(The main implementation is awaiting and all, but it's not really async because it always awaits, there's one synchronous chain to follow and never more than one thread in use)
right, but there is no other test queued; they await it as soon as they start the task, one task at a time
Well, I guess if you count the parallel testing outside of collections, then yeah there is a purpose and it's async
I think those actually explicitly use Parallel...
I'm tempted, but, tbh I wrote the thing at work. I could rewrite it here. But that's not cool
I'll just flesh it out and make a package for it at work
You were an excellent rubber duck 😛 though I do wish I could hear from someone why this is a terrible idea
But if you also think it's dumb that xUnit doesn't allow that, then it seems like a good idea to me
that's fair, but a test that fails intermittently is still a failing test (ironically, doing this solved a lot of those problems...)
meh, as long as they are all solid to begin with, they'll notice if some are brittle. You're very right, but right now it's all solid for once
The old way had some of those issues, I think because we were spinning up multiple instances of something that isn't made to have multiple instances and there was some weird cross contamination
that might be part of the answer on why this is wrong... it is certainly suspicious that refactoring xUnit made my tests errors go away
though tbh I can't remember if those intermittent errors were even in the part of the tests that I changed
Looks like nothing has happened here. I will mark this as stale and this post will be archived until there is new activity.