❔ xUnit - totally isolated serial parameterized tests?
Hopefully someone with more experience with different testing frameworks can help me out- I'm not married to using only xUnit. Though this isn't for unit testing since the test conditions depend on the file system/database state, I'm starting with xUnit just because my unit tests used it. I'm trying to find a good way to do parameterized integration tests. xUnit supports using ClassData and MemberData attributes to provide the parameters to the tests, but after a while, I found out that all the strange behavior I was seeing was due to xUnit creating instances of the parameters for all tests, all at the same time, before any test runs.
This is a no-go since the multiple IPersistentDataStorage objects will try to register multiple handlers with the storage providers (ie. file watchers) all at the same time and I can't have that. Is there another testing framework that would let me better control object lifetimes for use in integration tests or at least some other way to use xUnit to run tests completely serially?
12 Replies
Don't have any xunit-native solution, but you could just 'add another layer' and pass in data storage factory that you .Create() in the test setup to give your IPersistentStorage
I was hoping to have it show in the test runner as separate tests for each system under test, but it's looking like that might not be possible, at least with xUnit. Not being able to use xUnit isn't a problem, any thought on other test frameworks?
They do show some things in the test runner, but probably only simple compile time params like bools enums etc. If you're passing a bunch of different impls through an interface param I doubt any different test fw will help there
Using ClassData and MethodData provided me with a glimmer of hope in that it did everything I wanted in terms of test visibility in the runner and not having to duplicate my tests for each system under test. it's just that xunit seems to set up all the test parameters when the test class is first created rather than only immediately before a particular test is run. I guess I'm going to have to experiment with other frameworks' parameterized tests to see if they behave the same
Pretty sure this isn't possible with NUnit either
guess I won't start with that one then 
Although I'm still not sure what your problem is here :/
I mean, what really are you trying to accomplish
In the example test class (well, those lines are in a bigger test class) I posted, each of the two tests get their IPersistentDataStorage parameter using the StorageSystemUnderTest member. I'd like each tests' parameter object to be created immediately before the test gets run and disposed immediately after. Due to infrastructure restrictions only one can ever be alive at any time. With xUnit, it seems that there is no control over the instance lifetime. When I run just the two tests, it creates 8(!) different instances of IPersistentDataStorage. 1. The first 4 are created during (I'm guessing) discovery of all the different combinations of the parameterized tests before any tests are run (I don't think it disposes them either which is another problem on it's own). 2. Before the first test runs, it'll create two IPersistentDataStorage instances 3. Runs the first test with the first instance, and then again with the second instance. 4. Disposes both instances. 5. Creates two more instances and runs the second test with each instance. 7. Disposes the two instances. I need 1. Create parameter instance, run test, dispose parameter instance. and then repeat for every test/parameter combination.
@merp_mcderp you probably want fixtures https://xunit.net/docs/shared-context
I not trying to share data between tests, but use multiple, isolated instances to pass each test. (one test, multiple SUTs) Parameterized tests are the way to go. My workaround is to just do this- public class TestClass
{
[Theory, MemberData(nameof(StorageSystemUnderTest))]
public async Task TestWriteNotification(Func<IPersistentDataStorage> sutter)
{
using var sut = sutter();
//...
}
[Theory, MemberData(nameof(StorageSystemUnderTest))]
public async Task TestAutoReload(Func<IPersistentDataStorage> sutter)
{
using var sut = sutter();
//...
}
public static IEnumerable<object[]> StorageSystemUnderTest =>
new List<object[]>
{
new object[] { ()=>GetInMemoryStorage },
new object[] { ()=>GetSystemFileStorage },
new object[] { ()=>GetDatabaseStorage },
//...
};
}
so what I said 😁
Was this issue resolved? If so, run
/close - otherwise I will mark this as stale and this post will be archived until there is new activity.