NacDan
NacDan
CC#
Created by NacDan on 5/1/2023 in #help
Iterating over AsyncEnumerable and writing to MemoryStream
Gotcha that makes sense. Thanks
25 replies
CC#
Created by NacDan on 5/1/2023 in #help
Iterating over AsyncEnumerable and writing to MemoryStream
Of what it looks like under the hood
25 replies
CC#
Created by NacDan on 5/1/2023 in #help
Iterating over AsyncEnumerable and writing to MemoryStream
I am seeing the compiler transformations right now yes I see now
25 replies
CC#
Created by NacDan on 5/1/2023 in #help
Iterating over AsyncEnumerable and writing to MemoryStream
My main concern was unintentionally writing to the stream concurrently. I'm not all too familiar with AsyncEnumerables
25 replies
CC#
Created by NacDan on 5/1/2023 in #help
Iterating over AsyncEnumerable and writing to MemoryStream
the current enumerator
25 replies
CC#
Created by NacDan on 5/1/2023 in #help
Iterating over AsyncEnumerable and writing to MemoryStream
Is that because it is disposed before the next iteration?
25 replies
CC#
Created by NacDan on 5/1/2023 in #help
Iterating over AsyncEnumerable and writing to MemoryStream
Force synchonization while writing to the MemoryStream
25 replies
CC#
Created by NacDan on 5/1/2023 in #help
Iterating over AsyncEnumerable and writing to MemoryStream
Correct, WriteToParquet takes in the existing memorystream
25 replies
CC#
Created by NacDan on 5/1/2023 in #help
Iterating over AsyncEnumerable and writing to MemoryStream
For instance
SemaphoreSlim semaphoreSlim = new(1, 1);
await foreach (var batch in batches)
{
try
{
await semaphoreSlim.WaitAsync();
var records = batch.AsCollection();
totalBatches++;
totalBatches += records.Count;

await WriteToParquet(memoryStream, records);
}
finally
{
semaphoreSlim.Release();
}
}
SemaphoreSlim semaphoreSlim = new(1, 1);
await foreach (var batch in batches)
{
try
{
await semaphoreSlim.WaitAsync();
var records = batch.AsCollection();
totalBatches++;
totalBatches += records.Count;

await WriteToParquet(memoryStream, records);
}
finally
{
semaphoreSlim.Release();
}
}
25 replies