C
C#•8mo ago
glacinefrox

Brotli compression with all cores

I want to compress a stream but using all cpu cores for fastest compression
3 Replies
glacinefrox
glacinefroxOP•8mo ago
my current code 
    using (var compStream = new BrotliStream(tempCompressorStream, CompressionLevel.SmallestSize, true))
    {
        while ((bytesRead = await dataStream.ReadAsync(buffer, 0, buffer.Length)) != 0)
        {
            await compStream.WriteAsync(buffer, 0, bytesRead);
            totalRead += bytesRead;

            double percentage = (double)totalRead / dataStream.Length * 100;
            string line = $"Compressing file please wait: {percentage.ToString(DecimalMask)}%";

            Console.SetCursorPosition(0, consoleTop);
            Console.Write(line + new string(' ', Console.WindowWidth - line.Length));
            Console.SetCursorPosition(0, consoleTop);
        }

        Console.WriteLine("");

        await compStream.FlushAsync();
    }
    using (var compStream = new BrotliStream(tempCompressorStream, CompressionLevel.SmallestSize, true))
    {
        while ((bytesRead = await dataStream.ReadAsync(buffer, 0, buffer.Length)) != 0)
        {
            await compStream.WriteAsync(buffer, 0, bytesRead);
            totalRead += bytesRead;

            double percentage = (double)totalRead / dataStream.Length * 100;
            string line = $"Compressing file please wait: {percentage.ToString(DecimalMask)}%";

            Console.SetCursorPosition(0, consoleTop);
            Console.Write(line + new string(' ', Console.WindowWidth - line.Length));
            Console.SetCursorPosition(0, consoleTop);
        }

        Console.WriteLine("");

        await compStream.FlushAsync();
    }
Auger
Auger•8mo ago
This is a harder problem than it probably first appears... In order to have multithread compression, you'd have to break the file into chunks (Look into the Partitioner class for that), and then use a Parallel.ForEach to compress each chunk. And while that would work... you'd have to decompress each chunk and re-combine the exact same way. The order of chunk has to be persisted as metadata in order to decompress later.
glacinefrox
glacinefroxOP•7mo ago
but then i cant use the dictionary of first chunks onto the next ones so i would end with larger file

Did you find this page helpful?