C#•7mo ago

Brotli compression with all cores

I want to compress a stream but using all cpu cores for fastest compression

3 Replies

my current code

    using (var compStream = new BrotliStream(tempCompressorStream, CompressionLevel.SmallestSize, true))
    {
        while ((bytesRead = await dataStream.ReadAsync(buffer, 0, buffer.Length)) != 0)
        {
            await compStream.WriteAsync(buffer, 0, bytesRead);
            totalRead += bytesRead;

            double percentage = (double)totalRead / dataStream.Length * 100;
            string line = $"Compressing file please wait: {percentage.ToString(DecimalMask)}%";

            Console.SetCursorPosition(0, consoleTop);
            Console.Write(line + new string(' ', Console.WindowWidth - line.Length));
            Console.SetCursorPosition(0, consoleTop);
        }

        Console.WriteLine("");

        await compStream.FlushAsync();
    }

    using (var compStream = new BrotliStream(tempCompressorStream, CompressionLevel.SmallestSize, true))
    {
        while ((bytesRead = await dataStream.ReadAsync(buffer, 0, buffer.Length)) != 0)
        {
            await compStream.WriteAsync(buffer, 0, bytesRead);
            totalRead += bytesRead;

            double percentage = (double)totalRead / dataStream.Length * 100;
            string line = $"Compressing file please wait: {percentage.ToString(DecimalMask)}%";

            Console.SetCursorPosition(0, consoleTop);
            Console.Write(line + new string(' ', Console.WindowWidth - line.Length));
            Console.SetCursorPosition(0, consoleTop);
        }

        Console.WriteLine("");

        await compStream.FlushAsync();
    }

Auger•7mo ago

This is a harder problem than it probably first appears... In order to have multithread compression, you'd have to break the file into chunks (Look into the Partitioner class for that), and then use a Parallel.ForEach to compress each chunk. And while that would work... you'd have to decompress each chunk and re-combine the exact same way. The order of chunk has to be persisted as metadata in order to decompress later.

glacinefroxOP•6mo ago

but then i cant use the dictionary of first chunks onto the next ones so i would end with larger file

Gaming

Programming

Brotli compression with all cores