C#•2w ago

Most optimized download system

Hi, I need to download a file separated into different chunks (between 100 and 400 mb each) from a server and make it as fast as possible. Except that I'm a bit confused, I've seen that there are lots of different methods for downloading files, with the added bonus of possibly downloading more than one at the same time. I was wondering if you had any advice on how to go about it? The code I'm currently using:

ProgressTask progress = ctx.AddTask("Downloading chunks", true, fileInfos.size);
                
AnsiConsole.MarkupLine($"[gray]LOG: [/] Started download of [blue]{fileInfos.chunksCount}[/] chunks");
for (int i = 1; i <= fileInfos.chunksCount; i++)
{
  await DownloadChunk(fileInfos.host, fileInfos.wd, fileInfos.id, i, fileInfos.chunkSizes[i - 1], progress);
}

AnsiConsole.MarkupLine("[yellow]Assembling chunks...[/]");
CombineChunks(tempDir, Path.Combine(downloadDir, fileInfos.filename), fileInfos.id);

ProgressTask progress = ctx.AddTask("Downloading chunks", true, fileInfos.size);
                
AnsiConsole.MarkupLine($"[gray]LOG: [/] Started download of [blue]{fileInfos.chunksCount}[/] chunks");
for (int i = 1; i <= fileInfos.chunksCount; i++)
{
  await DownloadChunk(fileInfos.host, fileInfos.wd, fileInfos.id, i, fileInfos.chunkSizes[i - 1], progress);
}

AnsiConsole.MarkupLine("[yellow]Assembling chunks...[/]");
CombineChunks(tempDir, Path.Combine(downloadDir, fileInfos.filename), fileInfos.id);

DownloadChunk:

using (var client = new WebClient())
{
  long previousBytesReceived = 0;
  client.DownloadProgressChanged += (sender, e) =>
  {
     progress.Increment(e.BytesReceived - previousBytesReceived);
     previousBytesReceived = e.BytesReceived;
  };
  client.DownloadFileCompleted += (sender, e) => AnsiConsole.MarkupLine($"[gray]LOG: [/]Finished download of chunk [green]{chunkNumber}[/]");
  await client.DownloadFileTaskAsync(new Uri(url), destinationPath);
}

using (var client = new WebClient())
{
  long previousBytesReceived = 0;
  client.DownloadProgressChanged += (sender, e) =>
  {
     progress.Increment(e.BytesReceived - previousBytesReceived);
     previousBytesReceived = e.BytesReceived;
  };
  client.DownloadFileCompleted += (sender, e) => AnsiConsole.MarkupLine($"[gray]LOG: [/]Finished download of chunk [green]{chunkNumber}[/]");
  await client.DownloadFileTaskAsync(new Uri(url), destinationPath);
}

41 Replies

ero•2w ago

i would handle this quite differently. first of all, use HttpClient instead of WebClient. then, i would try making use of Parallel.For instead of the for loop, or just straight up starting 4 DownloadChunk tasks at once and using Task.WhenAll to await all 4 of them so potentially something like this

const int DefaultCopyBufferSize = 81920;

var task = ctx.AddTask("Downloading chunks...", maxValue: fileInfos.size);

var chunkPaths = await Task.WhenAll(Enumerable.Range(0, fileInfos.chunksCount).Select(DownloadChunk));
// ...

async Task<string> DownloadChunk(int i)
{
    var path = Path.GetTempFileName();

    using var client = new HttpClient();

    using var response = await client.GetAsync("");
    response.EnsureSuccessStatusCode();

    using var source = await response.Content.ReadAsStreamAsync();
    using var destination = File.OpenWrite(path);

    var bytesRead = 0;
    var buffer = new byte[DefaultCopyBufferSize];

    while ((bytesRead = await source.ReadAsync(buffer)) > 0)
    {
        await destination.WriteAsync(buffer.AsMemory(0, bytesRead));
        task.Increment(bytesRead);
    }

    AnsiConsole.MarkupLine($"[gray]LOG:[/] Finished download of chunk [green]{i}[/]");

    return path;
}

const int DefaultCopyBufferSize = 81920;

var task = ctx.AddTask("Downloading chunks...", maxValue: fileInfos.size);

var chunkPaths = await Task.WhenAll(Enumerable.Range(0, fileInfos.chunksCount).Select(DownloadChunk));
// ...

async Task<string> DownloadChunk(int i)
{
    var path = Path.GetTempFileName();

    using var client = new HttpClient();

    using var response = await client.GetAsync("");
    response.EnsureSuccessStatusCode();

    using var source = await response.Content.ReadAsStreamAsync();
    using var destination = File.OpenWrite(path);

    var bytesRead = 0;
    var buffer = new byte[DefaultCopyBufferSize];

    while ((bytesRead = await source.ReadAsync(buffer)) > 0)
    {
        await destination.WriteAsync(buffer.AsMemory(0, bytesRead));
        task.Increment(bytesRead);
    }

    AnsiConsole.MarkupLine($"[gray]LOG:[/] Finished download of chunk [green]{i}[/]");

    return path;
}

KirioxOP•2w ago

Thank you very much, it may be because I am in .net 4.7 but ReadAsync and WriteAsync also require offset and count arguments

ero•2w ago

sure, you can use WriteAsync(buffer, 0, bytesRead)

KirioxOP•2w ago

Thanks After testing it works really well on small files, but for larger ones it shows me the error: System.OutOfMemoryException, do you know how to fix this? DownloadChunk code after adjustments:

            const int DefaultCopyBufferSize = 81920;

            var client = new HttpClient();

            var response = await client.GetAsync(url);
            response.EnsureSuccessStatusCode();

            var source = await response.Content.ReadAsStreamAsync();
            var destination = File.OpenWrite(destinationPath);

            int bytesRead;
            var buffer = new byte[DefaultCopyBufferSize];

            while ((bytesRead = await source.ReadAsync(buffer, 0, DefaultCopyBufferSize)) > 0)
            {
                await destination.WriteAsync(buffer, 0, bytesRead);
                args.Progress.Increment(bytesRead);
            }

            destination.Close();

            const int DefaultCopyBufferSize = 81920;

            var client = new HttpClient();

            var response = await client.GetAsync(url);
            response.EnsureSuccessStatusCode();

            var source = await response.Content.ReadAsStreamAsync();
            var destination = File.OpenWrite(destinationPath);

            int bytesRead;
            var buffer = new byte[DefaultCopyBufferSize];

            while ((bytesRead = await source.ReadAsync(buffer, 0, DefaultCopyBufferSize)) > 0)
            {
                await destination.WriteAsync(buffer, 0, bytesRead);
                args.Progress.Increment(bytesRead);
            }

            destination.Close();

ero•2w ago

without knowing what causes the exception, no idea. also you should not remove using on the variables which had it those need to be disposed why are you on .net 4.7 anyway if i may ask?

KirioxOP•2w ago

Because it is one of the last versions included in Windows, if it improves the program I can always update It is only available from .net 8

ero•2w ago

this has been a ridiculous reason for years. you can embed the .net runtime in your app. not correct, i'm not sure where you read this

KirioxOP•2w ago

Directly in visual studio (I'm sorry it's in French, it says: The "Using Declarations" functionality is not available in C# 7.3 (it's not my version I don't understand why it says that). Use a version language 8.0 or higher)

ero•2w ago

language version, not .net version

KirioxOP•2w ago

Will I have problems if I change now?

ero•2w ago

you can also use full using statements. you were doing it here. or bump your language version to latest or 8 whatever

KirioxOP•2w ago

How can I do it?

ero•2w ago

migrating from .net framework to .net might be a small hassle, sure. if you have version control set up, it's obviously not going to be an issue. there might be a setting for it in the vs project properties, i don't know i don't use vs i edit project files by hand. which is not gonna be a good idea for netfx project files

KirioxOP•2w ago

"version control"? Like git ?

ero•2w ago

like git if something messes up while migrating (i want to say vs has a button for that too), you can just revert

KirioxOP•2w ago

Which version of .net do you recommend?

ero•2w ago

.net 9, the latest

KirioxOP•2w ago

@ero The migration is complete, so what would be the best method to download now?

ero•2w ago

what do you need to download?

KirioxOP•2w ago

Chunks

ero•2w ago

oh, you don't really need to change anything? make sure you dispose of all the things that are disposable and you're good, really you could maybe reduce allocations a bit more by renting an array

KirioxOP•2w ago

But you told me to update my .net, wasn't it to improve the download?

ero•2w ago

not really, no i asked why you're on .net 4.7. i didn't tell you to update

KirioxOP•2w ago

Wow I'm really stupid Well, it will always be useful What array?

ero•2w ago

the buffer

KirioxOP•2w ago

Increase DefaultCopyBufferSize ?

ero•2w ago

not really? you can keep it as big or small as you want, really. the number i chose i just the default Stream uses in general but you can probably do

int bytesRead;
var buffer = ArrayPool<byte>.Shared.Rent(DefaultCopyBufferSize);
var memory = buffer.AsMemory();

while ((bytesRead = await source.ReadAsync(memory)) > 0)
{
    await destination.WriteAsync(memory.Slice(0, bytesRead));
    args.Progress.Increment(bytesRead);
}

ArrayPool<byte>.Shared.Return(buffer);

int bytesRead;
var buffer = ArrayPool<byte>.Shared.Rent(DefaultCopyBufferSize);
var memory = buffer.AsMemory();

while ((bytesRead = await source.ReadAsync(memory)) > 0)
{
    await destination.WriteAsync(memory.Slice(0, bytesRead));
    args.Progress.Increment(bytesRead);
}

ArrayPool<byte>.Shared.Return(buffer);

KirioxOP•2w ago

Thank you, but it's like last time, if the chunks are too big it shows me that there are exceptions and does not download (I put the using correctly):

Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpIOException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll

Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll
Exception thrown: 'System.Net.Http.HttpIOException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll

ero•2w ago

without the exception message or stack trace, this is not useful

KirioxOP•2w ago

it doesn't crash my program so I don't get a message and when I display call stack there is nothing in it What will this number change on the download?

ero•2w ago

it definitely should, but i don't know the rest of your code.

KirioxOP•2w ago

Isn't there a log file or something similar that would indicate that?

ero•2w ago

if you don't log anything, where should such a log file be created?

KirioxOP•6d ago

@ero It's okay, I managed to solve the problem, thank you very much And I was wondering why don't you make requests to download the chunk in parallel?

ero•6d ago

Hm?

KirioxOP•6d ago

This code downloads end by end, wouldn't it be possible to ensure that it downloads several ends at the same time?

while ((bytesRead = await source.ReadAsync(memory)) > 0)
{
    await destination.WriteAsync(memory[..bytesRead]);
    progress.Increment(bytesRead);
}

while ((bytesRead = await source.ReadAsync(memory)) > 0)
{
    await destination.WriteAsync(memory[..bytesRead]);
    progress.Increment(bytesRead);
}

ero•6d ago

What do you mean "end by end"?

KirioxOP•6d ago

bit by bit

ero•6d ago

Well that's what the default buffer size is for The buffer is simply large enough to fit the whole chunk I assume anyway

KirioxOP•6d ago

Thanks, I don't know much about it 😅

KirioxOP•6d ago

And sometimes I get this error, is it coming from the server?

Gaming

Programming

Most optimized download system