C#•2w ago

Downloading large file

Hello, I need to be able to download files up to 400Mb asynchronously and track the download in a progress bar. I have this code except that it doesn't work totally, on 50Mb files it's fine but 130Mb it crashes and moreover it takes a while before starting the download. So I wanted to know if this was the right method and if it was possible to optimize it?

c#
int DefaultCopyBufferSize = 81920;
string destinationPath = Path.GetTempFileName();
using var client = new HttpClient();

using var response = await client.GetAsync(url);
response.EnsureSuccessStatusCode();
using var source = await response.Content.ReadAsStreamAsync();
using var destination = File.OpenWrite(destinationPath);

int bytesRead;
var buffer = ArrayPool<byte>.Shared.Rent(DefaultCopyBufferSize);
var memory = buffer.AsMemory();

while ((bytesRead = await source.ReadAsync(memory)) > 0)
{
    await destination.WriteAsync(memory[..bytesRead]);
    progress.Increment(bytesRead);
}

c#
int DefaultCopyBufferSize = 81920;
string destinationPath = Path.GetTempFileName();
using var client = new HttpClient();

using var response = await client.GetAsync(url);
response.EnsureSuccessStatusCode();
using var source = await response.Content.ReadAsStreamAsync();
using var destination = File.OpenWrite(destinationPath);

int bytesRead;
var buffer = ArrayPool<byte>.Shared.Rent(DefaultCopyBufferSize);
var memory = buffer.AsMemory();

while ((bytesRead = await source.ReadAsync(memory)) > 0)
{
    await destination.WriteAsync(memory[..bytesRead]);
    progress.Increment(bytesRead);
}

46 Replies

canton7•2w ago

You need to pass HttpCompletionOption.ResponseHeadersRead I think, stop it downloading everything at once? Also that buffer you get back from the ArrayPool might be significantly bigger than what you asked for There's no point trying to asynchronously write to a file if the file wasn't opened in async mode - it does nothing

KirioxOP•2w ago

However I tried to use Write instead and it was much slower How do I know to what extent?

canton7•2w ago

It shouldn't be. It still does an sync write, but it does it on a ThreadPool thread You can look at the size of the Memory you get back, but it's unspecified. You can slice the Memory to the right size if you depend on that

asdf•2w ago

How is that relevant, it should just potentially increase performance

canton7•2w ago

Also you might end up calling progress.Increment for every byte received, if ReadAsync ends up giving them to you one by one, which might well overload other parts of your app They might end up reporting progress too infrequently

asdf•2w ago

You could make sure the whole buffer is filled before writing it to the file

KirioxOP•2w ago

Well, I have the impression that the progress only increases at the end of the file download.

canton7•2w ago

Also using a buffer that's too large might not play well with the cache Did you try using ResponseHeadersRead? That was my first suggestion, and probably the cause of your problem The other things I mentioned are just other problems I spotted

KirioxOP•2w ago

It's miraculous, thank you very much

canton7•2w ago

$close

MODiX•2w ago

If you have no further questions, please use /close to mark the forum thread as answered

canton7•2w ago

Glad to hear!

KirioxOP•2w ago

Um, in fact it starts faster and the progress bar reacts well but on the other hand I have the impression that overall it's slower

canton7•2w ago

It probably will be: it's doing more work than it was before Try using a smaller buffer (1024 or so), and keep calling ReadAsync until the buffer is (nearly) full (before calling progress.Increment and flushing the buffer to file)

KirioxOP•2w ago

And even if it crashes I still have these errors in the console Exception thrown: 'System.IO.IOException' in System.Private.CoreLib.dll Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll Exception thrown: 'System.Net.Http.HttpRequestException' in System.Net.Http.dll Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll Exception thrown: 'System.Threading.Tasks.TaskCanceledException' in System.Private.CoreLib.dll Exception thrown: 'System.Threading.Tasks.TaskCanceledException' in System.Private.CoreLib.dll Exception thrown: 'System.Threading.Tasks.TaskCanceledException' in System.Net.Http.dll Exception thrown: 'System.Threading.Tasks.TaskCanceledException' in System.Private.CoreLib.dll Exception thrown: 'System.OperationCanceledException' in System.Private.CoreLib.dll Exception thrown: 'System.OperationCanceledException' in System.Private.CoreLib.dll Exception thrown: 'System.OperationCanceledException' in System.Private.CoreLib.dll

canton7•2w ago

Also make sure you're not on a UI thread or anything - I don't know how this method is being called

KirioxOP•2w ago

From the main of a console application

canton7•2w ago

That looks exciting. Open the exceptions window, tell the debugger to break when those exceptions happen, see what's going on

KirioxOP•2w ago

How can I open it?

canton7•2w ago

Debug - Windows iirc?

KirioxOP•2w ago

I don't have anything close to it (sorry it's in French)

canton7•2w ago

Second one

KirioxOP•2w ago

Does this mean that the problem comes from the remote server?

canton7•2w ago

Probably?

KirioxOP•2w ago

Is there a way to confirm this?

canton7•2w ago

I'd probably reach for Wireshark, see what's actually going on

KirioxOP•2w ago

I installed it but I have no idea how to do it

canton7•2w ago

I'm afraid I'm a bit busy. You might have to learn a bit how tcp works

KirioxOP•2w ago

Thanks, I'll take care of other things in the meantime

Unknown User•2w ago

Message Not Public

KirioxOP•2w ago

No, but I know the owner

Unknown User•2w ago

Message Not Public

KirioxOP•2w ago

But is it something that happens only during long sessions of use?

Unknown User•2w ago

Message Not Public

KirioxOP•2w ago

Could it be caused by cloudflare?

Unknown User•2w ago

Message Not Public

KirioxOP•2w ago

How can I do it?

canton7•2w ago

Which bit of that are you struggling with?

KirioxOP•2w ago

keep calling ReadAsync until the buffer is (nearly) full

canton7•2w ago

What about that don't you understand?

KirioxOP•2w ago

Yes, I understand roughly what that means, I've just never worked with memory management and so I don't know how to do that in C#

canton7•2w ago

Call ReadAsync in a loop until the buffer it's reading into is (nearly) full

KirioxOP•2w ago

But at this point, how can I continue my condition in the other loop?

while ((bytesRead = await source.ReadAsync(memory)) > 0)

while ((bytesRead = await source.ReadAsync(memory)) > 0)

I try this, but it don't work

c#
int totalBytesRead = 0;
while (true)
{
    int bytesRead = await source.ReadAsync(buffer.AsMemory(totalBytesRead, DefaultCopyBufferSize - totalBytesRead));
    if (bytesRead == 0)
        break;

    totalBytesRead += bytesRead;
    if (totalBytesRead >= DefaultCopyBufferSize || bytesRead == 0)
    {
        await destination.WriteAsync(buffer.AsMemory(0, totalBytesRead));
        progress.Increment(totalBytesRead);
        totalBytesRead = 0;
    }
}

c#
int totalBytesRead = 0;
while (true)
{
    int bytesRead = await source.ReadAsync(buffer.AsMemory(totalBytesRead, DefaultCopyBufferSize - totalBytesRead));
    if (bytesRead == 0)
        break;

    totalBytesRead += bytesRead;
    if (totalBytesRead >= DefaultCopyBufferSize || bytesRead == 0)
    {
        await destination.WriteAsync(buffer.AsMemory(0, totalBytesRead));
        progress.Increment(totalBytesRead);
        totalBytesRead = 0;
    }
}

canton7•2w ago

while (true)
{
    // Fill the buffer
    int bytesReadIntoBuffer = 0;
    int bytesRead;
    while (bytesRead = await source.ReadAsync(memory[bytesReadIntoBuffer..]) > 0)
    {
        bytesReadIntoBuffer += bytesRead;
        // Have we filled the buffer? Allow a small amount of space at the end
        if (bytesReadIntoBuffer > memory.Length - 128)
        {
            break;
        }
    }

    if (bytesRead == 0)
    {
        // We're done reading
        break;
    }

    // Copy from the buffer into the stream
    await destionation.WriteAsync(memory[..bytesReadIntoBuffer]);
    progress.Increment(bytesReadIntoBuffer);
}

while (true)
{
    // Fill the buffer
    int bytesReadIntoBuffer = 0;
    int bytesRead;
    while (bytesRead = await source.ReadAsync(memory[bytesReadIntoBuffer..]) > 0)
    {
        bytesReadIntoBuffer += bytesRead;
        // Have we filled the buffer? Allow a small amount of space at the end
        if (bytesReadIntoBuffer > memory.Length - 128)
        {
            break;
        }
    }

    if (bytesRead == 0)
    {
        // We're done reading
        break;
    }

    // Copy from the buffer into the stream
    await destionation.WriteAsync(memory[..bytesReadIntoBuffer]);
    progress.Increment(bytesReadIntoBuffer);
}

Something like that?

KirioxOP•2w ago

Yes, thank you, but I have the impression that the download is even slower Could this be impacted by the fact that my program downloads lots of files at the same time?

KirioxOP•2w ago

Oh, a new error to add to my pokedex

Gaming

Programming

Downloading large file

Did you find this page helpful?