C#•4mo ago

Background upload of an IFormFile to Azure blob storage

I have a few endpoints that are dealing with files in my app. Some of them may accept IFormFileCollection or regular IFormFile. During the request hanlding content of those files is being uploaded to azure blob storage. The problem I encountered is that process of uploading to the blob storage can take 3-5 seconds. Are there any options to upload files in the background and return after persisting data to the db? As far as I understand it would be a bad practice to reference/use IFormFile after returning response the the client, so just creating a background job and passing it as a reference would not be an option

10 Replies

SwaggerLife•4mo ago

@Fern I would personally say, don't worry about the time. You are uploading files at the end of the day. However, you could change the TransferOptions and make it faster that way.

/// <summary>
///     Handles the uploading of a file to storage.
/// </summary>
public class UploadFileToStorageHandler(BlobServiceClient client)
    : IRequestHandler<UploadFileToStorageCommand, Result<Uri>>
{
    public async Task<Result<Uri>> Handle(UploadFileToStorageCommand request, CancellationToken cancellationToken)
    {
        try
        {
            // Get the BlobContainerClient corresponding to the container name
            var containerClient = client.GetBlobContainerClient(request.ContainerName);

            // Ensure that the container exists, creating it if it doesn't
            await containerClient.CreateIfNotExistsAsync(cancellationToken: cancellationToken);

            var blobClient = containerClient.GetBlockBlobClient(request.BlobName);

            // Create a BlobUploadOptions to specify the parameters for uploading to a Blob.
            var blobUploadOptions = new BlobUploadOptions
            {
                TransferOptions = new StorageTransferOptions
                {
                    MaximumConcurrency = 16,
                    MaximumTransferSize = 4 * 1024 * 1024
                },
                HttpHeaders = new BlobHttpHeaders
                {
                    ContentType = request.ContentType
                }
            };

            await blobClient.UploadAsync(request.Stream, blobUploadOptions, cancellationToken);
            return Result<Uri>.Success(blobClient.Uri);
        }
        catch (Exception e)
        {
            // If an exception occurs during upload, return failure result.
            return Result<Uri>.Failure("Something unexpected occured.", [e.ToString()], 500);
        }
    }
}

/// <summary>
///     Handles the uploading of a file to storage.
/// </summary>
public class UploadFileToStorageHandler(BlobServiceClient client)
    : IRequestHandler<UploadFileToStorageCommand, Result<Uri>>
{
    public async Task<Result<Uri>> Handle(UploadFileToStorageCommand request, CancellationToken cancellationToken)
    {
        try
        {
            // Get the BlobContainerClient corresponding to the container name
            var containerClient = client.GetBlobContainerClient(request.ContainerName);

            // Ensure that the container exists, creating it if it doesn't
            await containerClient.CreateIfNotExistsAsync(cancellationToken: cancellationToken);

            var blobClient = containerClient.GetBlockBlobClient(request.BlobName);

            // Create a BlobUploadOptions to specify the parameters for uploading to a Blob.
            var blobUploadOptions = new BlobUploadOptions
            {
                TransferOptions = new StorageTransferOptions
                {
                    MaximumConcurrency = 16,
                    MaximumTransferSize = 4 * 1024 * 1024
                },
                HttpHeaders = new BlobHttpHeaders
                {
                    ContentType = request.ContentType
                }
            };

            await blobClient.UploadAsync(request.Stream, blobUploadOptions, cancellationToken);
            return Result<Uri>.Success(blobClient.Uri);
        }
        catch (Exception e)
        {
            // If an exception occurs during upload, return failure result.
            return Result<Uri>.Failure("Something unexpected occured.", [e.ToString()], 500);
        }
    }
}

Here is an example.

cumslvt13•4mo ago

The problem is with UI part that needs to wait for the end of the response. What I'd really like to achieve is to quickly return response with generated id's for UI to update optimistically and later notify the UI via signalR

SwaggerLife•4mo ago

@Fern Don't make it complicated. Show a nice loading spinner. There really is no need for signalR etc. You can later on, improve the feature. By making it faster etc.

cumslvt13•4mo ago

Application is sort of OneDrive, so there's need to notify other people that are viewing the same page that something changed

SwaggerLife•4mo ago

There you have an additional reason not to really worry about time. People usually understand, that uploading files does take time.

cumslvt13•4mo ago

I don't really think that 2-3 files up to in sum 3mb should take 5 seconds. But thanks for the hint with transfer options, I'll look into that

SwaggerLife•4mo ago

The reason, I'm telling you this is. I have often found myself asking this questions which is good generally but I have noticed that I end up spending so much time on it that I don't get anything done. Play a bit with StorageTransferOptions. Start with a simple spinner for the time being. If you still feel like it's taking to much time then try and implement a better approach. At least at that point, you have something to go from.

cumslvt13•4mo ago

I'm not starting from scratch, more like a refactoring existing service with improvements of design flaws. One point that customer wants to improve is overall speed and responsiveness of the app

D.Mentia•4mo ago

yes, do this. A background task would work but you need to be able to guarantee it will complete, through retries and even if your app crashes halfway through, when it starts back up it should try again The usual way to make that kinda stuff happen is through a message bus and potentially a separate service doing the uploads though looking back at this I think I was misunderstanding... you shouldn't be uploading an IFormFile to Azure, you should be saving it to local disk and telling them you have it, then you fire off a background task to upload it (now a normal file, not IFormFile) to Azure and delete it from disk once successful The existence of the file on disk, probably named with some correlating info in your db, is enough that you could re-attempt that upload on server startup or whenever, if it failed for whatever reason. And if the client needs the file some time before you've uploaded it, you have it to give to them

SwaggerLife•4mo ago

You mean a SAS url?

Gaming

Programming

Background upload of an IFormFile to Azure blob storage