How to mass upload a list of files to blob storage?
I have a large number of files I need to upload to a blob storage, and I'm currently using the
BlobClient.UploadAsync(filePath, options); method to accomplish this. However, it's too slow, probably because I'm uploading files one by one.
I've seen some recommend the Microsoft.Azure.Storage.DataMovement for this use case online, that being a bulk upload of files.
However, my use case requires me to filter the files by matching with a CSV file first, which maps the current file names to the new names, and to avoid unnecessary files.
Has anyone have had to deal with a similar use case before? Bulk uploading files, having to filter them first? I'd rather not have to copy all the files to a new directory if I could avoid it.1 Reply
Without knowing too much about azure APIs, is there a reason you can't do multiple
BlobClient.UploadAsync calls and wait for them? Batch into 10s, 20s, etc
The other option I would think of is to use Microsoft.Azure.Storage.DataMovement to transfer a folder by first copying the files you want to transfer there, then uploading the whole folder