✅ Suggestion on file upload?
When uploading file to azure storage. Should I upload the files through the server (api)? or is there another way. The thing that scares me is that if multiple people upload at the same time than we will have a memory issue. The clients in this case are mobile apps? Any insights would be appreciated.
30 Replies
why does an upload stream worries you about memory?
@dont If the upload process is done through the server. The mobile apps will send the files to the server (API), and then the server will upload the files to Azure Storage.
If there are multiple people uploading files at the same time, there will of course be a load on the server.
it will have a load but memory issue would depend on how the file is being sent, if you use a stream then it will send chunks at a time and not the whole file thus I dont see how that would cause a memory issue
what's your spec, like 5, 10, 15 uploads contemporaneously?
50? 100?
Still small chunks can quickly lead to large amount.
I don't really know as of now. It depends on how many people download the apps.
no, because they dont remain in memory, they are sent as chunks to lessen the memory to disk process
but if u have a 1m users all sending files a the same time u have to account for that 1m*the chunk size etc
because that would be part of your load
I guess, that does make sense. The issue is that I need to validate the content before. I could use
Shared Access Signatures. To allow the clients to upload the files directly to a specific blob.validate in what way?
like u want to ensure it is a type of mime or
See if there is any sexual contents etc.
i guess so, but you still have to build an initial system, if you say i go big and support 100_000 uploads from the beginning it's going to be pretty different from 100 uploads
so we are talking about video/images only?
Exactly
and currently how do u check that manually watching?
I guess, maybe I'm thinking to far ahead.
I do a check through the client.
that doesn't say much
On the server side, I was thinking of using
Azure content moderator.but that would be after u passed the file down to azure no?
I use an AI library that does the validation.
you can give token to decidated space to user. Let him upload to cloud
then you can have some job that moves from there files to correct places
short lived token of course
If I use
Shared Access Signatures, than yes. If I go with the manual process. I can validate the content before storing it.but you said you're going to use azure's content safety
so you're double validating ?
Yes, I'm double validating. I don't really trust the validation on the client side. There are always people that will try and bypass.
well then why not just letting azure handle it
or u mean u dont trust azure's ai content safety
You mean like, create a container that holds the not yet validated content. Than directly after the user uploads the content through
Shared Access Signatures. I call the api, and tell it to validate and than move it to the correct container?any upload goes to a moderated container by default
and are moved to the correct place once validated
I do trust the azure's ai. I was just hesitant on how to upload the files to prevent issues.
I mean given everything u said I think that would be the most conventional way to avoid issues but I could be wrong
any new upload goes to a moderated container that verifies and moves the files to a safe container
Yeah this is probably the best way
Exactly.
Thanks for your insight, guys. I really appreciate it.
mobile app it's calling cloud host (azure) to do that he needs token. You can create seperate container and then give it access to correct path (for each user so other users won't be able to see / write other files)
Then you might have some background job that moves is to correct place (if you want to have some confirmation done, that can be also there, for moderation purposes)
Sounds like a plan.