Tracking request/useMutation progress
I have a tRPC endpoint that takes in images in base64, and in the frontend i want the user to be able to watch the progress of the files being uploaded. Not sure on how to really do that. is there a function im missing? I tried looking at the documentation and cant find what im looking for.
105 Replies
Uhh like a progress bar?
if you upload images as base64 u wont be able to track that. U can progress track the client side upload. Usually user has to upload the file first then u upload it after its fully loaded into the client side
yeah pretty much my goal
even tho im deploying in the US
there are still parts in the US where they have slow wifi
i couldnt figure out how to do file(image) uploads
through tRPC
File uploads can be done with native js/ts. https://dev.to/jbrocher/react-tips-tricks-uploading-a-file-with-a-progress-bar-3m5p
DEV Community
React Tips & Tricks: Uploading a File With A Progress Bar
Forms are often tricky to get right with React. While there are great libraries like formik or React...
the whole point is to use this within tRPC
Im not sure if tRPC supports that
yeah thats what i feared
I think its because its using json rpc so you can't use it on its own to do the file uploading cause the binary data might get morphed (just my guess?). I do puts to s3 presigned urls on the client side and it works well
TomDoesTech
YouTube
Use Presigned PUT URLs to Easily Upload Files to AWS S3
Theo's Tweet: https://twitter.com/t3dotgg/status/1564019039877271552
Repository: https://github.com/TomDoesTech/pre-signed-put-url
0:00 Intro
0:43 Data flow
1:45 Bootstrap application
2:33 S3 bucket setup
5:07 Get PUT URL handler
10:36 Form
15:24 Does it work?
🌎 Follow me here:
Discord: https://discord.gg/4ae2Esm6P7
Twitter: https://twitter.c...
hmm that might be the move
just upload the files client side
it works very well
How big are the files you want to upload?
10mb 5-10 images
should be able to do it without multi part uploads i think so that makes things easier
10mb is the limit
for progress just update some bit of state and render it if you go down this route
"Uploaded 1/5 files" yadda yadda
ok thanks ill have a look
im pretty sure this should work with r2
thanks for that tho
r2 is a object blob storage ?
s3 alternative
yeah should be fine
u still use the aws-sdk
the directions won't be 1:1 but basically you need an endpoint you can dump data into (s3 presigned url is basically that) and do thing
Lemme see what I used
I've been doing image uploads using the rust beta aws sdk
haha
how is that going
ik i wrote the stuff for our frontend but its probably the aws sdk
works 😄
yeah it is the aws-sdk
how doeds the presigned url work
So I have a mutation that'll generate a link for me for 5 or so minutes
Thats the "presigned url"
User fills out the form which has file inputs and then you just do a casual put to the url
and you are done
so if nothing gets uploaded or its canceled it expires
yep
thats perfect
the generation is a protected procedure too
🔥
ok that is nice
You shouldn't need to use the sdk on the client side at all. Just mentioning this so you don't leak your aws keys
the video i linked is good
yep
i can have a server side function right
im guessing thats the idea
You just fetch/axios/http request to the url
anything running in the trpc land is safe
so the presigned generation can use the sdk cause it lives on the server
the client side just gets the url back, PUT request to it with your data and done
alright thanks a lot
will have a look at the video and the aws-sdk docs
thanks a lot for that
i completely forgot about it
Presigned URLs · Cloudflare R2 docs
Presigned URLs are an S3 concept for sharing direct access to your bucket without revealing your token secret. A presigned URL authorizes anyone with …
over engineery moment
have fun 
thank u appreciate it
would it make sense to zip files after you verify they are images
then make a presigned url and upload that?
that way instead of making 5-10 s3 calls
u just make 1
I personally wouldn’t lose sleep over the s3 calls but you can zip on the clientside using jszip from my googling
You will have to unpack it at some point if you want to work with it
yes ofcourse i already do that
and also use jszip
If it works it’s fine
just making sure there is no weird shit that can possible happen
surely
can they?
i mean they will get a presigned url and its gonna upload a zip file
but is there a way to make sure the presigned url gets a zip file that only has 5-10 images
thats a bit sketch actually
you could do checksums to ensure the files are the same on both sides
cause they would be able to see the presigned url and upload anything
You can limit what content types they can upload and sizes
through the presigned urls correct
I think both? My form limits the choices a user can dump on to the web app but you also can provide meta data to the thing your uploading on the sdk
thats the thing
what if someone uploads a zip bomb
the input is already just for images
but i guess what my question is
ask for 5 - 10 images then zip it yourself
no worry about zip bombs
would they be able to get the presigned url or not
Yeah they can if they want to, trpc returns http which is easily snooped
hmm
this needs some more thought
Security best practices for Amazon S3 - Amazon Simple Storage Service
Learn about guidelines and best practices for addressing security issues in Amazon S3.
ill have a look at that
Are these users signed up btw?
as in you are limiting uploads behind a protected procedure
yes
authed
by clerk
shouldnt be worried then i guess right?
cant i have custom auth header
for the presigned url
nah fuck it i will just do the 5-10 s3 calls
i dont think u can really verify the .zip images
within the presigned url
s3 is pretty cheap just make sure you don't lose the uploads cause asking for reupload is awkward
You would use a lambda or something to verify the upload
You can listen for changes in a bucket/new uploads and spin off handlers based on that
if you wanted to get fancy
hmmm
what if i had a multipart image upload
for the presigned url
and the output is zipped
multipart is something specific
where you have a extremely big file (something you can't dump into ram entirely) and you have to upload in parts
your 50mb is nothing
hmm
well something to do with uploading all images into 1 endpoint and it being zipped
i think i just gotta do a nextapi protected route
and call it a day
this is so stupid
why is it so annoying in JS?
typescript is becoming the standard 😉
😂
i am using TS
ðŸ˜
but handling files is the most BS activity ever
Oh...
its so easy and simple in python
AND rust
yeah i prefer rust over everything
maybe after i deploy ill look into using rust in the backend ontop of nextjs
just not sure how i would/can use clerk
u using clerk for auth?
yeah
cant be asked to code everything from scratch just yet
idk there are no providers for auth for rust and js. jwt and some other standard auth stuff can be implemented
i know that you can use clerk api directly
then thats possible
but u would have to write everything if there is no library
i mean
https://crates.io/crates/clerk-rs
there is one
the only thing i'd really need is to verify the auth header
interesting
for protected api routes
yah thats normal
but thats about it
i guess
usually people just make middleware for auth stuff and check anything there
yeah i know
but still some protected routes
ya
would be very interesting
i used expressjs alot of auth
might whip something up with fastapi
jwt
yeah i did the same but in fastapi
but i just couldnt be asked to always have to reset everything up for every new project
yea u can do it in any http server framework
email verification, sms verification, login, oauth
clerk just lets me plug n play
very nice
and IMO pretty cheap
never used clerk
i use nextauth for auth
i hate js and ts
i tried to get into nextauth
and i just couldnt
i love ts. after i got decent with rust. Ts was so much easier
im also decent with rust
just hate ts
are u using nextjs?
yes
u can use middleware for protected pages
yeah i know
thats what im doing
thats not my problem ðŸ˜
my problem is file uploading
rip
it works rn
im converting to base64 and using tRPC
but doing so doesnt let me track the progress
yikes
that will be hard probably
idk how can you track a base64 string xd
tracking how many bytes are loaded is easier
as in
im tryna figure out how much bytes are loaded while sending
if that makes sense
thats with native js browser file input field
explain please
ok
no trpc.
gonna see
FileReader: progress event - Web APIs | MDN
The progress event is fired periodically as the FileReader reads data.
thanks gonna have a look
i might just not add a progress bar in the end