[Rails + UT] How to use UploadThing outside of JS ecosystem, using Rest API?
Hey, I am building file upload functionality for my Expo+Rails app. I was hoping to use UploadThing for it's robust FE library and the managed s3 experience.
Firstly, the docs aren't friendly for non-JS folks, please look into that - a lot of contradicting/variety of information.
I am calling the
/v6/prepareUpload
endpoint in my /api/uploadthing
api controller but it is not returning presigned urls as one would hope reading the REST api docs. Here is the shape of the output:
I am using expo in the FE, here is the code that calls my /api/uploadthing
api:
My understanding is that the response of prepareUpload should be the result of my uploadthing controller and the FE library will read it and take care of the direct upload.
Please advise what can be done here.99 Replies
The createUpload expects the API response to be
but that's not what the api is returning
hey, it seems like you're trying to use v7 of the package with v6 endpoints. that's probably where your type mismatch expectations are coming from
Here is my ruby code, i think I am calling the right api
The response from
fetch_presigned_urls
is the response that the client receives
anything @julius 🙂
I am paying for the product, hoping to get some support. mailed [email protected] as well, but nothing so farhey sorry - didn't see this. thanks for the ping
brought this up with the team and we were a bit unsure what you were having issues with specifically.
what is the api response that's unexpected?
also I made a python sdk a while back, maybe you can take some inspiration from there.
it was made a bit in between v6 and v7 so it's invoking some endpoints that doesn't exist, but the overall concept might be useful: https://github.com/juliusmarminge/uploadthing-py/blob/main/uploadthing_py/request_handler.py#L99
i was also in the process of upgrading that to v7 when we noticed it's quite hard for others to re-implement the necessary crypto stuff to generate correct signed urls, this is something we're looking to fix. https://github.com/juliusmarminge/uploadthing-py/blob/v7/uploadthing_py/utapi.py#L105-L136
but the v6 endpoints according to the open api spec should definetely match the expected output
i was just a bit confused since
rnHelper.createUpload
is a v7 thing and you were calling v6 endpoints. the v7 frontend is not compatible with v6 backend endpointsCan the frontend libraries (specifically the
createUpload
) directly consume the response of "/v6/prepareUpload".
I was expecting this code to:
1. call by /api/uploadthing api with the files
2. get the presigned urls from "/v6/prepareUpload"
3. upload the file to UT cloud
however, that's not happening. only pt 1 works as expectedthe frontend libraries, yes. but you need a v6 version of the frontend sdk
yes i looked into your python package but couldnt find a working example or directions 🙂
you can use this function https://v6.uploadthing.com/api-reference/react#uploadfiles
React – uploadthing
Docs for the best file uploader to date
okkk that makes sense, does the v7 not have prepareUpload yet?
v7 is completely different, you don't request presigned urls. you upload directly to us https://docs.uploadthing.com/uploading-files
Uploading Files - UploadThing Docs
Uploading files is the first step in the process of uploading files to UploadThing. This page explains the general process of uploading files and how you can use the UploadThing API to upload files. There are two ways to upload files to UploadThing:
that page describes how to build an adapter for v7
yes i was very confused, the rest api docs had the API but this page talked about sqids
but the difficult thing we've noticed is getting the crypto here to match since different langauges does math differently xd https://docs.uploadthing.com/uploading-files#generating-presigned-urls, this is something we're looking into how to fix, but iirc i got the python package to work in this regard
Uploading Files - UploadThing Docs
Uploading files is the first step in the process of uploading files to UploadThing. This page explains the general process of uploading files and how you can use the UploadThing API to upload files. There are two ways to upload files to UploadThing:
i can verify in a bit. just gotta wrap up a tihng
Well, i'll give it a try. Can you guys work on a short guide for a DIY setup for non nextjs folks?
Specifically what I want to know is that once I compute the file key, how will I get that to work with my FE library? Like what function should i call with the filekey/presigned url.
yea definetely something i want to make easier. v7 in theory makes this very trivial, since all your server should do is generate a URL and send it to the frontend
yes - all i could find was a direct PUT request to the presigned URL which I want to avoid because of disconnections, multipart etc etc
you guys should handle that! that's where the money is!
we actually expose resumable uploads instead of multipart for v7
which is what the
createUplaod
you were trying to use is for, that allows you to pause/resume the upload any way you wantcool, i'll give it a shot. in the meantime, if you can give me a short guide, plain md is also fine – that'd be awesome
about to fire up the python v7 branch and see if that worked
this generates valid signed urls: https://github.com/juliusmarminge/uploadthing-py/blob/v7/uploadthing_py/utils.py#L42-L104
GitHub
uploadthing-py/uploadthing_py/utils.py at v7 · juliusmarminge/uploa...
(UNOFFICIAL) Python SDK for UploadThing. Contribute to juliusmarminge/uploadthing-py development by creating an account on GitHub.
then you just return the urls from your server to the client formatted like
awesome,
this will happen on the backend itself right?
yea that you can do concurrently as the files are being uploaded
when the upload is done, what will be the file url? assuming I am sending customId as well
Working with Files - UploadThing Docs
After your files have been uploaded, you will most likely want to do something with them. This page shows how to work with your uploaded files.
the put request also responds with the url
alright, i dont see a rest api for
getSignedURL
(for accessing pvt files), can you guys add that pls/v6/requestFileAccess
yea just was it 🙂 - confused. by v6/v7
thanks! I'll ping you if I come across any issues - otherwise i'll try to create an activestorage adapter for rails folks!
👍 happy to help if you run into issues (although not with the Rails parts since i've never used Rails myself 😅 )
It’s never too late to be a rails convert. xD
Yo, I got back to working on uploads and I am confused about '/callback-result' endpoint in this documentation https://docs.uploadthing.com/uploading-files. What is this? Dont see mentions anywhere else
also, what are these
onUploadComplete
and onUploadError
callbacks? are they backend apis? are they functions exported by the FE library?
awaitServerData: If set to true, the upload request will not respond immediately after file upload, but instead wait for your server to call the /callback-result endpoint with the result of running the onUploadComplete callback. Enable this only if your client needs to get data from the server callback as it will increase the duration of the upload.This is under the
route-metadata
part of the documentation. Does the upload not complete in the FE if this is true or does the route-metadata api call not resolve, 😕it is after running this function: https://docs.uploadthing.com/file-routes#on-upload-complete
File Routes - UploadThing Docs
File Routes is a core concept of UploadThing that defines what your users can upload
if you're doing server uploads you don't need to worry about those
correct - the connection would hold until we receive the data you report through
/callback-result
. This allows you to return some data from onUploadComplete
and access it in onClientUploadComplete
sorry, where is the documentation on callback-result?
Question 1: is it an endpoint i have to build in my app or is it a uploadthing endpoint
Question 2: When i register a bunch of file keys in route-metadata, will i receive a single callback or multiple?
Question 3: When isDev is true, each chunk received will be equivalent to a single callback? - if the answer to 2 is that there are N callbacks for N registered files, then each chunk should correspond to a single callback.
callbacks are sent individually as each file is uploaded. but due to how streaming work a chunk may have multiple json blobs
it's an endpoint on the ingest server, similar to route-metadata
can you point me to the documentation of that endpoint please. and when/where to call it. and the python library doesn't call that endpoint either
the search only points to this page and this page only says that we have to call that endpoint, that's it
oh, is it /v6/serverCallback?
no i don't think we have it documented
can you tell me how and when i have to call it
what are you struggling with?
you call it after you have run the onUploadComplete
alright, and what does the request look like? headers, body, query etc
yea no i never got to building client side adapter for v7 yet
cool
and this endpoint replaces
/v6/serverCallback
endpoint?yes
but it's on the ingest server, not the api
yep
Yo, getting "failed to verify url" error when the client tries to upload the file to the presigned url, how can i debug what's happening.
the url in question:
It is probably failing at hmac signature comparison, can you share the code you guys have for comparing signatures
I've tried everything - using hex hmac signature with and without prefix, using b64 of the binary signature. nothing seems to work (copying your py library / following the uploading files doc)
GitHub
uploadthing/packages/shared/src/crypto.ts at 1b23106bfdbeb0c06d2103...
File uploads for modern web devs. Contribute to pingdotgg/uploadthing development by creating an account on GitHub.
I also added some references here https://docs.uploadthing.com/uploading-files#generating-presigned-urls
Uploading Files - UploadThing Docs
Uploading files is the first step in the process of uploading files to UploadThing. This page explains the general process of uploading files and how you can use the UploadThing API to upload files. There are two ways to upload files to UploadThing:
I got it to work! (mostly) Thanks for the hand holdling.
Upon upload when I request the file, I get a 202 response with almost a nil response when the uploaded file is 10kb
if I reload my page, which re-requests the file, it works fine magically - probably you guys take some time to serve the file?
Hmm how do you mean ”upon upload when I request the file”
It should be available fairly instantly after the upload request closes, although some delay may be expected. You can also preload the image something like this before displaying it on the page https://github.com/pingdotgg/uploadthing/blob/1b23106bfdbeb0c06d21034ef13e5307ec833580/examples/profile-picture/src/app/_components/profile-pic-uploader.tsx#L190
GitHub
uploadthing/examples/profile-picture/src/app/_components/profile-pi...
File uploads for modern web devs. Contribute to pingdotgg/uploadthing development by creating an account on GitHub.
How long delays are you seeing between upload complete -> file loading
I have an API which uploads the image and returns the file url on successful upload which is utfs.io/f/<f-key>. My expo frontend tries to render the file using this url but it fails because utfs.io/f/<f-key> returns an empty response with status 202.
This is what i mean by "upon upload when I request the file" .
I dont know the exact delay but i am seeing the image on a reload post upload. Can you check your backend APIs to see when 202 is returned?
202 is returned from the worker when it’s still uploading
Huh, but the file is uploaded from our server for sure (because my request only resolves when the upload request resolves). Is there a quick way to know the actual status of the file?
I can create a client component which retries when it sees a 202 response, but clearly that is not a proper solution 🙂
Afaict there should be no way to get a 202 if you request the file after the upload request has succeeded. utfs reads db for a field that’s set and awaited before the upload request finishes
here's me rendering the images in the dom as well:
Can you try a server upload? Create an API that:
1. uploads the file
2. returns the file url
Here is how I am uploading the file
It's ruby code, but should be easy to understand. Faraday is a http client library.
Oh you’re doing v6 - let’s think
Yea that won’t have the request resolve before the url is updated, so there’s some race conditions you might run into there
i am using v7 sdk, i am using v6 here because the rest api docs didnt have any docs for v7 server upload
That probably won’t work very well. v6/uploadFiles starts multipart uploads which you then need to complete. This is not compatible with the way the v7 front end SDK works
I added a /v7/prepareUpload endpoint that you can use to get presigned urls for v7 if you were unable to generate working ones
That endpoint will return urls which you can return to the frontend sdk
but I need to do server uploads
Ngl I am so confused with your setup. You say you use ruby and v7 uploadthing sdk? What sdk?
lol - I have both frontend and backend uploads.
Can you make some sequence diagram esque of what you’re doing
ok
That’s fine. You can use v7 for both and the process is very similar.
alright, if i get the presigned url using the prepareUpload api - how do i upload the file?
How is the upload request structured? What are the query params/headers or body I need to send
Server uploads you can see here: https://github.com/juliusmarminge/uploadthing-py/blob/9bfa50b7e93e86601b2c413b3c2b9b11ce58b3d8/uploadthing_py/utapi.py#L106
just generate urls (or request if you can’t generate working ones) and PUT the formdata
GitHub
uploadthing-py/uploadthing_py/utapi.py at 9bfa50b7e93e86601b2c413b3...
(UNOFFICIAL) Python SDK for UploadThing. Contribute to juliusmarminge/uploadthing-py development by creating an account on GitHub.
PrepareUpload is documented in the REST doc
The upload itself is a simple PUT to the URL with the file embedded in formdata
Then you can add resumability by sending a HEAD request first and get an offset, but that’s optional
alright, thanks
I should wrap up the python PR to have client uploads as well - it’s a good reference to have 😅
I suppose i need to register the upload using the route-metadata endpoint, right? Would've been nice if I didn't have to do that.
for my server uploads, i dont want a callback - can i pass nil to callback url and slug in the router-metadata request?
if i am correct, you can just PUT request to V7 api for that
i tried that and it looks like the PUT request waits till the file-key is registered with UT
i think it is still faster than using uploadFile from v6
or atleast should be the same
Okay, I will give you a rundown since you are new on this thread. I am trying to do server uploads and here are the issues I am facing:
With v6 uploadFile, the file is uploaded correctly but it is not available immediately, my client receives an empty response when requesting the file (via an <img tag). This is fixed on immediately reloading my page, so I am assuming the lag is ~1sec.
With v7 prepareUpload, I will have to generate the signed url, register the upload, and then upload the file (3 calls). Since my use case is server uploads, I do not need a callback and hence want to avoid the upload registration entirely. If i dont register the upload, the PUT request never resolves and the file is stuck in uploading state.
Lmk if that makes sense 🙂
you dont need to register the upload if you are not using any FileRoute
and you can generate signed url without extra request
on your own server
do you use FileRoute?
I am on rails, there is no concept of file route.
you dont need to register the upload if you are not using any FileRouteok this is news, will try that
and you can generate signed url without extra requestI am aware of this but the crypto is not matching with what UT server expects and I gave up and used prepareUpload
oh the last one relatable, i am facing the same situation rn
I am giving up on this for now and adding a very very bad fix for now - hopefully the UT team can give me a better solution
you don't for server uploads
flow is documented here https://docs.uploadthing.com/uploading-files#server-side-uploads
Uploading Files - UploadThing Docs
Uploading files is the first step in the process of uploading files to UploadThing. This page explains the general process of uploading files and how you can use the UploadThing API to upload files. There are two ways to upload files to UploadThing:
what language? i added some reference examples for python, php and go that i've tested myself to work
i think the problem is not with fileKey but with algorithm 256. But not sure.
https://ptb.discord.com/channels/966627436387266600/1302368823630102579
what language?
Java
I can give you the code that generates signedKey if it would help
sure - i don't have a way to run java rn though
if it will not bother you, you can run it online, because i use build in classes, let me chekc if its possible myself first
send away i'll try to look tomorrow. gotta go to sleep now
absolutely, i appreciate it
has anyone used the REST API to send things from python?
I have an extremely edge case that I cannot use any external libraries so I'm converting everything from the Rest API but after I send the files correctly with /v6/uploadFiles (at least my understanding) is that I need to call the /v6/completeMultipart but I'm having issues understanding some keys on the api:
-How can I get the uploadId
-If i'm uploading only one file, is the default config enough?
you should probably post a new question because this thread is mostly about v7.
So sorry! I though this was on main!