Ticket: Frontend Upload Error with Cloudflare Storage Integration
@Sven Just created a ticket
Question: Frontend issue when using Cloudflare as storage provider. I'm trying to use Cloudflare as storage provider. It works fine for Avatars etc. but when trying to upload an image it doesn't work and I see this error in the Browser console: [Uppy] [17:29:51] Expected a
Attempted: Switched to local, there it works without issues, but I would really prefer for the uploaded assets to be on cloudflare.
Working On: -
Take it from here @Mod 🔥
Question: Frontend issue when using Cloudflare as storage provider. I'm trying to use Cloudflare as storage provider. It works fine for Avatars etc. but when trying to upload an image it doesn't work and I see this error in the Browser console: [Uppy] [17:29:51] Expected a
endpoint option containing a URL, or if you are not using Companion, a custom getUploadParameters implementation.
Attempted: Switched to local, there it works without issues, but I would really prefer for the uploaded assets to be on cloudflare.
Working On: -
Take it from here @Mod 🔥
53 Replies
Did you connect a custom domain?
drop your .env
I'm using the R2.dev subdomain. As I said, the avatars are properly uploaded and displayed, it's just the 'manual' upload of assets where I get the error in the frontend
Yes the avatars have a diffrent way of uploading from the media uploader
most likely you are getting cors errors
can you drop your cors configuration from cloudflare?
And about the .env (or rather the environment of the docker container, as the .env get's ignored -> still have another ticket open about that).
Nothing special there,
( I guess there is nothing else in there that should matter)
Also Iif I would get CORS errors I shouldn't I see that in teh browser console?
All I see is the error from Uppy which doesn't sound like it's related to CORS but rather something wrong with the config
Can you try in AllowedOrigins to put *
instead of the current domains
Tried it, doesn't seem to make any difference
what error are you getting?
from CloudFlare?
Postiz in production uses Uppy with cloudflare
it works
I don't get a error from Cloudflare.
All I get is
[Uppy] [17:29:51] Expected a endpoint option containing a URL, or if you are not using Companion, a custom getUploadParameters implementation.
in CLOUDFLARE_BUCKET_URL, did you put your custom url?
yeah
the r2 url
Should be something like this:
and that works fine, if it's wrong I wouldn't be able to see the avatars
the avatars have a different technic to upload to cloudflare
upload yes, but the URL to deliver the avatars is using the bucket_url
it looks like cloudflare domain has limitations with rate limiting and caching,
Maybe try to connect a custom subdomain?
Afaik I can't just use a subdomain, I first would've to have a domain in teh Cloudflare DNS 😦
I can give you a subdomain to check
try to add sven.git.sn
and give me your records
We can test it now
It's Cloudflare that doesnt' allow it. It needs to manage the domain to be able to use it as a custom domain
Oh
Was a bit surprised by that too
I honestly don't know what is the error
local uploading is not good for you?
yeah, it's running in a VM/Docker and I would rather not using the disk on there to store data
If there is no other way then sure, I'll do it, but I would prefer not to if possible.
So had a look without the container, got the repo set everything up to get it running and I'm seeing the same issue
Good thing is that I now also have the sourcemaps and can actually see whats going on in the frontend JS 😉
@Nevo David
So I'm one step further, as teh error indicated I can fix that error by actually setting a endpoint (in libraries\react-shared-libraries\src\helpers\uppy.upload.ts line:26). Now the question becomes what the correct endpoint is 😉
(maybe it's not needed, but as uppy complains that it's not that it's either that or something else is setup wrong for uppy)
I guess it's supposed to go the presigned route, but for some reason it ends up somewhere were uppy requires the endpoint (which should not be needed for presigned)
@Nevo David Found the actual issue. And it's really a issue in the code.
The problem is that it's setup to do teh presigning for multipart uploads. Uppy by default handles only files with a size of <= 100MiB as multipart, smaller files are send as a single chunk.
So one possible fix is just adding
shouldUseMultipart(file : any) {
return true;
},
to the uppy options. (uppy.upload.ts) to handle all files as multipart. (and this does indeed fixes the issue for me).
Obviously would be better to handle smaller files properly (implement the getUploadParameters callback).
So ended up implementing the getUploadParameters callback, it 'works'... uploads work fine, but they don't appear in Postiz. Any ideas why would be appreciated (the frontend callbacks uppy2.on('complete' ... and uppy2.on('upload-success'.... are executed and data seem ok)
If you want to check what I've done so far:
https://github.com/gitroomhq/postiz-app/compare/main...SvenRH:postiz-app:fix-cloudflare
GitHub
Comparing gitroomhq:main...SvenRH:fix-cloudflare · gitroomhq/postiz...
📨 Schedule social media posts, measure them, exchange with other members and get a lot of help from AI 🚀 - Comparing gitroomhq:main...SvenRH:fix-cloudflare · gitroomhq/postiz-app
And another update on that, just noticed that it saves it to the media server on 'complete-multipart-upload', which isn't called for singlepart uploads (and there seems to be no such callback for singlepart uploads)
So there probably needs to be another endpoint for finishing the singlepart files that needs to be called by the frontend (probably from uppy2.on('upload-success'...)
At this point the question becomes if it's worth even handling singlepart uploads or just just always force multipart to keep things simple
Also here a version with the simpler version that enables multipart uploads for all files
https://github.com/SvenRH/postiz-app/tree/fix-coudflare-altnernate
GitHub
GitHub - SvenRH/postiz-app at fix-coudflare-altnernate
📨 Schedule social media posts, measure them, exchange with other members and get a lot of help from AI 🚀 - GitHub - SvenRH/postiz-app at fix-coudflare-altnernate
This one works for you in production?
It did yesterday in dev. 😉
As I said earlier, taht is a bit more a 'hack' than implementing dingleparts uploads properly, but it's an easy fix
Can you make a pull request?
Sure, was just waiting for your feedback (if you want the simple solution or the more 'correct' one)
Can you elborate why this one works for you and without the multi-part it doesn't work for you?
sure.
When not using multipart the upload works different. Uppy doesnt' do all the multipart upload callbacks arent' used (createMultipartUpload etc.).
In this case it tries to use he getUploadParameters callback (which is not defined) and as this doesn't exist it will try to use the endpoint (so sending via a server).
So the two possible solutions ended up being
1. send everythign as multipart
2. implement getUploadParameters
I don't mind converting it to multi-part, but just interesting
So how does it work in production?
You should be able to tset it yourself. When you try to upload a file smaller tahn 100MB it will try to upload singlepart and that fails
(on platform.postiz.com)
no idea. I can only go by what I see and what teh documentation of uppy says should happen.
Can you create a PR so I can check it? 🙂
give me a few minutes, just wanted to check locally again and messed up something with my environment
PR has been created
Cool, will check it soon 🙂
Do you know why uupy suggest to use multipart for files higher than 100mb?
What's the logic behind it? less calls?
Yeah, multipart requires more calls. each part (I think they are each 5MB) needs to be presigned so causing a lot more calls.
With single chunk there is only on presigning needed.
@xcons I know you are on a small vacation, but I don't have any knowledge about the PROs and CONs happy if you can get a look at this also
You can look at https://github.com/gitroomhq/postiz-app/compare/main...SvenRH:postiz-app:fix-cloudflare you can see that I implemented the presigning for single chunk uploads.
Just the beavour when the upload is finished is not done there (so the added file is never added to postiz)
GitHub
Comparing gitroomhq:main...SvenRH:fix-cloudflare · gitroomhq/postiz...
📨 Schedule social media posts, measure them, exchange with other members and get a lot of help from AI 🚀 - Comparing gitroomhq:main...SvenRH:fix-cloudflare · gitroomhq/postiz-app
I would've finished it, but that would require a bit more changes and raises the question if the handling for completion of multipart uploads shoudl also be changed (would make senes to keep it consistent)
And about pros/cons. The Uppy docs (https://uppy.io/docs/aws-s3/) have a nice summary of it:
Multipart uploads start to become valuable for larger files (100 MiB+) as it uploads a single object as a set of parts. This has certain benefits, such as improved throughput (uploading parts in parallel) and quick recovery from network issues (only the failed parts need to be retried). The downside is request overhead, as it needs to do creation, signing (unless you are signing on the client), and completion requests besides the upload requests. For example, if you are uploading files that are only a couple kilobytes with a 100ms roundtrip latency, you are spending 400ms on overhead and only a few milliseconds on uploading.
Uppy
Sleek, modular open source JavaScript file uploader
Small update. Noticed that the change caused a type error when building a docker container. Fixed the issue and it compiles fine now.
Will this effect only cloudflare, or also local uploading?
only cloudflare
@Nevo David you may have alraedy noticed, someone else tested my fix and it fixed cloudflare for them (https://discord.com/channels/1243410146097500202/1296546635245158442)
Yup, will merge it soon
I see it got merged while waiting for me on feedback - https://github.com/gitroomhq/postiz-app/pull/373/files - but yeah, generally I would have said this was fine.
@xcons in case you haven't read everything. I also started implementing it 'properly' (making singlepart uploads work), but that requires additional changes to handle when the upload has finished which I haven't done (didn't want to make a lot of additonal changes).
@Sven No worries, are you still working on this out of interest?