cdn questions for media
Project ID : dba6cd6e-5ea1-494c-b77f-81377e24fbde
Hello everyone,
I have been a hobby plan subscriber before, then I tried my already purchased server on A2 hosting with my project. It seemed very limited compared to what I had here. Now I am back.
I have a project with FastAPI backend and React Native Mobile Expo Go on the front end.
It works fine, expo consumes the API I have here fine.
However, I have many static files including videos ( 115 of them so far ) 5-10mb range and lesson posters ( 150KB range ). I would like to use CDN if possible.
I had setup a media volume here, thanks to Brody for that BTW, which works fine.
I have a domain that I could use to point this server and use free cloudflare CDN to test.
I can pay a little if necessary.
I am thinking as the app grows, I might end up having 2-300 videos all around 5 to 20mb range.
This is for a jazz education software.
What is my best option?
I am thinking my options are...
1. Leave API here and have it point to other servers with CDN for videos
I don't like this option very much since it requires a not so small re-write for API.
2. Use the volume I have here and get a CDN for it somehow ( may be using my custom domain to point here ? )
3. Use a bigger service like Azure or AWS for static serving files?
Which I do not like since they can get expensive petty quick.
4. Do nothing and leave everything as is and hope performance will be OK once I go live with my app.
I am expecting/hoping 50 to 100 users in the next few months and may be 1000 users in a year?
If that happens I will have to upgrade my account here from hobby to pro of course but I don't how I could prepare to grow ( if it happens.. )
I do not have any performance issues now BTW.
Any suggestions?
61 Replies
Project ID:
dba6cd6e-5ea1-494c-b77f-81377e24fbdefor option 1, how not small of a rewrite would it be? you are using a storage engine that ties into postgres and uses local storage (a volume when on railway)
does this storage engine provide an S3 layer? that way you could use cloudflare R2 with a custom domain and then you'd have a no egress bucket and files would be cached
I guess I could do that. Now that I think about it, mysql has a record of the file name, to get started, I could modify the the db records adding prefix or suffixes, and have the records point to other server with cloudflare. This is rather a minimal rewrite. May be not even a rewrite. This would save me a lot of network bandwidth as well I think. Let me think about this a bit but thanks Brody.
I dont know if you remember but you helped me get my media drive up and running for videos and streaming stuff. You even did an example with fastapi
yes I do!
I forget the fine details of it though, so if the storage engine supports S3 + MySQL as a backend, it should be relatively simple, store the files in a cloudflare R2 bucket, and using a custom domain to get the files cached
https://developers.cloudflare.com/r2/buckets/public-buckets/#caching
Yes, that works. I'll check that out. I event have the files on an other server too with a custom domain. So, I could use that server for videos. I just need to integrate the admin panel to save the video to the other server. If integrating admin does not work out, I know admin that I use supports S3 so, I'll what you suggested now. Thanks.
When you initially helped me, my files werenot surviving between deploys since I used the wrong mount point. Now I use /app/media.. Also static files did not have both slashes etc.. You made an example for it. Thanks for that
always happy to help!
Thanks Brody 🙂
no problem! 🙂
let me know what you end up doing!
I'll post it here when I make a move. I am reading the R2 option looks promising. I use SQL Alchemy Admin with FastAPI which has S3 Storage option, so it should be compatible with what you suggested ( Cloudflare R2 )
Working with Files and Images - SQLAlchemy Admin
Flexible admin dashboard for SQLAlchemy.
perfect, it does indeed have an S3 storage backend!
Unfortunately, I could not set this up after spending a full day. R2 docs are not that good. I could not find any examples doing this with fastapi_storage. I found one example for regular S3. I will check that out and if that does not work, then I have to find another solution for CDN.
cloudflare R2 is AWS S3 compatible
so yes you'd want to use the S3 backend
I managed to get file info and download a file from the bucket. I was working on it with an example FastAPI project. If I can manage everything, including PresignedUrl etc... , I will post the github example here. Let's see how it's gonna workout.
awesome, I look forward to that!
Here are the examples...
1. This one shows FastAPI, SqlAdmin, CloudFlare R2 integration. boto3 wrapped with another class called S3Storage for easier integration to admin.
GitHub
GitHub - yavuzakyazici/s3_example
Contribute to yavuzakyazici/s3_example development by creating an account on GitHub.
Here is the second example with regular fastapi upload/download file without any admin integration. This one is done with boto3 library..
GitHub
GitHub - yavuzakyazici/cf_s3_fastapi
Contribute to yavuzakyazici/cf_s3_fastapi development by creating an account on GitHub.
you work fast!
I'm trying 🙂 Now, I will look into generating PresignedUrl for each video with 20 seconds expiration. This way, I can keep the bucket access private and cache CDN content. When I manage to work it, I'll add that to examples.
So I updated boto3 version with PresignedUrl method.
If you don't want your bucket contents public, you might just want your client app display it or whatever the reason you might have. You can just return a PresignedUrl expires in seconds. Once it is consumed by your client app thru API, it expires 🙂 Nice isn't it ?
GitHub
GitHub - yavuzakyazici/cf_s3_fastapi
Contribute to yavuzakyazici/cf_s3_fastapi development by creating an account on GitHub.
awesome work!
looks like this is turning out to be exactly what you want?
Yes it did. Thanks to your suggestion 🙂 I was not expecting it to be this convenient. One thing I could not do was adding CNAME record to my custom domain at A2 hosting. I have to get suport. Adding custom domain to CloudFlare was very very easy though.
yeah A2 likely wouldnt support root level cnames, best to use cloudflare for your nameservers
I'll ask and see what they say 🙂
please note you can not use A or ALIAS types despite what they may say, that will break your site
OK. Good to know thanks. I may not even need to get a custom domain for api. CDN was done and helps caching. Having a custom domain for the api not a must I think
yeah thats up to you, but switching to cloudflare's nameservers is incredibly easy, they even copy over your dns settings for you!
Yes, I noticed it was very easy and It took like a minute to update.
do you have different root domains for the cdn and what you would use for the api?
yes.
they were bot subdomains. CloudFlare worked. But I did it from cloudflare settings. I did not need to touch A2 Panel.
To change the api railway instructs to add a CNAME but A2 does not let me the way it is suggested. So, did not work. But I'll ask support ands see what they say
you tried in the style of api.domain.com? any dns provider should support cnames on a subdomain
api.domain.com? yes exactly..
can you try again and show me what a2 says?
oki.. I'll try and take screenshots as well
It says dns zone is invalid
have you tried just "api" for the name?
I did. It does not let me. If I reverse the values. It accepts it but It adds stuff
like this
have you tried "api" as the name, and then the record "domain.up.railway.app." (with a period at the end)
let me try now
Suggested domain at railway is yynudahm.up.railway.app
please see my message
replace domain with my domain right?
It does not accept it
correct, please show me what you tried?
I tried this
please try what I suggested
when I type api, it completes it to what you saw
may I ask what you removed the image?
Too many images.
I can put it back in
I think discord can handle a few more images lol
A2 too complicated, move nameservers to cloudflare, there's no reason to stay with A2 anyway right?
No...
I'll look into it. Let me talk to support first.
may I ask why you are inclined to stay with A2?
I'm not inclined.
I paid the services up front for 2 yrs
ah I see
It was before I had anything here
long before 🙂
gotcha
Let me see what happens. I am happy the way things are if it does not work out. I might move dns to cloudflare if I can
sounds good
It turns out, DNS was hosted at CloudFlare. I managed to change the CNAME there but now I get too many redirects error to reach the subdomain
Thanks. Fixed it right away! 🙂 Works now.
no problem!