S3 (Self-Hosting) bucket policy
Having used the documentation site to look for this answer (by the way, great search approach!), I can't find documentation on the setup of an AWS S3 bucket that includes the entire setup of policies. I can see CORS although that probably be more restrictive in production. I see in the source it looks to require 'public-read' for example.
Do you have a recommended production settings?
9 Replies
Hi @todd
We use S3 primarily for two things, brand logo storage and email attachment storage
So you can create a s3 bucket with create, read bucket + create, read and delete object permissions
We would recommend to use {"Action": ["s3:*"]} as Novu needs to be able to run full CRUD.
We have also not invested much of time in ensuring "proper" aws recommended use of this bucket as there are many other priorities we want to work on.
@Zac Clifton thanks for response.
Here are some of my notes for future reference if anyone is interested. Feel free to correct any mistakes or make clarifications.
General design:
- public read access (for logo and email attachments—all data is open for read to anyone)
- write access is via pre-signed url (client-side PUT to S3 after server-side request)
- private write access (get, put, delete) [<-- not sure you actually need list actions in policy]
- objects are written with organisationIdTenancy/objectId (at least for brand logos)
Specifics for access:
- CORS access on bucket (this is documented)
- bucket object do not need versioning (as all uploads are deemed unique)
- ACL on buckets is 'public-read' (as part of the pre-signed URLs)
- image uploads (as pre-signed) could also have rules to limit (DENY policy) on extension types (although UI also does this but leaves open a vector)
- AWS (specific): ensure that Allow owenership controls is 'BucketOwnerPreferred' to allow ACL access (rather than role/user) (ie PutObjectAcl)
Here is an example extract of some terraform for provisioning aws s3 bucket
@todd You rock man, would you kindly do us a favor and make a PR to our docs repository otherwise this will get barried.
https://github.com/novuhq/docs
If you are unable, let me know and I can move it in instead.
if you are able, Please add my github user @cliftonz and reference this discord thread so I know its not a regular hacktoberfest PR.
Again Thank you so much for the detailed documentation!
@todd I have some time today, I will go ahead and take this documentation and merge it in.
@Zac Clifton That would be rally great thanks. Agreed not to loose it. Like everyone struggling I'm struggling with workloads and the previous post was already an hour out of my day. Happy to review changes.
There is the wider decision context for self-hosting on S3:
- Provider: eg aws, gc, azure? and then other options - sorry ignorance
- Deployment: terraform, cli
Also, there are the wider design decisions that I think are important to convey that aren't about self hosting (eg the use of pre-signed urls rather than API-based proxy—I couldn't see where the indexing/key generation mechanism was)
Just merged in @Zac Clifton's PR about this, we should have this live in a bit on the docs. Thanks @todd for this amazing contribution