PostgreSQL S3 backups: Allowing to Upload to a Bucket with Object Lock enabled
Hello everyone,
When I tried to use the PostgreSQL S3 backups template (https://github.com/railwayapp-templates/postgres-s3-backups)
to upload backups to a bucket that has object lock enabled
(a very important security feature) I got the following error message:
Error while running backup: InvalidRequest: Missing required header for this request: Content-MD5
at throwDefaultError (/app/node_modules/@smithy/smithy-client/dist-cjs/index.js:838:20)
It seems that the content MD5 is needed
if object lock is active. See here https://github.com/aws/aws-sdk-php/issues/1694
How should I calculate the MD5 content here? And does it make sense from your point of view to build this feature directly into the template,
because object lock is a standard security measure for backups.
Thank you in advance for your time
40 Replies
Project ID:
N/AN/A
so if I understand this correctly, essentially the backup service just needs to provide an md5 hash of the backup file when uploading the file to S3?
Hey Brody,
Yes, it looks like that. Also, the md5 hash apparently has to be encoded with base64 afterwards
that part is definitely easy enough
md5 hashing a potentially large backup file could end up being quite resource intensive so if it is implemented it would need to be off by default for the buckets that don't require it, at least that's my thinking
Yes, it can be disabled by default.
Apparently you only need it if Object Lock is enabled for the bucket.
I'd be happy to take a crack at this, but I don't know if cloudflare R2 has object lock? I don't have access to an aws account to test with s3
I believe wasabi and backblaze have implemented object locking if you have an account there
I do have an account with backboaze!
that is nice (:
Because i am trying to use the template with backblaze and got the error
I'll see if I can get NodeJS to md5 hash a file efficiently, if node has to load the entire file into memory I don't think that would be worth it to implement
thank you brody!
no problem, I shall report back
got node to md5 hash a large (20gb) file without loading it into memory, and it's not actually too slow, will work on integrating that into the backup service when I'm back at the computer and then I'll do some testing, if that goes well, I would like to ask you to test as well
thats sounds great! Thank you very much!
I'm happy to test when you give me the ping
will do