21 Replies
The events of a couple days ago behind us, I think it makes sense to rotate our keys out on a regular basis. Our key may leak in the future and it may not be something we catch for one reason or another
We should throw together a schedule for handling this and implement planning for transitioning over to a new key
To do this correctly, we'd need a better idea of how frequently users are updating so that we have some reasonable period set aside for people to upgrade to builds with the new key
More details in a bit
Something I've not really thought about. Would it be a terrible idea hosting our pub key on a remote server, frequently pulling that to /etc/pki/containers outside the regular OS upgrades?
A cron-type service that keeps the public key up-to-date rather than relying on the key we ship baked into the image.
As I said, I've not thought about it from a security side, and we should to a mini threat modelling exercise to find out what could go wrong, but it would simplify rotation significantly as we just push a new file to GitHub/a website and switch builds to the new private key.
I think this can be compared to how some YUM repositories work, where you configure GPG checks using a remote public key.
We could even have it as it as a pre-requisite to the rpm-ostreed.service, so every time the machine tries to update, it first pulls our public key down
I actually was thinking about that. That the public key should be somewhere that can be updated on our side and client machines will pick up.
The downside to something like that is usually the updater is called without privileges, so writing to /etc would be a separate call.
I also think this would be no different than updating GPG keys for apt/pacman/dnf
The downside to something like that is usually the updater is called without privilegesOur updater may be called without privileges, but rpm-ostreed (which looks like the thing that actually does the update) is a system service
For pacman, there is a service they have running weekly to rebuild/update the GPG keyring
Sorry all. Very busy afternoon yesterday
Given other distributions pull down keys for updates, I'm not opposed to us automating this. We just need to be sure the key matches what we ship in our config at all times
When we rotate the key, we'll also need to resign older images with the new key to permit rollbacks since our policy doesn't allow us to use multiple keys
This would prevent us from needing to sign images with the older key for an extended time so from a security perspective, I think this is the way
We should still have some reasonable period we continue to sign with it though to prevent support headaches. I.E., if this is a service we run once weekly that isn't entangled with rpm-ostreed, we should do so for at least 2 weeks. Thinking about how we can prevent that though and avoid doing that entirely
Given the service would take a second or two to complete, I'd say we run it as much as we can. Either daily / on boot or tie it into rpm-ostree
On boot + weekly may be a good compromise because rebooting is one of the first things people (should) try when debugging, and it will not interrupt other activities on the system
And signing older images should be fairly trivial. I wrote an action which allows Bazzite to re-sign individual images, so it would be just changing that to get an array of images pushed in the last 180 days (or another reasonable timeframe) and iterate over those
Or we have another look at rekor(?) or whatever the keyless signing is, where it uses information about the GitHub org, repo, workflow and job to sign images
There is falcur and oidc as well
We could also move to good old GPG
We could keep old public key on repo for a bit as well .
We also could consider having a repo who's job is to control actions for other repos.
I personally like the idea of on boot + weekly/daily checks. It's a trivial amount of data
what's wrong with signing with the older key? or rather, what's more secure in hosting the pubkey on some server?
We could not figure out how to support trusting two different keys
And AFAICT the container policy doesn't support a remote file
OIDC is the keyless signing system that at least @Luke Skywunker has implemented signing with in bluebuild, but there were some errors related to support in rpm-ostree AFAIK, so i'm not sure if any keyless signing method is possible yet for us
Yeah I think we determined that until keyless without email is supported, we can't make use of CI based signing.
I think I'm in the camp of us having a service that fetches current key on reboot and then on a weekly cycle.
We could also have it run before ublue-update as well
Here's our tracking issue https://github.com/blue-build/cli/issues/84
GitHub
fix: Updating policy.json to properly verify images · Issue #84 · b...
We need to look into some workflow to handle properly verifying images before rebasing on a signed keyless image. https://www.mankier.com/5/containers-policy.json#Examples https://github.com/bsherm...
I'm on board with this
I think this is the best approach at the moment. It's inexpensive and straightforward enough to implement
I would like for us to have short URL so it doesn't look as weird for a user if they go inspecting.... But unsure if that's actually necessary
I was just going to curl the pubkey from either main or config but you're right
should we put the public key in our github org page?
Either in the GitHub repo, or as a website asset. Either sounds okay to me
Do we need to do an RFC from maintainers before implementing this sort of change?
The voting thing we did a few times
I think we should schedule a key rotation pattern. And that's probably something we should vote on
Agree. I was thinking every 6 months, inline with Fedora releases