UB
Universal Blue•10mo ago
Luke Skywunker

testing keyless cosign oidc signing

I rebase to any of my images directly 
$> rpm-ostree rebase ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-laptop:39
Pulling manifest: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-laptop:39
Importing: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-laptop:39 (digest: sha256:ec09f68334ca2fb664fab366e402f0be9313dedc1bd7b8559fb588da4af8ad70)
ostree chunk layers already present: 65
custom layers already present: 1
custom layers needed: 1 (694.2 MB)
$> rpm-ostree rebase ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-laptop:39
Pulling manifest: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-laptop:39
Importing: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-laptop:39 (digest: sha256:ec09f68334ca2fb664fab366e402f0be9313dedc1bd7b8559fb588da4af8ad70)
ostree chunk layers already present: 65
custom layers already present: 1
custom layers needed: 1 (694.2 MB)
60 Replies
bsherman
bsherman•10mo ago
FYI I've done some testing of my own custom image using the PR in bluefin as a guide ( @Gerblesh @Robert (p5) you may be interested ) Here's my findings: 1) yes, I can sign my image with OIDC / keyless 2) sadly, the PR in Bluefin probably won't actually sign successfully as it attempts to sign tags not an imageURI, but I got it working in my custom image repo 3) yes, I can directly rebase from ublue-os/silverblue-main to my oidc signed image with a URI like ostree-image-signed:docker://ghcr.io/bsherman/silverblue-custom:latest ... BUT! 4) because of our containers policy which has insecureAcceptAnything... it is actually doing a blind trust... I could just as easily rebase to an ancient image of mine which was not signed with OIDC since that policy file is part of the ublue-os main repo, it'll impact any downstream trying to do this, given a false sense of security about validating the target rebase image Here's an example of a rebase from stock upstream silverblue: First establish status 
$ rpm-ostree status
State: idle
Deployments:
\u25cf fedora:fedora/39/x86_64/silverblue
                  Version: 39.20240129.0 (2024-01-29T05:35:53Z)
               BaseCommit: bdd0afb451c1768bf872aa30c7d6ee5767f72fd75a75275a54a75f6b30804b61
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
          LayeredPackages: langpacks-en
$ rpm-ostree status
State: idle
Deployments:
\u25cf fedora:fedora/39/x86_64/silverblue
                  Version: 39.20240129.0 (2024-01-29T05:35:53Z)
               BaseCommit: bdd0afb451c1768bf872aa30c7d6ee5767f72fd75a75275a54a75f6b30804b61
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
          LayeredPackages: langpacks-en
Now attempt to rebase to an old image with generated keypair signing 
$ rpm-ostree rebase ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:38-20230701
Pulling manifest: ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:38-20230701
error: Preparing import: Fetching manifest: containers-policy.json specifies a default of `insecureAcceptAnything`; refusing usage
$ rpm-ostree rebase ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:38-20230701
Pulling manifest: ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:38-20230701
error: Preparing import: Fetching manifest: containers-policy.json specifies a default of `insecureAcceptAnything`; refusing usage
Now attempt to rebase to new image with OIDC signing 
$ rpm-ostree rebase ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:39
Pulling manifest: ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:39
error: Preparing import: Fetching manifest: containers-policy.json specifies a default of `insecureAcceptAnything`; refusing usage
$ rpm-ostree rebase ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:39
Pulling manifest: ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:39
error: Preparing import: Fetching manifest: containers-policy.json specifies a default of `insecureAcceptAnything`; refusing usage
Here's that first image again, failing oidc validation: 
$ cosign verify --certificate-oidc-issuer "https://token.actions.githubusercontent.com"     --certificate-identity-regexp "https://github.com/bsherman/ublue-custom"   ghcr.io/bsherman/silverblue-custom:38-20230701
Error: no matching signatures: error verifying bundle: nil certificate provided
main.go:69: error during command execution: no matching signatures: error verifying bundle: nil certificate provided
$ cosign verify --certificate-oidc-issuer "https://token.actions.githubusercontent.com"     --certificate-identity-regexp "https://github.com/bsherman/ublue-custom"   ghcr.io/bsherman/silverblue-custom:38-20230701
Error: no matching signatures: error verifying bundle: nil certificate provided
main.go:69: error during command execution: no matching signatures: error verifying bundle: nil certificate provided
But the second succeeds: 
$ cosign verify --certificate-oidc-issuer "https://token.actions.githubusercontent.com"     --certificate-identity-regexp "https://github.com/bsherman/ublue-custom"   ghcr.io/bsherman/silverblue-custom:39-20240223

Verification for ghcr.io/bsherman/silverblue-custom:39-20240223 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates
--- snip ---
$ cosign verify --certificate-oidc-issuer "https://token.actions.githubusercontent.com"     --certificate-identity-regexp "https://github.com/bsherman/ublue-custom"   ghcr.io/bsherman/silverblue-custom:39-20240223

Verification for ghcr.io/bsherman/silverblue-custom:39-20240223 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates
--- snip ---
I'm up WAY past my bedtime, but I'll look at this some more... I believe we need a policy to validate ublue-os images signed this way... which will improve our overall security, and... not give our users false sense of validation which seems to be happening at this time.
Luke Skywunker
Luke SkywunkerOP•10mo ago
That's some really good info. Thanks for finding this out!
bsherman
bsherman•10mo ago
I did a bit of looking for info on container policy.json and found some examples... I haven't had a chance to test yet: https://www.mankier.com/5/containers-policy.json#Examples
containers-policy.json: syntax for the signature verification polic...
Signature verification policy files are used to specify policy, e.g.
bsherman
bsherman•10mo ago
Gerblesh
Gerblesh•10mo ago
tysm for looking into this! @Luke Skywunker should we make a tracking issue for this on bluebuild?
Luke Skywunker
Luke SkywunkerOP•10mo ago
Yes, it sounds like we'll still need to make changes to the policy.json. We'll just have to figure out some workflow on the user side to make it easier for them.
Gerblesh
Gerblesh•10mo ago
I can modify the signing module to check for no pubkey and use OIDC policy.json instead of the normal one
Luke Skywunker
Luke SkywunkerOP•10mo ago
Oh do we have a special OIDC policy.json?
Gerblesh
Gerblesh•10mo ago
Ehh, we could still probably just use yq
bsherman
bsherman•10mo ago
One is required in order to verify these images from rpm-ostree
Gerblesh
Gerblesh•10mo ago
It looks pretty straightforward Judging from the reference sent
bsherman
bsherman•10mo ago
my eyes glazed over the other night, I was too tired 😄
Gerblesh
Gerblesh•10mo ago
And you can still directly rebase, even if it isn't verified I'm worried about non-host OIDC images though
Luke Skywunker
Luke SkywunkerOP•10mo ago
I wonder if we could incorporate updating the policy.json using bluebuild to get it to trust the image source before we start the rebase process
bsherman
bsherman•10mo ago
correct, but it's a bit of a lie, because the rpm-ostree process has a URI which indicates it has verified the image, but it has not
Luke Skywunker
Luke SkywunkerOP•10mo ago
Since it gets baked in the image at /usr/etc/containers/ that means it becomes available in /etc/containers/ and becomes "writeable" There is a section in the policy.json for ostree. I wonder if setting that to be strict to the user's repo would allow rpm-ostree to properly verify the image before rebasing
Gerblesh
Gerblesh•10mo ago
Fair yeah Is it a bug then? Since it's ostree-image-signed
bsherman
bsherman•10mo ago
a bug in ublue maybe, but not in rpm-ostree itself, our /etcc/containers/policy.json forces insecureAcceptAnything and that's why it passes the way i read the example containers policy ddoc I sent... i think if ublue-os used oidc signing, we'd need something like this in our policy: 
"ghcr.io/ublue-os": [
                {
                    "type": "sigstoreSigned",
                    "fulcio": {
                        "caPath": "/path/to/fulcio_v1.crt.pem",
                        "oidcIssuer": "https://token.actions.githubusercontent.com",
                        "subjectEmail": "https://github.com/ublue-os/"
                    },
                    "rekorPublicKeyPath": "/path/to/rekor.pub",
                }
            ]
"ghcr.io/ublue-os": [
                {
                    "type": "sigstoreSigned",
                    "fulcio": {
                        "caPath": "/path/to/fulcio_v1.crt.pem",
                        "oidcIssuer": "https://token.actions.githubusercontent.com",
                        "subjectEmail": "https://github.com/ublue-os/"
                    },
                    "rekorPublicKeyPath": "/path/to/rekor.pub",
                }
            ]
Gerblesh
Gerblesh•10mo ago
"default" is "reject" though How do we get the fulcio cert? I thought cosign only needs the oidc issuer and github ref to verify
bsherman
bsherman•10mo ago
correct... if you are starting from the default silverblue containers policy, rpm-ostree would correctly fail to rebase to ostree-image-signed:docker://ghcr.io/ublue-os/foo-main:latest regardless of type of signing i'm just reading the docs, i've not testedyet also that link I shared is bascially the same content as man containers-policy.json
Gerblesh
Gerblesh•10mo ago
Do you know?
bsherman
bsherman•10mo ago
there's a URL in the doc... i haven't pursued it further
Gerblesh
Gerblesh•10mo ago
Ah, here: https://github.com/sigstore/sigstore/tree/main/pkg/tuf/repository/targets
GitHub
sigstore/pkg/tuf/repository/targets at main · sigstore/sigstore
Common go library shared across sigstore services and clients - sigstore/sigstore
Gerblesh
Gerblesh•10mo ago
Gerblesh
Gerblesh•10mo ago
            "ghcr.io/gerblesh/uhypr": [
                {
                "type": "sigstoreSigned",
                "fulcio": {
                      "caPath": "/etc/pki/containers/fulcio_v1.crt.pem",
                      "oidcIssuer": "https://token.actions.githubusercontent.com",
                      "subjectEmail": "https://github.com/gerblesh"
                 },
                 "rekorPublicKeyPath": "/etc/pki/containers/rekor.pub",
                 "signedIdentity": { "type": "matchRepository" }
                }
            ],
            "ghcr.io/gerblesh/uhypr": [
                {
                "type": "sigstoreSigned",
                "fulcio": {
                      "caPath": "/etc/pki/containers/fulcio_v1.crt.pem",
                      "oidcIssuer": "https://token.actions.githubusercontent.com",
                      "subjectEmail": "https://github.com/gerblesh"
                 },
                 "rekorPublicKeyPath": "/etc/pki/containers/rekor.pub",
                 "signedIdentity": { "type": "matchRepository" }
                }
            ],
put the following into my policy.json and updating with rpm-ostree gave me this error: 
Pulling manifest: ostree-image-signed:docker://ghcr.io/gerblesh/uhypr:latest
error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: A signature was required, but no signature exists
Pulling manifest: ostree-image-signed:docker://ghcr.io/gerblesh/uhypr:latest
error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: A signature was required, but no signature exists
cosign verify is working so the image is definitely signed @Luke Skywunker got any ideas here? I have both the certs: 
user@localhost ~ [1]> ls /etc/pki/containers/
 fulcio_v1.crt.pem  󰷖 rekor.pub  󰷖 ublue-os.pub
user@localhost ~ [1]> ls /etc/pki/containers/
 fulcio_v1.crt.pem  󰷖 rekor.pub  󰷖 ublue-os.pub
this is all manual atm as i'm just trying to figure out what configuration actually works before I automate everything in the image build
Luke Skywunker
Luke SkywunkerOP•10mo ago
I'm not sure. We might have to do some digging into the docs for rpm-ostree
Gerblesh
Gerblesh•10mo ago
getting somewhere It now errors here: 
user@localhost ~> rpm-ostree update
Pulling manifest: ostree-image-signed:docker://ghcr.io/gerblesh/uhypr:latest
error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: Required email github.com/gerblesh/uhypr not found (got []string(nil))
user@localhost ~> rpm-ostree update
Pulling manifest: ostree-image-signed:docker://ghcr.io/gerblesh/uhypr:latest
error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: Required email github.com/gerblesh/uhypr not found (got []string(nil))
I needed to specify to use sigstore attachments with container-registries.d like before
Luke Skywunker
Luke SkywunkerOP•10mo ago
So I think it's looking for your git email Or whatever email was used to run the action?
Gerblesh
Gerblesh•10mo ago
idk 
"Subject": "https://github.com/gerblesh/uhypr/.github/workflows/build.yml@refs/heads/live"
"Subject": "https://github.com/gerblesh/uhypr/.github/workflows/build.yml@refs/heads/live"
this is what it gives me when I use cosign verify
Luke Skywunker
Luke SkywunkerOP•10mo ago
Could you use that instead of subjectEmail? So it would be: 
            "ghcr.io/gerblesh/uhypr": [
                {
                "type": "sigstoreSigned",
                "fulcio": {
                      "caPath": "/etc/pki/containers/fulcio_v1.crt.pem",
                      "oidcIssuer": "https://token.actions.githubusercontent.com",
                      "subject": "https://github.com/gerblesh/uhyper/.github/workflows/build.yml@refs/heads/live"
                 },
                 "rekorPublicKeyPath": "/etc/pki/containers/rekor.pub",
                 "signedIdentity": { "type": "matchRepository" }
                }
            ],
            "ghcr.io/gerblesh/uhypr": [
                {
                "type": "sigstoreSigned",
                "fulcio": {
                      "caPath": "/etc/pki/containers/fulcio_v1.crt.pem",
                      "oidcIssuer": "https://token.actions.githubusercontent.com",
                      "subject": "https://github.com/gerblesh/uhyper/.github/workflows/build.yml@refs/heads/live"
                 },
                 "rekorPublicKeyPath": "/etc/pki/containers/rekor.pub",
                 "signedIdentity": { "type": "matchRepository" }
                }
            ],
Gerblesh
Gerblesh•10mo ago
tried that, same error got []string(nil) does this mean that there's no subject email?
Gerblesh
Gerblesh•10mo ago
yet in the JSON it omits the config option when it's blank here: https://github.com/containers/image/blob/afda0f0452d6b3fe41c99890ddfcff7b91aba123/signature/policy_types.go#L156
GitHub
image/signature/policy_types.go at afda0f0452d6b3fe41c99890ddfcff7b...
Work with containers' images. Contribute to containers/image development by creating an account on GitHub.
Gerblesh
Gerblesh•10mo ago
I haven't used go but does this mean that signature verification of keyless images signed in github actions doesn't work with skopeo? might be missing something here
Luke Skywunker
Luke SkywunkerOP•10mo ago
Maybe use this type of signing? https://github.com/containers/image/blob/afda0f0452d6b3fe41c99890ddfcff7b91aba123/signature/policy_types.go#L111
GitHub
image/signature/policy_types.go at afda0f0452d6b3fe41c99890ddfcff7b...
Work with containers' images. Contribute to containers/image development by creating an account on GitHub.
Luke Skywunker
Luke SkywunkerOP•10mo ago
Wait that's still part of it Not used to reading golang
Gerblesh
Gerblesh•10mo ago
Luke Skywunker
Luke SkywunkerOP•10mo ago
    // FIXME: Match more subject types? Cosign does:
    // - .DNSNames (can’t be issued by Fulcio)
    // - .IPAddresses (can’t be issued by Fulcio)
    // - .URIs (CAN be issued by Fulcio)
    // - OtherName values in SAN (CAN be issued by Fulcio)
    // - Various values about GitHub workflows (CAN be issued by Fulcio)
    // What does it… mean to get an OAuth2 identity for an IP address?
    // FIXME: How far into Turing-completeness for the issuer/subject do we need to get? Simultaneously accepted alternatives, for
    // issuers and/or subjects and/or combinations? Regexps? More?
    // FIXME: Match more subject types? Cosign does:
    // - .DNSNames (can’t be issued by Fulcio)
    // - .IPAddresses (can’t be issued by Fulcio)
    // - .URIs (CAN be issued by Fulcio)
    // - OtherName values in SAN (CAN be issued by Fulcio)
    // - Various values about GitHub workflows (CAN be issued by Fulcio)
    // What does it… mean to get an OAuth2 identity for an IP address?
    // FIXME: How far into Turing-completeness for the issuer/subject do we need to get? Simultaneously accepted alternatives, for
    // issuers and/or subjects and/or combinations? Regexps? More?
Gerblesh
Gerblesh•10mo ago
rip
Luke Skywunker
Luke SkywunkerOP•10mo ago
// - Various values about GitHub workflows (CAN be issued by Fulcio)
// - Various values about GitHub workflows (CAN be issued by Fulcio)
Yeah....
Gerblesh
Gerblesh•10mo ago
@bsherman we hit a dead end here we could try drafting up a PR that fixes this, but I know next to nothing about go
bsherman
bsherman•10mo ago
this is my concern with using OIDC github signing, I've yet to see how to validate from containers-policy.json... and even if we can... I'm really don't believe this is a significant improvement over the keypair method i guess it does remove the keypair generation step
Gerblesh
Gerblesh•10mo ago
it improves security yeah, and gets rid of the keypair step that is, it would improve security if containers-policy.json supported it
bsherman
bsherman•10mo ago
i disagree that using oidc github signing is a signficant security improvement over keypairs... I do agree that it removes a step from the signing process
Luke Skywunker
Luke SkywunkerOP•10mo ago
bluebuild could totally add a workflow to update a user's policy.json before rebasing to their image But
Gerblesh
Gerblesh•10mo ago
even if we were to merge the functionality into containers/image we'd have to wait for a release, skopeo to bump the library, and then rpm-ostree to bump skopeo
Luke Skywunker
Luke SkywunkerOP•10mo ago
That dead end I'll look and see if there are any open issues for this functionality
Gerblesh
Gerblesh•10mo ago
I wouldn't mind taking the time to figure out how to implement it
bsherman
bsherman•10mo ago
I do find the topic interesting... and I got involved here to test and validate the PR in bluefin. But... at least on the ublue/bluefin/bazzite side of things... I don't see the current situation as a problem needing a fix. My current main focus is improving the ublue CI processes to be more reliable.
Gerblesh
Gerblesh•10mo ago
yeah I understand, originally I thought containers-policy.json would automatically verify the signatures so it would've made rebasing easier, until you looked into it
bsherman
bsherman•10mo ago
yes, and i agree, if that had been the case, I would have helped make the change fairly quickly.
Luke Skywunker
Luke SkywunkerOP•10mo ago
There's this issue which is kinda close https://github.com/containers/image/issues/2027
GitHub
Request for Sigstore signature verification enhancement and flexibi...
Currently, we are able to verify container images with sigstore signatures using public key/Fulcio/Rekor, as described here. However, there are additional verifications supported by cosign, such as...
Luke Skywunker
Luke SkywunkerOP•10mo ago
Fully agreed. There needs to be more upstream changes before we can even encourage using keyless
Gerblesh
Gerblesh•10mo ago
time to switch back to public-private key pair for now I guess might want to add a note that OIDC images can't be verified by rpm-ostree with containers-policy.json
Luke Skywunker
Luke SkywunkerOP•10mo ago
Yeah that's still the first tried option in bluebuild
Gerblesh
Gerblesh•10mo ago
I think keeping the signing logic is fine but we should outline that somewhere
Luke Skywunker
Luke SkywunkerOP•10mo ago
Agreed Thanks @bsherman for trying things out!
Gerblesh
Gerblesh•10mo ago
unfortunate we can't use OIDC signing but i'm really glad you were able to spot this! otherwise we would've thought out images were being verified when in reality they weren't
bsherman
bsherman•10mo ago
One thing to note, and I'm about to test... I THINK one can SIGN with both methods, but only verify with keypair... so you could implement that future proofing
Luke Skywunker
Luke SkywunkerOP•10mo ago
Doing both, that's a good idea
Gerblesh
Gerblesh•9mo ago
https://github.com/containers/image/pull/2235
GitHub
Accept URI for Sigstore Signed Images by lukewarmtemp · Pull Reques...
Fixes coreos/rpm-ostree#4272 (comment) Allows for container images signatures with a URI instead of a email address as the SAN. This is needed in certain cases such as using a github workflow to si...
Want results from more Discord servers?
Add your server