SBOMs
With the SBOM and security scan results...
There are a few ways we can do it.
The easy way would be to append two steps on the back of each build job. It'll add 5+ minutes onto builds. The medium-difficulty way would be to create a new dedicated workflow with the same matrixes as the build steps that pull, scan and upload the SBOMs and security results, running this on a schedule. The difficult way would be to push the container images with a PAT, and trigger the scan workflow automatically for each image upload (event-driven). Thoughts?
The easy way would be to append two steps on the back of each build job. It'll add 5+ minutes onto builds. The medium-difficulty way would be to create a new dedicated workflow with the same matrixes as the build steps that pull, scan and upload the SBOMs and security results, running this on a schedule. The difficult way would be to push the container images with a PAT, and trigger the scan workflow automatically for each image upload (event-driven). Thoughts?
40 Replies
so...
when i read this, i think understand that both easy and difficult ways would block publishing of images which fail security scans
medium-difficulty would not change current build process but would alert us to any problems on a schedule
I was not planning on blocking anything as there's quite a few false positives right now. Mainly VSCode's NPM dependencies are all flagged. It should be possible to exclude these if we wanted.
This is my current workflow on my image, and is the easy way.
https://github.com/rsturla/eternal-images/actions/runs/8570285191/job/23487870654
There's the regular image builds and pushes at the start of the workflow, then the last two generate and scan the SBOM. Even though there are critical issues, builds are passing
i see
it's oddly amusing to me that it takes nearly 6 minutes to compile the SBOM but only 9 seconds to scan it 🙂
i like that we wouldn't be blocking anything at first, we definitely want to work towards the scan providing value, removing false positives, etc first
but if we did have high confidence in the scan results, then i would hope we do eventually block image publishing to GHCR
Said another way:
if we had high confidence today, I'd vote for "easy" path of adding steps to our current build workflow and blocking publish on scan fail
but since we don't, I'm hesitant to add steps which we know will be producting false positive errors... as that seems like it will only add potential confusion
So, I wonder, should this be a long running PR which we leave in draft mode while iterating on confidence of the scan? or should it be the medium difficulty, schedule scans and make it clear we are working on getting those results solid, or should it just be worked on outside the official repo until we have confidnce
We would likely want to bundle a default configuration alongside the action, so we could just add the action, point the workflows to
@main and iterate over time.
To start with, all results would be added as job artifacts but the end goal would be to upload the SBOM to R2 and security scans to CodeQL once the false positives are ironed out.
I'm in two minds whether the findings are false positives or not. And how we would decide?
They are all vulnerabilities for something installed on the machine - be it electron, Podman's dependencies or others. Whether that particular code path is used, it will be near impossible to tell.Apprediate the info...
I think if we're never going to block builds based on the security results, I would probably prefer the SBOM generation and security scan is decoupled from the build.
That makes sense if there are ONLY daily builds... but what happens for the extra builds like when things are failing and we have to build several times to get good builds... there will be images in our registry which aren't covered by an SBOM and scan
That makes sense if there are ONLY daily builds... but what happens for the extra builds like when things are failing and we have to build several times to get good builds... there will be images in our registry which aren't covered by an SBOM and scan
Am finally getting round to implementing this
https://github.com/ublue-os/bluefin/pull/1161
@j0rge here's the SBOM thread 🙂
We have scans!
Possibly the most hacky way of doing things, but if anyone has any other ideas, would love to hear them
What I have done:
- Add new artifact in each build_container matrix with a file containing the registry and digest
- Add new step in the "check" job to download and merge all artifacts from each matrix into one artifact
- Do some string magic to translate that into '["<image>@<digest>", ...]'
- Set that string as a workflow output
- Create a new reusable workflow that accepts the string as an input
- JSONify that string so it can be used in the matrix job
- Run Syft and Grype to create and scan the SBOM
I'm thinking the GitHub runners don't have enough memory to scan the Nvidia images 😦
Generating the SBOM cancels after 7 minutes for no apparent reason during Nvidia.
Edit: Excluding some files (like /sysroot/ostree/repo/objects) made the scans succeed, but it's delaying the issue.
PR ready for review!
https://github.com/ublue-os/bluefin/pull/1161
GitHub
feat: add image scanning workflow by p5 · Pull Request #1161 · ublu...
Creates a reusable workflow to scan images. This scan generates a SBOM and checks that SBOM for known vulnerabilities.
The only way I was able to merge the outputs of each matrix run was to use ar...
oh that's fine, that's great feedback for github
Would be great to get some reviews when you both have a second.
Want to roll it out to Bluefin and Aurora in separate PRs
Want to roll it out to Bluefin and Aurora in separate PRs
this is beyond my tech skill but I think we want this. 😄
full sboms and stuff would be baller, nice choice of syft too, looking to ship grype in bluefin-dx
like this would be a cool announcement to make
syft scan docker:ghcr.io/ublue-os/bluefin
I wish this is what the ostree update would look like!
@Robert (p5) how would an end user consume the sbom? or are we waiting for them to go into the registry and then figure that out?So the SBOMs are mainly for us, but will still be accessible to the user (ideally by downloading them from R2).
We generate the SBOMs and scan the packages contained with Grype. The outputs can then be pushed to GitHub's Security tab where we (members of the GitHub org) can see what potential vulnerabilities are in our image.
End users may want/need to download the SBOMs for compliance purposes. I've needed to provide SBOMs of EC2s and Lambdas to security teams in the past as they need to check things like licenses and vulns.
End users may want/need to download the SBOMs for compliance purposes. I've needed to provide SBOMs of EC2s and Lambdas to security teams in the past as they need to check things like licenses and vulns.
oh ok, I understand
this is so awesome
In the future, I guess we could use it to generate our own changelogs by comparing two SBOMs.
I think that would be quite easy to do
I think that would be quite easy to do
And there's also this, which I'm not sure how people view it. You upload your SBOMs to GitHub through the API, and they put it somewhere
https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/using-the-dependency-submission-api
My PR is only for generating and scanning the SBOMs, storing the results in job artifacts, but I feel it's enough for a first iteration
yeah I'm curious on how people actually use it in practice
I guess we'll find out, heh
@j0rge
@popey @tulip🌷 @Caleb @M2
So I've gotten SBOMs generating during workflows.
Syft needs a lot of memory so using a swapfile helps prevent an OOM situation.
Attestation is the next challenge. It seems that there is a limit for how large they can be when pushing to ghcr.
Right now my cosmic and Ucore images can push the attestation. Bluefin, Bazzite, and Aurora are all too big.
should we get the SBOMs going on the product images with the swap files, etc before moving on to attenstations?
Yeah. That's basically where my bluefin PR is at. Going to remove the mess that is the summary step
So pushing to job artifacts for now?
Yeah. I think we can add them to releases as a temporary idea
I like adding it to Releases for the main repos. Trying to think of a good way to surface those to a user. Easy and consistent place if someone wants to build around that.
yeah
then it's just an everyday thing
Would suggest to use
cosign attest to add the SBOMs to container image repo, signing them
https://docs.sigstore.dev/cosign/verifying/attestation/Sigstore
In-Toto Attestations
Cosign also has built-in support for in-toto attestations. The specification for these is defined here.
You can create and sign one from a local predicate file using the following commands:
$ cosign attest --predicate --key cosign.key All of the standard key management systems are supported. Payloads are signed using the DSSE signing spec, def...
On one of my personal project, this is how I’m attesting the SBOMs
https://gitlab.com/flattrack/flattrack/-/blob/main/hack/publish.sh?ref_type=heads#L66
and this is how it’s verified and downloaded
https://gitlab.com/flattrack/flattrack/-/blob/main/docs/deployment.md?ref_type=heads#fetch-sbom
(side note, need to have a page of all the cncf projects we use)
and LF!
that's what i'm currently doing. There seems to be a size limit for the SBOMs
https://github.com/m2Giles/m2os/blob/18026a26cfbbb392edc68007a9aa8769517ec42d/Justfile#L728
my largest sbom's are pushing 200MBs which could be an issue
Yikes that's pretty big
Ah! I follow!
so i can generate them... i also don't know why they are massive. I'm unsure if some stuff are getting duplicated but that is a ton of text
so cosign attest of my largest sbom i think is failing because the base64 of it is unbelievably massive....
base64 encoding it brings it up to 230MB....
so instead of attesting them... we can blob-sign them
so we'll have signatures for the sboms that can be confirmed... but they won't live on the registry with them. For our smaller images attestation could work
At the risk of sounding naive, having a SBOM of over 200MB of text is gonna be a problem. I know that syft has different formats it can output to. Maybe one of the others can cut down on that size?
I wonder if maybe we break up the SBOM into tiers? Like base image SBOM then another SBOM for the pieces we layer on top? Something that's more manageable and can be used easily
GitHub
chore: use cosign sign-blob instead of attest · m2Giles/m2os@14a0a02
Contribute to m2Giles/m2os development by creating an account on GitHub.
Verify-blob working during the run
sbom artifact that is uploaded is the sbom and the signature