SBOMs
With the SBOM and security scan results...
There are a few ways we can do it.
The easy way would be to append two steps on the back of each build job. It'll add 5+ minutes onto builds. The medium-difficulty way would be to create a new dedicated workflow with the same matrixes as the build steps that pull, scan and upload the SBOMs and security results, running this on a schedule. The difficult way would be to push the container images with a PAT, and trigger the scan workflow automatically for each image upload (event-driven). Thoughts?
The easy way would be to append two steps on the back of each build job. It'll add 5+ minutes onto builds. The medium-difficulty way would be to create a new dedicated workflow with the same matrixes as the build steps that pull, scan and upload the SBOMs and security results, running this on a schedule. The difficult way would be to push the container images with a PAT, and trigger the scan workflow automatically for each image upload (event-driven). Thoughts?
18 Replies
so...
when i read this, i think understand that both easy and difficult ways would block publishing of images which fail security scans
medium-difficulty would not change current build process but would alert us to any problems on a schedule
I was not planning on blocking anything as there's quite a few false positives right now. Mainly VSCode's NPM dependencies are all flagged. It should be possible to exclude these if we wanted.
This is my current workflow on my image, and is the easy way.
https://github.com/rsturla/eternal-images/actions/runs/8570285191/job/23487870654
There's the regular image builds and pushes at the start of the workflow, then the last two generate and scan the SBOM. Even though there are critical issues, builds are passing
i see
it's oddly amusing to me that it takes nearly 6 minutes to compile the SBOM but only 9 seconds to scan it 🙂
i like that we wouldn't be blocking anything at first, we definitely want to work towards the scan providing value, removing false positives, etc first
but if we did have high confidence in the scan results, then i would hope we do eventually block image publishing to GHCR
Said another way:
if we had high confidence today, I'd vote for "easy" path of adding steps to our current build workflow and blocking publish on scan fail
but since we don't, I'm hesitant to add steps which we know will be producting false positive errors... as that seems like it will only add potential confusion
So, I wonder, should this be a long running PR which we leave in draft mode while iterating on confidence of the scan? or should it be the medium difficulty, schedule scans and make it clear we are working on getting those results solid, or should it just be worked on outside the official repo until we have confidnce
We would likely want to bundle a default configuration alongside the action, so we could just add the action, point the workflows to
@main and iterate over time.
To start with, all results would be added as job artifacts but the end goal would be to upload the SBOM to R2 and security scans to CodeQL once the false positives are ironed out.
I'm in two minds whether the findings are false positives or not. And how we would decide?
They are all vulnerabilities for something installed on the machine - be it electron, Podman's dependencies or others. Whether that particular code path is used, it will be near impossible to tell.Apprediate the info...
I think if we're never going to block builds based on the security results, I would probably prefer the SBOM generation and security scan is decoupled from the build.
That makes sense if there are ONLY daily builds... but what happens for the extra builds like when things are failing and we have to build several times to get good builds... there will be images in our registry which aren't covered by an SBOM and scan
That makes sense if there are ONLY daily builds... but what happens for the extra builds like when things are failing and we have to build several times to get good builds... there will be images in our registry which aren't covered by an SBOM and scan
Am finally getting round to implementing this
https://github.com/ublue-os/bluefin/pull/1161
@j0rge here's the SBOM thread 🙂
We have scans!
Possibly the most hacky way of doing things, but if anyone has any other ideas, would love to hear them
What I have done:
- Add new artifact in each build_container matrix with a file containing the registry and digest
- Add new step in the "check" job to download and merge all artifacts from each matrix into one artifact
- Do some string magic to translate that into '["<image>@<digest>", ...]'
- Set that string as a workflow output
- Create a new reusable workflow that accepts the string as an input
- JSONify that string so it can be used in the matrix job
- Run Syft and Grype to create and scan the SBOM
I'm thinking the GitHub runners don't have enough memory to scan the Nvidia images 😦
Generating the SBOM cancels after 7 minutes for no apparent reason during Nvidia.
Edit: Excluding some files (like /sysroot/ostree/repo/objects) made the scans succeed, but it's delaying the issue.
PR ready for review!
https://github.com/ublue-os/bluefin/pull/1161
GitHub
feat: add image scanning workflow by p5 · Pull Request #1161 · ublu...
Creates a reusable workflow to scan images. This scan generates a SBOM and checks that SBOM for known vulnerabilities.
The only way I was able to merge the outputs of each matrix run was to use ar...
oh that's fine, that's great feedback for github
Would be great to get some reviews when you both have a second.
Want to roll it out to Bluefin and Aurora in separate PRs
Want to roll it out to Bluefin and Aurora in separate PRs
this is beyond my tech skill but I think we want this. 😄
full sboms and stuff would be baller, nice choice of syft too, looking to ship grype in bluefin-dx
like this would be a cool announcement to make
syft scan docker:ghcr.io/ublue-os/bluefin
I wish this is what the ostree update would look like!
@Robert (p5) how would an end user consume the sbom? or are we waiting for them to go into the registry and then figure that out?So the SBOMs are mainly for us, but will still be accessible to the user (ideally by downloading them from R2).
We generate the SBOMs and scan the packages contained with Grype. The outputs can then be pushed to GitHub's Security tab where we (members of the GitHub org) can see what potential vulnerabilities are in our image.
End users may want/need to download the SBOMs for compliance purposes. I've needed to provide SBOMs of EC2s and Lambdas to security teams in the past as they need to check things like licenses and vulns.
End users may want/need to download the SBOMs for compliance purposes. I've needed to provide SBOMs of EC2s and Lambdas to security teams in the past as they need to check things like licenses and vulns.
oh ok, I understand
this is so awesome
In the future, I guess we could use it to generate our own changelogs by comparing two SBOMs.
I think that would be quite easy to do
I think that would be quite easy to do
And there's also this, which I'm not sure how people view it. You upload your SBOMs to GitHub through the API, and they put it somewhere
https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/using-the-dependency-submission-api
My PR is only for generating and scanning the SBOMs, storing the results in job artifacts, but I feel it's enough for a first iteration
yeah I'm curious on how people actually use it in practice
I guess we'll find out, heh