40 Replies
talk to me about this
i feel dense, and i've tried to understand more deeply this issue, but i still feel somewhat ignorant
So...
Currently our actions use a single GITHUB_TOKEN secret (abstracted from us) which provides read/write permissions to all registries in that GitHub repo. Any PR has access to this GITHUB_TOKEN, so could in theory overwrite the "production" tags (user-facing tags - i.e. ":latest"). If we had two Quay (or other provider) registries, we can store credentials for each in GitHub Actions. One "production" registry with creds only accessible from the main branch in GitHub, and another "sandbox" registry with creds accessible in PRs. So a build in a PR will use the sandbox credentials, and the exact same workflow running from the main branch would have access to the prod registry credentials. There's no way for an image built and pushed from a PR would end up in the production registry. We could even sign with a different key The only requirement for this to work is separate credentials for each registry
Currently our actions use a single GITHUB_TOKEN secret (abstracted from us) which provides read/write permissions to all registries in that GitHub repo. Any PR has access to this GITHUB_TOKEN, so could in theory overwrite the "production" tags (user-facing tags - i.e. ":latest"). If we had two Quay (or other provider) registries, we can store credentials for each in GitHub Actions. One "production" registry with creds only accessible from the main branch in GitHub, and another "sandbox" registry with creds accessible in PRs. So a build in a PR will use the sandbox credentials, and the exact same workflow running from the main branch would have access to the prod registry credentials. There's no way for an image built and pushed from a PR would end up in the production registry. We could even sign with a different key The only requirement for this to work is separate credentials for each registry
thank you for a good explanation.
it seems this 100% is tied to the fact that the default GITHUB_TOKEN is used for everything in github actions
simple to understand when you explain it like this 😄
I'd be down to use/push to quay.io - ideally later
because there could also be people who have bad github connectivity.
I am going to implement this in my own custom image.
As a proof of concept.
I already push and sign PR images anyway because my use case is so simple and I have no contributors on it.
All we'd need to switch over is a systemd unit that runs the rebase command I think. Add it in the final image pushed to GHCR and remove it in the first image pushed to Quay
as far as removing our key?
I think our key can stay. I was meaning to automate
rpm-ostree rebase to the new quay registryAlso @j0rge I think we should be on more than just ghcr and quay. I think we should be on the docker registry too.
yeah but that one is more complicated
the docker registry? how so?
ratelimiting etc.
oh, would we have to pay or something?
like we have to apply to be exempt, it's paperwork
agree we should tho
helps avoid a single point of failure.
though if Github goes down, we are screwed anyway lol
I want a proxy infront of them all
Might experiment tonight
actually.
that is a really good idea.
we could do a hosted proxy through cloudflare or something.
I'd love to say it was my own, but it's been mentioned before
we investigated this earlier in the project but nacked it
if we had a proxy, folks could pull down from whatever registry is closest to them
(it's why ublue.it exists)
yeah, I don't think it is a big enough problem, ghcr has served us well enough.
I do think having a separate sandbox registry would solve our problem though for PRs.
Would also be nice to show people you could point to your own caching registry if you want to only pull from the registries to one local server
the gitea discussion in #🦈bluefin got me wondering about this thread again...
so... we need distinct credentials...
What about this idea?
Create a PAT which only allows publishing images to GHCR on a distinct Github org (eg, https://github.com/ublue-os-test/) and provide said PAT to our workflows, then our workflows can publish all PRs to that location.
hmm... it works, I guess, but doesn't protect from a bad actor changing the workflow in PR to publish to our main repo
Would kinda need to be the other way around - move the "production" registry to another org.
We can restrict access to secrets based on the branches, but not restrict usage of GITHUB_TOKEN
exactly
i wonder if we can remove the "publish to registry" provilege from the GITHUB_TOKEN default permissions... then we can use a PAT by environment for both PRODUCTION (ublue-os) and STAGING (ublue-os-test)
GITHUB_TOKEN permissions are not able to be limited like that. You can set the default permissions for each repo / the entire org to read-only, but can't prevent someone adding
permissions:\n packages: write in the workflowi think we can
hmm
YES! That would work!
I DO think we probably need to implement an org management tool to enforce settings/permissions before trying to implement something like this...
seems easy to miss something manually
No API for this setting when I checked 2 months ago 😦
@j0rge forgive me for asking probably the 10th time... what's the latest product you think we should try for org management?
i want to create an issue in project plan
Minder
I thought I disconnected from it so you could manage it?
you did... but I lose track of things quickly without a tracker 😄
https://github.com/stacklok/minder
You want their cloud service
draft item in project
I'll be home soon and whack in some details
Goal for doing environments to enable pushing images on PRs
https://github.com/orgs/ublue-os/projects/1/views/1?pane=issue&itemId=61241610
@Noel
@Kyle Gospo
@EyeCantCU