88 Replies
@bsherman in here
w00t
cc @bketelsen
right, so bluefin-dx right
it has to be a seperate image because we're adding a bunch of stuff right
what if the
just devmode switch just added all the things we need in dx to normal bluefin, and you could add/remove any of those components and add more without having to make it a seperate image
like if you want dx but you don't want vscode, you're stuck with itI love this so so much
yes, i think this could be killer for dev
yeah let's try it with incus imo if brian wants to prototype
brian is already prototyping
I like how you (Brian) added docker to bluefin with it... i think this could give us a story for support docker, but not having to pre-install it... even in ucore.
It's one of the things I don't like a lot since it's a duplicate of podman unless users really need it.
I'm thinking about the pattern
not the exact package
but to use docker as an example
you know how the -dx updates are like 1.6GB?
what if docker was an extension instead of in the bluefin image?
(there I go again, getting stuck in specifics)
like my question would be, what if the containerfile for -dx was "pull in these extensions", they'd be diffed updates, etc and it wouldn't be this huge custom layer
right, but let's say someone was like "but I want docker in ucore"
you tell them go make a custom image and they go do that
or they just add a docker sysext
yes, I agree, especially with "dev-mode"/-dx developers need different stuff, so one size fits all is huge, this makes it much more managable and per-user custom
actually what flatcar did is remove docker, containerd, etc OFF of their base image
so you could do flatcar + podman if you wanted to
or flatcar + the entire k8s stack
so the philosophical question is, "what if our base image was actually base-main and people could atomically bolt on the desktop stuff but it wouldn't be a custom image for each one"
actually that'd be an interesting exercise, take base-main and systext in the gnome from GNOME OS.
also what's cool, you can do this on any linux, so it's additive
this is key, it's a philosophical question
like someone will do ublue but with sysext, no problemo
so, if we use Fedora, but strip all it's packages out, and just use these sysexts, then are we Fedora still or a new thing? 🙂
I'm going to reserve my opinion until I try incus. 😄
seems to me we'd be providing our own sysext packages and maintaining those, vs maintaining a few RPM COPRs and just putting things together
yeah, or submitting them to this bakery flatcar is doing
* but enabling those same sysext packages to be used in any systemd linux
DISTRO? Where we're going we don't need a DISTRO
https://github.com/flatcar/sysext-bakery/releases
this is what they have
also, I am not advocating "let's add our own extensions to maintain alongside our existing stuff." that's a road to madness
but if it's "let's get incus from the automated release pipeline upstream instead of hoping someone will package incus in fedora" then I'm 100% behind that
I was thinking you were heading that way and was confused.
that's kyle's schtick. 😄
lol
please enlighten me... i may have misread: It seemed that with sysext-bakery, as I read the butane yaml, there's still a need for someone to host
1) the scripts to build docker/torcx/wasm/inucs sysexts
2) also (maybe?) the resulting
docker.raw images (example)
Am I incorrect, and yes, scripts need to exist(be hosted), but every system would build the images locally?the scripts need to exist (the bakery repo hosts them in this example), and the resulting images are hosted in github releases
in the model they're using they're built on github, nothing is built on the host.
ok, yeah, so i didn't misunderstand...
in my mind, that's at least part of a distro
that's not the only option
for example, you could run a Podman container that builds the output image, then copy the docker.raw output into the correct place
yup, that makes sense too... and is defintely more DIY/dev stuff
it may be the "right" way for some scenarios too. I'm looking at incus and it has lots of build deps like lxc and lxcfs that need to be compiled. Compiling them on an Ubuntu github action builder and copying them into the output image seems like a recipe for disaster
sounds like incus will be a very good prototype then
yeah, I am curious how the selinux stuff will end up looking like
update: running the sysext builder in a fedora toolbox is the thing
current status: boiling the ocean
incus relies on several other things that need to be built
heh
world's longest weekend bk
whoa
so I got the build working fine with a lot of elbow grease
BUT
You can only choose ONE sysext to merge at a time
I tried to add incus and docker. Nope.
/usr already merged
removed the docker sysext, added incus (haven't created unit files yet) and the binaries are there and work as expectedhuh
yeah, that changes my excitement level
theoretically we can get around that at the sysext composition level
i.e. build the docker extension, build the incus extension, merge the two, then squashfs it
so you'd have both
and also early days
which would sort of guide us to having a "dx" sysext, or a matrix of options. "incus-docker.raw", "code-incus-docker.raw" IDK
yeah I expect to see a bunch of best practice things come out of this
reading
I already whined to andy about needing better docs, etc.
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
whoa
The systemd-confext concept follows the same principle as the systemd-sysext(1) functionality but instead of working on /usr and /opt, confext will extend only /etc. Files and directories contained in the confext images outside of the /etc/ hierarchy are not merged, and hence have no effect when included in the image. Formats for these images are of the same as sysext images. The merged hierarchy will be mounted with "nosuid" and (if not disabled via --noexec=false) "noexec".
OH wait
I think I did it wrong
brb
yay!
i used
systemd-sysext merge which works when you haven't merged any yet. I wanted systemd-sysext refresh to add another to a running system
now I have incus and docker
OK this is Game On@bketelsen omg
these are just file trees right?
before they are squashed yes
rpmfusion
and technically you can put the whole tree under the extension directory
so you don't have to squash
explain?
what if all the codes and stuff were just an extension
instead of this multi RPM/repo garbo fire we have now
but I would have kernel questions
yeah we could probably get there
I still have questions about load order
<--- all theorycrafting right now
damn I really regret missing all systems go
the sysext-bakery readme makes a point about only shipping static binaries. Technically that isn't required if you specify the target OS version correctly. incus specifically has a bunch of deps and requires tools/libs to build, so we'd want to build by fedora version. Different images for f38, f39, etc.
right
I have that question also
why does docker work?
oh, golang
most of the things we'd be adding would be fine for static binaries
but a few of the bigger things use a bunch of shared crap
so I had to build those and put them in the extensions /usr/lib
nod
which translates to a build matrix [38,39] sort of build system
for the dynamically linked ones
OK as I wrap up my work day, here's the final status. I have incus compiling and "packaged" as a sysext, minus the unit files. It has some of the dependency versions hard coded in the shell script, and it isn't a static binary (and won't ever be). This means the sysext we build will be system & version specific e.g. fedora/39.
I created a docker container to do the building, which works great. Next steps to continue:
* add unit files and any other non-binary stuff
* write github actions matrix builder
* document required plumbing in the OS to support this (enable systemd-sysext, create target directory like /etc/extensions)
* write a just target to do stuff with extensions
* profit
unknowns: when merging a new extension with
systemd-sysext refresh what happens to running binaries? what happens to running docker containers or incus instances?
generally I think we shouldn't even encourage any changes to a live system, rather just install the extension and reboot
I think we can write a just target like just extension MYEXTENSION and use /etc/os-release to get the right variables to download and move the extension to the right place
summary: this still appears to be a viable path, some unknowns remain, but not nearly as many as this morning.just sharing our experiences will be awesome
moving this to a dockerized build with a justfile driving everything 🙂
chilling with xe in voice
THIS IS WORKING
is podman already on the non-dx images?
switching to podman because it'll be there for everyone
podman is on every image
i have docker, docker-compose, and almost all of incus building
fed the baby early this morning, didn't go back to bed because I was excited to continue. Now have all of incus dependencies (cowsql, raft, lxc, lxcfs) built and installing
hahahahah
amazing
what a story, I can picture Andy literally laughing at loud as you tell this
I like the new workflow. Clone the repo and ‘just docker’ —> done
beautiful
I can't believe you did it in a day, how long would the rpm taken you?
Weeks for me. I’ve never done one
son of a biscuit eater
I need systemd v255 to make this all work beautifully
it's in rawhide for F40
jeefy was thinking "this will be a 2H 2024, 2025 thing I bet"
but leaning forward in the saddle is always a good idea anyway
i can make it work inelegantly
but I'll need to do some exploring to figure out which systemd boot target to mess with
the flatcar systemd unit that auto-restarts systemd (which won't be needed with systemd 255) blocks graphical login
https://raw.githubusercontent.com/flatcar/init/ccade77b6d568094fb4e4d4cf71b867819551798/systemd/system/ensure-sysext.service
i'm not sure if it's a startup / kill loop when systemd restarts or what. If I could find a different target for the After= I think it would be better
so as of now I have beautiful extensions built, but you have to manually
systemctl daemon-reload after booting
now if you don't need to ship a systemd unit file (like docker and incus do) we're golden
we could ship all the friggin cli glam we want. just can't auto-start any services
however...
https://systemd.io/PORTABLE_SERVICES/
same concept, but already supported
i got some portable services working, but they're a strange duck. They don't overlay over /usr like sysexts do so like in the case of docker you could start the service/socket but can't put the client binary somewhere it would be found. It also litters unit files in /etc/systemd/system.attached that could be abandoned after the service is stopped and removed.
feels like a great fit for deploying a service with no cli, though. Like your own vaultwarden instance, etc. particularly interesting is that it carries some of the traits of systemd-nspawn. you can make your service in a debian file tree and run it on bluefin, for example.
interesting concept, not useful for deploying docker/incus.
@j0rge what if I whip up a fleek binaries only sysext? Atuin, Eza, nvim and friends
I suppose we could put vscode on there too
Or start with one and make multiples so you can pick and choose
It works fine as is as long as there are no systemd services so it’s a good vehicle to ship random binaries@bketelsen I think it's high time we did a :testing branch
I've been avoiding it but it might work out better
or a -next or something
with RAWHIDE???
because I'd love to test this out long term but also looking at the calendar, where F40 and Kubecon sit on the schedule, installer's close. I'm thinking we should start to become less churning in the main branch in bluefin
hahaha, you went there
don't threaten me with a good time
no I mean bluefin:testing like how bazzite does it
FINE
but also it sits in another repo right?
so it's a "just sysext install this one" right?
yes
sgtm
i currently have it set up for local build & install
nod
no github releases/artifacts/downloads
but that's a possibility if we like it
not complicated
cc @Kyle Gospo
goodbye layering
ostree = solid image, exts = things that need to be on the host that fedora doesn't have
@bketelsen how do updates work, assuming github is building vscode regularly, I mean on the client
the workflow TODAY is clone the repo, change version number variables in the justfile, then use the just recipes to (re)install
the next logical step is to build them in GH Actions and release them
then we'd build a helper script for the updating from github
flatcar has a service that runs to update/install them from a configuration file
GitHub
sysext-bakery/README.md at main · flatcar/sysext-bakery
Recipes for baking systemd-sysext images. Contribute to flatcar/sysext-bakery development by creating an account on GitHub.
this shit is so cool
I want this rn
i didn't even know systemd-sysupdate was a thing
@Kyle Gospo clone the repo and GO
GitHub
GitHub - ublue-os/sysext: ublue systemd-sysext
ublue systemd-sysext. Contribute to ublue-os/sysext development by creating an account on GitHub.
the documentation is ummm. sparse... but you can follow the vscode and neovim recipes in just and the create_xxx scripts if you want to make & contribute more
i intend to keep writing all the things I need until I don't need a
-dx image anymore
note: the incus recipe is incomplete. systemd 254 (which we ship in f39) can't start/restart units in extensions. 255 can. so when f40 lands, it's GAME ON for docker and incus and other system services.@Kyle Gospo we can cut image size down hardcore too