UB
Universal Blue•11mo ago
j0rge

systemd-sysext discussion

systemd-sysext discussion
88 Replies
j0rge
j0rgeOP•11mo ago
@bsherman in here
bsherman
bsherman•11mo ago
w00t
j0rge
j0rgeOP•11mo ago
cc @bketelsen right, so bluefin-dx right it has to be a seperate image because we're adding a bunch of stuff right what if the just devmode switch just added all the things we need in dx to normal bluefin, and you could add/remove any of those components and add more without having to make it a seperate image like if you want dx but you don't want vscode, you're stuck with it
bketelsen
bketelsen•11mo ago
I love this so so much
bsherman
bsherman•11mo ago
yes, i think this could be killer for dev
j0rge
j0rgeOP•11mo ago
yeah let's try it with incus imo if brian wants to prototype
bketelsen
bketelsen•11mo ago
brian is already prototyping
bketelsen
bketelsen•11mo ago
No description
bsherman
bsherman•11mo ago
I like how you (Brian) added docker to bluefin with it... i think this could give us a story for support docker, but not having to pre-install it... even in ucore. It's one of the things I don't like a lot since it's a duplicate of podman unless users really need it.
j0rge
j0rgeOP•11mo ago
I'm thinking about the pattern not the exact package but to use docker as an example you know how the -dx updates are like 1.6GB? what if docker was an extension instead of in the bluefin image?
bsherman
bsherman•11mo ago
(there I go again, getting stuck in specifics)
j0rge
j0rgeOP•11mo ago
like my question would be, what if the containerfile for -dx was "pull in these extensions", they'd be diffed updates, etc and it wouldn't be this huge custom layer right, but let's say someone was like "but I want docker in ucore" you tell them go make a custom image and they go do that or they just add a docker sysext
bsherman
bsherman•11mo ago
yes, I agree, especially with "dev-mode"/-dx developers need different stuff, so one size fits all is huge, this makes it much more managable and per-user custom
j0rge
j0rgeOP•11mo ago
actually what flatcar did is remove docker, containerd, etc OFF of their base image so you could do flatcar + podman if you wanted to or flatcar + the entire k8s stack so the philosophical question is, "what if our base image was actually base-main and people could atomically bolt on the desktop stuff but it wouldn't be a custom image for each one" actually that'd be an interesting exercise, take base-main and systext in the gnome from GNOME OS. also what's cool, you can do this on any linux, so it's additive
bsherman
bsherman•11mo ago
this is key, it's a philosophical question
j0rge
j0rgeOP•11mo ago
like someone will do ublue but with sysext, no problemo
bsherman
bsherman•11mo ago
so, if we use Fedora, but strip all it's packages out, and just use these sysexts, then are we Fedora still or a new thing? 🙂
j0rge
j0rgeOP•11mo ago
I'm going to reserve my opinion until I try incus. 😄
bsherman
bsherman•11mo ago
seems to me we'd be providing our own sysext packages and maintaining those, vs maintaining a few RPM COPRs and just putting things together
j0rge
j0rgeOP•11mo ago
yeah, or submitting them to this bakery flatcar is doing
bketelsen
bketelsen•11mo ago
* but enabling those same sysext packages to be used in any systemd linux DISTRO? Where we're going we don't need a DISTRO
j0rge
j0rgeOP•11mo ago
https://github.com/flatcar/sysext-bakery/releases this is what they have also, I am not advocating "let's add our own extensions to maintain alongside our existing stuff." that's a road to madness but if it's "let's get incus from the automated release pipeline upstream instead of hoping someone will package incus in fedora" then I'm 100% behind that
bsherman
bsherman•11mo ago
I was thinking you were heading that way and was confused.
j0rge
j0rgeOP•11mo ago
that's kyle's schtick. 😄 lol
bsherman
bsherman•11mo ago
please enlighten me... i may have misread: It seemed that with sysext-bakery, as I read the butane yaml, there's still a need for someone to host 1) the scripts to build docker/torcx/wasm/inucs sysexts 2) also (maybe?) the resulting docker.raw images (example) Am I incorrect, and yes, scripts need to exist(be hosted), but every system would build the images locally?
bketelsen
bketelsen•11mo ago
the scripts need to exist (the bakery repo hosts them in this example), and the resulting images are hosted in github releases in the model they're using they're built on github, nothing is built on the host.
bsherman
bsherman•11mo ago
ok, yeah, so i didn't misunderstand... in my mind, that's at least part of a distro
bketelsen
bketelsen•11mo ago
that's not the only option for example, you could run a Podman container that builds the output image, then copy the docker.raw output into the correct place
bsherman
bsherman•11mo ago
yup, that makes sense too... and is defintely more DIY/dev stuff
bketelsen
bketelsen•11mo ago
it may be the "right" way for some scenarios too. I'm looking at incus and it has lots of build deps like lxc and lxcfs that need to be compiled. Compiling them on an Ubuntu github action builder and copying them into the output image seems like a recipe for disaster
bsherman
bsherman•11mo ago
sounds like incus will be a very good prototype then
j0rge
j0rgeOP•11mo ago
yeah, I am curious how the selinux stuff will end up looking like
bketelsen
bketelsen•11mo ago
update: running the sysext builder in a fedora toolbox is the thing current status: boiling the ocean
export PKG_CONFIG_PATH="${SCRIPTFOLDER}/${SYSEXTNAME}/usr/local/lib/pkgconfig"
export C_INCLUDE_PATH="${SCRIPTFOLDER}/${SYSEXTNAME}/usr/local/include"
export LD_LIBRARY_PATH="${SCRIPTFOLDER}/${SYSEXTNAME}/usr/local/lib"
export PKG_CONFIG_PATH="${SCRIPTFOLDER}/${SYSEXTNAME}/usr/local/lib/pkgconfig"
export C_INCLUDE_PATH="${SCRIPTFOLDER}/${SYSEXTNAME}/usr/local/include"
export LD_LIBRARY_PATH="${SCRIPTFOLDER}/${SYSEXTNAME}/usr/local/lib"
incus relies on several other things that need to be built
j0rge
j0rgeOP•11mo ago
heh world's longest weekend bk
bketelsen
bketelsen•11mo ago
whoa so I got the build working fine with a lot of elbow grease BUT You can only choose ONE sysext to merge at a time I tried to add incus and docker. Nope. /usr already merged removed the docker sysext, added incus (haven't created unit files yet) and the binaries are there and work as expected
j0rge
j0rgeOP•11mo ago
huh
bketelsen
bketelsen•11mo ago
yeah, that changes my excitement level theoretically we can get around that at the sysext composition level i.e. build the docker extension, build the incus extension, merge the two, then squashfs it so you'd have both
j0rge
j0rgeOP•11mo ago
and also early days
bketelsen
bketelsen•11mo ago
which would sort of guide us to having a "dx" sysext, or a matrix of options. "incus-docker.raw", "code-incus-docker.raw" IDK
j0rge
j0rgeOP•11mo ago
yeah I expect to see a bunch of best practice things come out of this
bketelsen
bketelsen•11mo ago
reading
j0rge
j0rgeOP•11mo ago
I already whined to andy about needing better docs, etc.
bketelsen
bketelsen•11mo ago
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html whoa The systemd-confext concept follows the same principle as the systemd-sysext(1) functionality but instead of working on /usr and /opt, confext will extend only /etc. Files and directories contained in the confext images outside of the /etc/ hierarchy are not merged, and hence have no effect when included in the image. Formats for these images are of the same as sysext images. The merged hierarchy will be mounted with "nosuid" and (if not disabled via --noexec=false) "noexec". OH wait I think I did it wrong brb yay! i used systemd-sysext merge which works when you haven't merged any yet. I wanted systemd-sysext refresh to add another to a running system now I have incus and docker OK this is Game On
j0rge
j0rgeOP•11mo ago
@bketelsen omg these are just file trees right?
bketelsen
bketelsen•11mo ago
before they are squashed yes
j0rge
j0rgeOP•11mo ago
rpmfusion
bketelsen
bketelsen•11mo ago
and technically you can put the whole tree under the extension directory so you don't have to squash explain?
j0rge
j0rgeOP•11mo ago
what if all the codes and stuff were just an extension instead of this multi RPM/repo garbo fire we have now but I would have kernel questions
bketelsen
bketelsen•11mo ago
yeah we could probably get there I still have questions about load order
j0rge
j0rgeOP•11mo ago
<--- all theorycrafting right now damn I really regret missing all systems go
bketelsen
bketelsen•11mo ago
the sysext-bakery readme makes a point about only shipping static binaries. Technically that isn't required if you specify the target OS version correctly. incus specifically has a bunch of deps and requires tools/libs to build, so we'd want to build by fedora version. Different images for f38, f39, etc.
j0rge
j0rgeOP•11mo ago
right I have that question also why does docker work? oh, golang
bketelsen
bketelsen•11mo ago
most of the things we'd be adding would be fine for static binaries but a few of the bigger things use a bunch of shared crap so I had to build those and put them in the extensions /usr/lib
j0rge
j0rgeOP•11mo ago
nod
bketelsen
bketelsen•11mo ago
which translates to a build matrix [38,39] sort of build system for the dynamically linked ones OK as I wrap up my work day, here's the final status. I have incus compiling and "packaged" as a sysext, minus the unit files. It has some of the dependency versions hard coded in the shell script, and it isn't a static binary (and won't ever be). This means the sysext we build will be system & version specific e.g. fedora/39. I created a docker container to do the building, which works great. Next steps to continue: * add unit files and any other non-binary stuff * write github actions matrix builder * document required plumbing in the OS to support this (enable systemd-sysext, create target directory like /etc/extensions) * write a just target to do stuff with extensions * profit unknowns: when merging a new extension with systemd-sysext refresh what happens to running binaries? what happens to running docker containers or incus instances? generally I think we shouldn't even encourage any changes to a live system, rather just install the extension and reboot I think we can write a just target like just extension MYEXTENSION and use /etc/os-release to get the right variables to download and move the extension to the right place summary: this still appears to be a viable path, some unknowns remain, but not nearly as many as this morning.
j0rge
j0rgeOP•11mo ago
just sharing our experiences will be awesome
bketelsen
bketelsen•11mo ago
moving this to a dockerized build with a justfile driving everything 🙂
j0rge
j0rgeOP•11mo ago
chilling with xe in voice
bketelsen
bketelsen•11mo ago
THIS IS WORKING is podman already on the non-dx images? switching to podman because it'll be there for everyone
j0rge
j0rgeOP•11mo ago
podman is on every image
bketelsen
bketelsen•11mo ago
i have docker, docker-compose, and almost all of incus building fed the baby early this morning, didn't go back to bed because I was excited to continue. Now have all of incus dependencies (cowsql, raft, lxc, lxcfs) built and installing
j0rge
j0rgeOP•11mo ago
hahahahah amazing what a story, I can picture Andy literally laughing at loud as you tell this
bketelsen
bketelsen•11mo ago
I like the new workflow. Clone the repo and ‘just docker’ —> done
j0rge
j0rgeOP•11mo ago
beautiful I can't believe you did it in a day, how long would the rpm taken you?
bketelsen
bketelsen•11mo ago
Weeks for me. I’ve never done one son of a biscuit eater I need systemd v255 to make this all work beautifully it's in rawhide for F40
j0rge
j0rgeOP•11mo ago
jeefy was thinking "this will be a 2H 2024, 2025 thing I bet" but leaning forward in the saddle is always a good idea anyway
bketelsen
bketelsen•11mo ago
i can make it work inelegantly but I'll need to do some exploring to figure out which systemd boot target to mess with the flatcar systemd unit that auto-restarts systemd (which won't be needed with systemd 255) blocks graphical login https://raw.githubusercontent.com/flatcar/init/ccade77b6d568094fb4e4d4cf71b867819551798/systemd/system/ensure-sysext.service i'm not sure if it's a startup / kill loop when systemd restarts or what. If I could find a different target for the After= I think it would be better so as of now I have beautiful extensions built, but you have to manually systemctl daemon-reload after booting now if you don't need to ship a systemd unit file (like docker and incus do) we're golden we could ship all the friggin cli glam we want. just can't auto-start any services however... https://systemd.io/PORTABLE_SERVICES/ same concept, but already supported i got some portable services working, but they're a strange duck. They don't overlay over /usr like sysexts do so like in the case of docker you could start the service/socket but can't put the client binary somewhere it would be found. It also litters unit files in /etc/systemd/system.attached that could be abandoned after the service is stopped and removed. feels like a great fit for deploying a service with no cli, though. Like your own vaultwarden instance, etc. particularly interesting is that it carries some of the traits of systemd-nspawn. you can make your service in a debian file tree and run it on bluefin, for example. interesting concept, not useful for deploying docker/incus. @j0rge what if I whip up a fleek binaries only sysext? Atuin, Eza, nvim and friends I suppose we could put vscode on there too Or start with one and make multiples so you can pick and choose It works fine as is as long as there are no systemd services so it’s a good vehicle to ship random binaries
j0rge
j0rgeOP•11mo ago
@bketelsen I think it's high time we did a :testing branch I've been avoiding it but it might work out better or a -next or something
bketelsen
bketelsen•11mo ago
with RAWHIDE???
j0rge
j0rgeOP•11mo ago
because I'd love to test this out long term but also looking at the calendar, where F40 and Kubecon sit on the schedule, installer's close. I'm thinking we should start to become less churning in the main branch in bluefin hahaha, you went there
bketelsen
bketelsen•11mo ago
don't threaten me with a good time
j0rge
j0rgeOP•11mo ago
no I mean bluefin:testing like how bazzite does it
bketelsen
bketelsen•11mo ago
FINE
j0rge
j0rgeOP•11mo ago
but also it sits in another repo right? so it's a "just sysext install this one" right?
bketelsen
bketelsen•11mo ago
yes
j0rge
j0rgeOP•11mo ago
sgtm
bketelsen
bketelsen•11mo ago
i currently have it set up for local build & install
j0rge
j0rgeOP•11mo ago
nod
bketelsen
bketelsen•11mo ago
no github releases/artifacts/downloads but that's a possibility if we like it not complicated
j0rge
j0rgeOP•11mo ago
j0rge
j0rgeOP•11mo ago
cc @Kyle Gospo goodbye layering ostree = solid image, exts = things that need to be on the host that fedora doesn't have @bketelsen how do updates work, assuming github is building vscode regularly, I mean on the client
bketelsen
bketelsen•11mo ago
the workflow TODAY is clone the repo, change version number variables in the justfile, then use the just recipes to (re)install the next logical step is to build them in GH Actions and release them then we'd build a helper script for the updating from github flatcar has a service that runs to update/install them from a configuration file
bketelsen
bketelsen•11mo ago
GitHub
sysext-bakery/README.md at main · flatcar/sysext-bakery
Recipes for baking systemd-sysext images. Contribute to flatcar/sysext-bakery development by creating an account on GitHub.
Kyle Gospo
Kyle Gospo•11mo ago
this shit is so cool I want this rn
bketelsen
bketelsen•11mo ago
i didn't even know systemd-sysupdate was a thing @Kyle Gospo clone the repo and GO
bketelsen
bketelsen•11mo ago
GitHub
GitHub - ublue-os/sysext: ublue systemd-sysext
ublue systemd-sysext. Contribute to ublue-os/sysext development by creating an account on GitHub.
bketelsen
bketelsen•11mo ago
the documentation is ummm. sparse... but you can follow the vscode and neovim recipes in just and the create_xxx scripts if you want to make & contribute more i intend to keep writing all the things I need until I don't need a -dx image anymore note: the incus recipe is incomplete. systemd 254 (which we ship in f39) can't start/restart units in extensions. 255 can. so when f40 lands, it's GAME ON for docker and incus and other system services.
j0rge
j0rgeOP•11mo ago
@Kyle Gospo we can cut image size down hardcore too
Want results from more Discord servers?
Add your server