ublue/sysext Prototyping

sysext 🧡 follows
268 Replies
bketelsen
bketelsenOPβ€’11mo ago
it turns out I work with most of the people who created and maintain systemd. And almost all of the people behind overlays and systemd-sysext. How convenient! I had a call today with some of them to talk about strategy for bluefin-dx as a sysext. Here are my take-aways:
HikariKnight
HikariKnightβ€’11mo ago
that is very convenient
bketelsen
bketelsenOPβ€’11mo ago
* If you're building or shipping dynamically linked things (complicated apps like lxd/incus/virt-manager etc) then your sysext needs to be built on and for the same os that you target. * UNLESS you build into your own prefix (/usr/bluefin anyone?) and install all binaries and libraries there * systemd-sysext merges images to /usr * systemd-confext merges images to /etc * Yes, we can ship things to /etc in an overlay * Technically you can build your extension once and package it twice, once as a sysext and once as a confext To put all of this into some sort of practice I refactored the ublue-os/sysext code a little bit. I changed the incus build to build all deps and incus itself with a prefix of /usr/bluefin. When merged the extension is completely harmless unless you have systemd units calling binaries in /usr/bluefin/bin directly. Libraries installed there can't conflict with the base system either. I then refactored a bit more to move the sysext building into three phases: prep, compiling/wrangling, image building finally I changed some just recipes to build meta extensions that combine more than one of the previous extensions together. This serves as a proof of concept for one bluefin-dx sysext with all the dev stuff included. It worked perfectly. finally, one of my coworkers wrote an OCI proxy for sysext so you can store your extensions in an oci repo like GHCR so if we go down this route we'd create a matrix of builds for each fedora version we support [38,39,40] then we can build each extension and push them to ghcr. Updating them can be part of the ublue-update service. Technically you can live merge/unmerge extensions but I think we shouldn't. that's about it. I'm very very excited about what this means for all of ublue, because the extensions we build will work outside of bluefin on any of our variants. bazzite-dx comes for free. @bsherman @Kyle Gospo @j0rge F40 ships systemd v255 which supports starting systemd units in extensions automatically
M2
M2β€’11mo ago
so our hacky workaround scripts would then be fine?
bketelsen
bketelsenOPβ€’11mo ago
so I think we should target ublue-dx alpha for the F40 release which ones, specifically? (I have a lot of hacky workaround scripts :))
M2
M2β€’11mo ago
Groups, putting things like a default vscode config, our SELinux stuff capabilities not being right I'm guessing this would be in the sysext so would be fine
bketelsen
bketelsenOPβ€’11mo ago
ah yes, all of those things can be put in systemd-confext images groups get written to /etc/sysusers.d/ for example when the extension merges, systemd creates the groups last thing - the ublue-os/sysext repo is doing things manually - the hard way but mkiso v20 and higher support building sysexts directly create a base image, then a sysext image. mkiso ships the delta between the base & sysext so that's really slick and powerful
Noel
Noelβ€’11mo ago
I am very interested to see where this goes. If I'm understanding correctly, this could be useful for packaging binaries and config files that can be very easily switched on and off while the system is running?
bketelsen
bketelsenOPβ€’11mo ago
correct @Noel kind of like Neo "I know kung-fu" "I have vscode and docker"
Noel
Noelβ€’11mo ago
So is there more that has to be done to get the binaries in the form needed or can they be sourced from RPM or Deb packages? Or are they compiled from source?
bketelsen
bketelsenOPβ€’11mo ago
that's a choice we would make early. if we're based on fedora 39 and we use f39 rpms to install stuff we'd be fine the biggest issue is libraries and compatibility. that's the story behind my 1st and 2nd points in this thread it's safest to put everything in it's own prefix. That's how I did incus, neovim, and vscode. So nothing they bring with them could hurt the base (host) OS. but... if we're only installing things that are available by RPM there should be no problem just using them in their default locations. That's what redhat/fedora do for us, make sure all that crap works together.
j0rge
j0rgeβ€’11mo ago
yeah if they're built on the OS then we should be fine? the /usr/bluefin thing, dunno, I'd need to read further like we don't need to build in a prefix right? we would just build it on a fedora builder?
bketelsen
bketelsenOPβ€’11mo ago
the OS owns /usr, the host owns /usr/local, and bluefin would own /usr/bluefin. it's the same concept, and not included in any PATHs by default so that makes it safe just like crap you install in ~/.local can't mess with /usr correct yes, but some apps bring in deps not packaged in fedora so you risk polluting system libs
j0rge
j0rgeβ€’11mo ago
ah ok so purposely in a prefix
M2
M2β€’11mo ago
like incus has some dependencies that are not in F40
bketelsen
bketelsenOPβ€’11mo ago
neovim uses libssl3.2 but fedora updates to 3.4, etc
j0rge
j0rgeβ€’11mo ago
ok so if the prefix is not in the path how do you "run incus" if it's not in the path?
bketelsen
bketelsenOPβ€’11mo ago
you add /usr/bluefin/bin to your user's path either in your dotfiles or in /etc/profile.d/ whatever but you do it intentionally
j0rge
j0rgeβ€’11mo ago
right, so say I do that, and I run neovim
bketelsen
bketelsenOPβ€’11mo ago
and the libraries are in /usr/bluefin/lib etc.
j0rge
j0rgeβ€’11mo ago
where does neovim look for the library?
Noel
Noelβ€’11mo ago
We could include scripting to do that maybe in our just file and also add a desktop file for desktop apps?
bketelsen
bketelsenOPβ€’11mo ago
the desktop files just work
M2
M2β€’11mo ago
Can confext just be added /etc/profile.d/ ?
bketelsen
bketelsenOPβ€’11mo ago
oh that was before i put it in /usr/bluefin
j0rge
j0rgeβ€’11mo ago
right because they're explicitly calling the full binary path right like the .desktop calls /usr/bluefin/bin/neovim right?
M2
M2β€’11mo ago
Yes a good dsktop file should do that
bketelsen
bketelsenOPβ€’11mo ago
if /usr/bluefin/bin/ is in your PATH and you type nvim it'll run from there
M2
M2β€’11mo ago
only if it's first in path
bketelsen
bketelsenOPβ€’11mo ago
if it's not then nvim won't be found right - I'm assuming a non-dx bluefin image with nothing else installed.
j0rge
j0rgeβ€’11mo ago
right, ok so what about complex apps that will have things that collide with the host?
bketelsen
bketelsenOPβ€’11mo ago
that's what the prefix is for they won't and can't collide in a prefix
M2
M2β€’11mo ago
So brew situation? An app had a dependency on p11-kit.
j0rge
j0rgeβ€’11mo ago
unless I put the bluefin dir in my path right
M2
M2β€’11mo ago
and brew was first on path meaning brew's p11-kit was used instead of fedoras
bketelsen
bketelsenOPβ€’11mo ago
like incus: It's built knowing that it lives in /usr/bluefin. It looks for its libs in /usr/bluefin/lib
j0rge
j0rgeβ€’11mo ago
so with brew I installed ffmpeg and that breaks it. if I installed ffmpeg this way and needed to put it in my path anyway then how does it keep it seperate.
bketelsen
bketelsenOPβ€’11mo ago
nothing else even knows /usr/bluefin exists so nothing else breaks
j0rge
j0rgeβ€’11mo ago
OH OH!
M2
M2β€’11mo ago
wait how?!
j0rge
j0rgeβ€’11mo ago
but that's assuming they use a .desktop file right? what happens if I type ffmpeg iin the command line
bketelsen
bketelsenOPβ€’11mo ago
when you comple stuff you tell it where it's going to live. that's the --prefix when you run gcc & friends the default prefix is /usr
M2
M2β€’11mo ago
tracking
bketelsen
bketelsenOPβ€’11mo ago
when you build without specifying that it installs to /usr/bin/nvim if you specify --prefix=/usr/bluefin it'll compile to /usr/bluefin/bin but it won't work unless its dependencies are in /usr/bluefin/lib and /usr/bluefin/share
j0rge
j0rgeβ€’11mo ago
right but in order to use ffmpeg I'd need to add it to my path right?
bketelsen
bketelsenOPβ€’11mo ago
so it's a little sandbox keeping your host from breaking yes, but just the binary PATH=$PATH:/usr/bluefin/bin
j0rge
j0rgeβ€’11mo ago
ok, got it sp the pk11 thing is never exposed
bketelsen
bketelsenOPβ€’11mo ago
/usr/bluefin/bin/ffmpeg is linked against all the stuff in /usr/bluefin/lib etc. and you never get conflicts no matter what you do on the host
Noel
Noelβ€’11mo ago
So similar to flatpak apps, these apps all ship with their own libraries? Or can some of them be shared?
bketelsen
bketelsenOPβ€’11mo ago
they can be shared our "base" could have gcc for example, installed in the normal place so everything can use it but incus, neovim, ffmpeg, etc are in /usr/bluefin so they'll only be found if you add them to yoru path or if your unit files call them directly i am really feeling this it's a really safe way to have a whole dev toolbox that's isolated from the host
j0rge
j0rgeβ€’11mo ago
it's nice because you can try it and then just throw it away low commitment
bketelsen
bketelsenOPβ€’11mo ago
YUP
M2
M2β€’11mo ago
This ends up sounding similar to brew. But instead put brew at end of $PATH whihc is what we have in Wolfi. /usr/bluefin/bin = /home/linuxbrew/.linuxbrew/bin But with a bit of magic. So only concerns I have is handling $PATH since gdm will bring in $PATH set by login shell to systemd-user session.
j0rge
j0rgeβ€’11mo ago
like, you could conceivably just add bluefin-dx to a normal ubuntu install lol
bketelsen
bketelsenOPβ€’11mo ago
rm /var/lib/extensions/bluefin-dx.raw && systemd-sysext refresh. -it's gone
Noel
Noelβ€’11mo ago
I'm really feeling this from the perspective I can add tools I need without requiring a reboot that have similar access to the system that containers take more setup to achieve.
j0rge
j0rgeβ€’11mo ago
sgtm +1, my only concern is we're adding a bunch of new work without closing out old work so it'd be nice to get some tech debt sorted since the ISO is our current countdown deadline.
bketelsen
bketelsenOPβ€’11mo ago
that's behaving as intended. If you add /usr/bluefin/bin to your path gdm should know it's there so you can find ffmpeg and neovim
j0rge
j0rgeβ€’11mo ago
2nd step would be another selection in the devmode chooser
M2
M2β€’11mo ago
when you use login, systemd-user session doesn't get those
bketelsen
bketelsenOPβ€’11mo ago
and from an end user perspective the apps/tools are there like they were installed from the ISO
j0rge
j0rgeβ€’11mo ago
No description
j0rge
j0rgeβ€’11mo ago
maybe like "use experimental system extension" and then a link or something, charm will make it look good
M2
M2β€’11mo ago
Can we get project board going for the major movements?
j0rge
j0rgeβ€’11mo ago
there's a prexisting projectboard at the org
j0rge
j0rgeβ€’11mo ago
GitHub
F39 Project Goals β€’ ublue-os
This board is for organization-wide images and infrastructure.
j0rge
j0rgeβ€’11mo ago
needs updates
bketelsen
bketelsenOPβ€’11mo ago
❯ which nvim
/usr/local/bin/nvim

~
❯ systemd-sysext
HIERARCHY EXTENSIONS SINCE
/opt none -
/usr meta Wed 2024-01-17 14:51:19 EST
❯ which nvim
/usr/local/bin/nvim

~
❯ systemd-sysext
HIERARCHY EXTENSIONS SINCE
/opt none -
/usr meta Wed 2024-01-17 14:51:19 EST
meta is a single sysext with vscode and neovim in it
sudo systemd-sysext unmerge
[sudo] password for bjk:
Unmerged '/usr'.

~ took 3s
❯ which nvim
/usr/bin/which: no nvim in (/home/bjk/.local/bin:/home/bjk/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin)
sudo systemd-sysext unmerge
[sudo] password for bjk:
Unmerged '/usr'.

~ took 3s
❯ which nvim
/usr/bin/which: no nvim in (/home/bjk/.local/bin:/home/bjk/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin)
Noel
Noelβ€’11mo ago
Blame me for not updating it. I'll try to get it updated soon.
bketelsen
bketelsenOPβ€’11mo ago
^^ the above example was built without a special prefix when I was testing earlier here's what it looks like with the prefix
sudo cp incus.raw /var/lib/extensions/

~
❯ sudo systemd-sysext refresh
Using extensions 'incus', 'meta'.
Merged extensions into '/usr'.

~
❯ which incus
/usr/bin/which: no incus in (/home/bjk/.local/bin:/home/bjk/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin)

~
❯ ls /usr/bluefin/bin/incus
/usr/b
sudo cp incus.raw /var/lib/extensions/

~
❯ sudo systemd-sysext refresh
Using extensions 'incus', 'meta'.
Merged extensions into '/usr'.

~
❯ which incus
/usr/bin/which: no incus in (/home/bjk/.local/bin:/home/bjk/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin)

~
❯ ls /usr/bluefin/bin/incus
/usr/b
notice /usr/bluefin/bin isn't in my path so it isn't found!
j0rge
j0rgeβ€’11mo ago
ok here's an idea how about a test
bketelsen
bketelsenOPβ€’11mo ago
~
❯ export PATH=$PATH:/usr/bluefin/bin

~
❯ which incus
/usr/bluefin/bin/incus
~
❯ export PATH=$PATH:/usr/bluefin/bin

~
❯ which incus
/usr/bluefin/bin/incus
j0rge
j0rgeβ€’11mo ago
could you just do homebrew this way?
M2
M2β€’11mo ago
yes.
bketelsen
bketelsenOPβ€’11mo ago
no
M2
M2β€’11mo ago
But we don't get the nice stuff of /usr unmerge
bketelsen
bketelsenOPβ€’11mo ago
homebrew on linus installs to /home/linuxbrew
j0rge
j0rgeβ€’11mo ago
lol, sorry I had to capture this moment:
No description
M2
M2β€’11mo ago
Why no?
bketelsen
bketelsenOPβ€’11mo ago
can't do user stuff in a sysext that's a homedir
j0rge
j0rgeβ€’11mo ago
ah ok
bketelsen
bketelsenOPβ€’11mo ago
i mean we might be able to symlink it or something
j0rge
j0rgeβ€’11mo ago
what if you did wolfi and then apk installed brew
bketelsen
bketelsenOPβ€’11mo ago
but there be dragons
M2
M2β€’11mo ago
okay. I answered thinking differently. I'm saying /home/linuxbrew/.linuxbrew/bin could be at end of $PATH
j0rge
j0rgeβ€’11mo ago
let's skip the dragons imo
M2
M2β€’11mo ago
agreed
bketelsen
bketelsenOPβ€’11mo ago
yup no dragons for us, please
M2
M2β€’11mo ago
I like brew in a container better
j0rge
j0rgeβ€’11mo ago
but I tell you what bk.
M2
M2β€’11mo ago
since then when we exit container.... the brew stuff is clearly not on path
j0rge
j0rgeβ€’11mo ago
you get me a wolfi user space on top of this thing so I can just use the packages natively ... and you might have found The Final Shape
Noel
Noelβ€’11mo ago
We still like dinosaurs πŸ¦–, but no dragons 🐲
bketelsen
bketelsenOPβ€’11mo ago
poor @bsherman has been typing for an hour
j0rge
j0rgeβ€’11mo ago
hahahaha
bketelsen
bketelsenOPβ€’11mo ago
@j0rge there just isn't enough in wolfi to make a full -dx
M2
M2β€’11mo ago
I assume phone typing
bsherman
bshermanβ€’11mo ago
My initial reaction having read the above: <:beagle_wow:1087436058414809202> 🀯 πŸ’™ This is really impressive!
I'm simultaneously very excited and terrified! πŸ˜€ I'll add details.
j0rge
j0rgeβ€’11mo ago
oh I mean I was just going to grab the base image and see what happens
bketelsen
bketelsenOPβ€’11mo ago
but all the wolfi stuff is statically linked so we could take all their binaries and ship them without fear FROM wolfi IMPORT AllTheThings
j0rge
j0rgeβ€’11mo ago
yeah I was like "Can we just use wolfi-base" because once you add that you have the distro in there so you could apk install everything they have
bsherman
bshermanβ€’11mo ago
Similar to Jorge's comment, I have concerns about adding new stuff without resolving old stuff... But... a couple things can make this feel solid for me: - first, important to scope it... initial scope seems to be limited to -dx use case... specifically in bluefin, after proven, we can add to other ublue-os images - second, do we patch to systemd 255 in bluefin as part of the "try it out" scope in F39 or is this on hold until F40? i'm indifferent if scoped to bluefin, just calling it out
bketelsen
bketelsenOPβ€’11mo ago
I have that working here:
bketelsen
bketelsenOPβ€’11mo ago
GitHub
sysext/builders/caddy/Containerfile.caddy at main Β· ublue-os/sysext
EXPERIMENTAL: ublue systemd-sysext. Contribute to ublue-os/sysext development by creating an account on GitHub.
Noel
Noelβ€’11mo ago
Wait, so you're saying we get everything packaged for wolfi can be used in sys-ext?!
j0rge
j0rgeβ€’11mo ago
I want to add to bsherman's first point. People have been asking for a -dx for kde, one for bazzite, etc. And doubling our image production would suck. This would ALSO solve that problem, which is a real one we have
bketelsen
bketelsenOPβ€’11mo ago
the only need for 255 is to restart systemd units on reboot that are installed as extensions
bsherman
bshermanβ€’11mo ago
- third, i'm relatively sold on the -dx use case... but unsure how far sysext can or should go... i still want this to be Fedora
j0rge
j0rgeβ€’11mo ago
like if "-dx" can you plug onto anything, including other distros then it'd immediately be interesting to folks outside of bluefin
bketelsen
bketelsenOPβ€’11mo ago
for the short term we could just inform users to do a systemd daemon-reload
bsherman
bshermanβ€’11mo ago
this seems reasonable, especially for a -dx
j0rge
j0rgeβ€’11mo ago
to add to this, we're not swapping this out midflight at all
bketelsen
bketelsenOPβ€’11mo ago
we're already pulling crap from containers all over the internet in our bluefin-dx build
j0rge
j0rgeβ€’11mo ago
like, once F40 is out the door it's chillops for the guts of bluefin, we'll make a testing branch etc.
bketelsen
bketelsenOPβ€’11mo ago
GitHub
bluefin/Containerfile at main Β· ublue-os/bluefin
An interpretation of the Ubuntu spirit built on Fedora technology - ublue-os/bluefin
bketelsen
bketelsenOPβ€’11mo ago
just search the COPY statements in there
bsherman
bshermanβ€’11mo ago
understood... but upstream ublue-os/main isn't... so I'm just pointing out there's probably some healthy boundary
bketelsen
bketelsenOPβ€’11mo ago
100% agree there is
bsherman
bshermanβ€’11mo ago
also, πŸ˜„ I have some disagreements with the "COPY ALL THE THINGS" approach LOL πŸ˜‰
j0rge
j0rgeβ€’11mo ago
and I don't hold a dogmatic view for Bluefin on "it's Fedora" but that's ok because
bsherman
bshermanβ€’11mo ago
but i love that the common base can learn and share and do what's needed for bazzite vs bluefin etc
j0rge
j0rgeβ€’11mo ago
the sysext is additive, we'd have DX but for someone who wanted to use this the base image is still normal bluefin and also need to remember that this runs on any distro
bketelsen
bketelsenOPβ€’11mo ago
yes. important to note you can have multiple sysexts and they can be shared with any system built on the same underlying os (f39 for example)
j0rge
j0rgeβ€’11mo ago
like when people are like "why can't we have a ublue suse edition" or whatever it's like, here you go, here's -dx.
bketelsen
bketelsenOPβ€’11mo ago
unless you tell it not to, which I'm proposing here. I'm saying we should build sysexts specifically for each fedora release
j0rge
j0rgeβ€’11mo ago
and legit if you could get our vscode setup and our awesome selection of tools on any distro? compton and long beach, you know you're in trouble. yeah
bketelsen
bketelsenOPβ€’11mo ago
we can package some things to be distro independent
j0rge
j0rgeβ€’11mo ago
oh and also
bketelsen
bketelsenOPβ€’11mo ago
vscode would work but incus wouldn't
j0rge
j0rgeβ€’11mo ago
you could have -dx on stock silverblue or rhel.
bketelsen
bketelsenOPβ€’11mo ago
yup but (as I'm proposing it) not on Ubuntu
j0rge
j0rgeβ€’11mo ago
not sure about moving to the Blue? run this sysext for a bit to learn how containers work. You don't even need to reinstall yeah, ubuntu is there as a transitionary container and I'm still interested in exploring sourcing more components from upstream via containers bsherman and I have to balance each other out
Noel
Noelβ€’11mo ago
Not to create more scope creep, but could this be useful for servers?
j0rge
j0rgeβ€’11mo ago
yes. Microsoft makes Flatcar Linux. πŸ˜„
bketelsen
bketelsenOPβ€’11mo ago
@Noel it was created with servers in mind and there are some distros that use it extensively gnome-os is one, flatcar another,
j0rge
j0rgeβ€’11mo ago
there will for sure be desktop and server distros that will consume these
bketelsen
bketelsenOPβ€’11mo ago
in gnome os everything is a sysext
j0rge
j0rgeβ€’11mo ago
we're just earlyish, not first, but early enough, heh. I love how no one has realized that we can take the base images that don't have a DE, add the Cosmic sysext and voila. You'd be done.
bketelsen
bketelsenOPβ€’11mo ago
pepperidge farms has already realized that it's on my todo list this guy is going to try cosmic without reinstalling my linux desktop
Noel
Noelβ€’11mo ago
It's a crazy world. I'm really excited about this!
j0rge
j0rgeβ€’11mo ago
Right. Ok, dramatic moment @bsherman play along ....
bketelsen
bketelsenOPβ€’11mo ago
duh duh dunnnn
j0rge
j0rgeβ€’11mo ago
You know how we're always like we're not making a distro, it's just Fedora <slowly turns head towards bsherman>
bketelsen
bketelsenOPβ€’11mo ago
why you always have to take it to the next level J
j0rge
j0rgeβ€’11mo ago
what if I told you we're just going to get rid of distros.
j0rge
j0rgeβ€’11mo ago
well, "we" aren't doing that
bketelsen
bketelsenOPβ€’11mo ago
in theory we could use ublue-os/main and make sysexts for gnome & kde but we aren't goign to do that either because we have day jobs
bsherman
bshermanβ€’11mo ago
i love the possibilities... i just want to do one step at a time
bketelsen
bketelsenOPβ€’11mo ago
@bsherman reel us in
j0rge
j0rgeβ€’11mo ago
I think the market is just realizing that distros are great, but removing the middle person is always efficient. And if you want, instead of paying that middle person to make RPMs for you, you pay them to keep that project healthy. and also someone will just make an rpmfusion systext OR some kind of distro generic one that will just put all the codecs in the right place
ChaiQi
ChaiQiβ€’11mo ago
if someone can make a kde sysext with its own home dir so all the files it creates/overwrites to mess with Gnome are isolated and containerized, that would be awesome. I don't know if that's how any of that works
bsherman
bshermanβ€’11mo ago
my concern is everything is a bandwagon: - brew - nix - sysext just because CAN do some things, doesn't mean we should
j0rge
j0rgeβ€’11mo ago
that's a different problem all this is system stuff not ~ brew is containerized now nix is not my fault and sysext is going to be huge in the industry there's no escape so that's more of a matter of having to adapt
ChaiQi
ChaiQiβ€’11mo ago
A sysext of KDE is worthless at best if it does not protect the existing system from KDE interference, but more likely an active hazard for users
bsherman
bshermanβ€’11mo ago
my comment was not in reference to implementations within ublue... just how they can be used to do similar things i could add snap and flatpak to the list
j0rge
j0rgeβ€’11mo ago
yeah
bsherman
bshermanβ€’11mo ago
so, i'm 100% on board for the -dx use case... and it scopes the effort into something conceptually managable, i think.
j0rge
j0rgeβ€’11mo ago
and bk knows I am 100% maxed out at work so this will be you and whoever you can recruit because the raid boss is the ISO.
bketelsen
bketelsenOPβ€’11mo ago
can we make a new /dev channel for sysext so folks can opt-in to hearing all this
ChaiQi
ChaiQiβ€’11mo ago
I guess a better way to phrase my point is: if packaging something this way does not actually solve a problem which that thing has, it is not worth doing, and if it enables or increases problems, it is worth not doing
j0rge
j0rgeβ€’11mo ago
actually bk that's what these threads are for, I can retitle it. And crucially it will capture this in answer overflow.
Noel
Noelβ€’11mo ago
Threading it is great actually.
j0rge
j0rgeβ€’11mo ago
how's that?
bketelsen
bketelsenOPβ€’11mo ago
does this persist?
j0rge
j0rgeβ€’11mo ago
yes.
bketelsen
bketelsenOPβ€’11mo ago
fine with me then
j0rge
j0rgeβ€’11mo ago
and in about an hour I'll have a link to it on the web so you can pass stuff along to people working on it, etc.
Noel
Noelβ€’11mo ago
Well yes and no. I think inactive threads do die eventually.
ChaiQi
ChaiQiβ€’11mo ago
It'll be archived after a period of inactivity, but it can be found and revived
j0rge
j0rgeβ€’11mo ago
yeah, and people revive old ones all the time, I'm actually becoming a fan of discord threads it's a "this is what's important right now" outside of github
bketelsen
bketelsenOPβ€’11mo ago
ohh look at that big fat thread button at the top sigh I just found it
j0rge
j0rgeβ€’11mo ago
as you post links pin them, answer overflow will also collect all the links you paste in here, it'll be useful, trust me.
ChaiQi
ChaiQiβ€’11mo ago
Also, hovering over a channel shows its threads, and the "see all" button includes archived ones
j0rge
j0rgeβ€’11mo ago
also idea milestone it when we say "-dx" that's a bunch of stuff just be like "vscode and docker" phase 1 phase 2 "incus" phase 3 "devbox and devpod" and so on like just vscode and docker alone would be a huge step towards bazzite-dx like, the rest is dressing, give them vscode and docker and then people will dive in and then you'd have an org-wide "dx" systext and since it's nice and clean, we shouldn't need to do anything on the actual images, just some custom just commands - a nice clean cut so we can keep bluefin-dx change averse. Like, we've picked our tools, I'm keen to not change any direction on that since it's basically too late @bketelsen tell me about meta, I'll explain
bketelsen
bketelsenOPβ€’11mo ago
meta was my initial name for a combined sysext. what we're calling -dx here one sysext with multiple tools installed that are build & guaranteed not to collide with or break each other
j0rge
j0rgeβ€’11mo ago
dx systext has vscode + kvm, blah blah but I need jetbrains can you meta them together? like if I wanted -dx but I work in a jetbrains shop
bketelsen
bketelsenOPβ€’11mo ago
if jetbrains can be installed under /usr then yes and if we WANTED we could make the IDEs separate meta + vscode meta + jetbrains or all in one meta includes both i just don't want to get bogged down in variations
j0rge
j0rgeβ€’11mo ago
yeah the reason I bring that up is baking in vscode into the image is awesome, unless you don't use vscode. And that sucks.
bketelsen
bketelsenOPβ€’11mo ago
so I'm feeling all or none now and that's the alpha v1 we can ship a selection tool ?
j0rge
j0rgeβ€’11mo ago
you're asking the wrong person, I think you'd know who has the ux for all this in mind more than I would lol
bketelsen
bketelsenOPβ€’11mo ago
in my world i'm happy to ignore vscode if i use jetbrains I ignore a dozen apps on my bluefin install
ChaiQi
ChaiQiβ€’11mo ago
Main thing I want for Jetbrains IDEs is for devcontainers to work
j0rge
j0rgeβ€’11mo ago
yeah but the IDE is kinda a core thing to -dx right, that's the primary tool, it's like your primary weapon in Halo, not your power weapon. It's your every day carry
ChaiQi
ChaiQiβ€’11mo ago
Last time I had to install the entire IDE and all of its dependencies in a distrobox to use a different GCC version
j0rge
j0rgeβ€’11mo ago
but like I'd just spike on vscode and docker and then I bet in 20 minutes someone will PR a jetbrains one.
bketelsen
bketelsenOPβ€’11mo ago
vscode took me 30 minutes jetbrains won't be much longer I'd venture to guess
j0rge
j0rgeβ€’11mo ago
and that's a large enough initial net, then just handle docker which sounds like you solved already
bketelsen
bketelsenOPβ€’11mo ago
yeah I already have docker working, but it won't auto start until F40 which is short-term pain
j0rge
j0rgeβ€’11mo ago
we switched to listen anyway didn't we? it doesn't start the thing until something pings the docker socket, sorry I forgot the right name
bketelsen
bketelsenOPβ€’11mo ago
systemd doesn't know they exist until you refresh it
j0rge
j0rgeβ€’11mo ago
oh
bketelsen
bketelsenOPβ€’11mo ago
you're right, socket activation but the socket is a systemd unit that is "new" on disk until you do a daemon-reload
j0rge
j0rgeβ€’11mo ago
and also a post config setup step compared to the manual config they would have had to do is still so much better "sorry you had to type this one line to get vscode/docker/kvm/incus and some kickass fonts"
bketelsen
bketelsenOPβ€’11mo ago
lol ok i'm going to run with this in the sysext repo. I'll clean up what I have, create issues, milestones, etc in the next day or so
j0rge
j0rgeβ€’11mo ago
I expect all the cloud stuff to grow these, but you don't think upstream vscode would do you?
bketelsen
bketelsenOPβ€’11mo ago
if anyone wants to help I strongly recommend getting your virt-manager world straight
j0rge
j0rgeβ€’11mo ago
I mean, I'm on like year 3 trying to get them to do an official flatpak
bketelsen
bketelsenOPβ€’11mo ago
make a bluefin install and snapshot it before you start changing it no but I do think there can and will be a vscode sysext
bsherman
bshermanβ€’11mo ago
can we put libvirt and friends in a sysext?
bketelsen
bketelsenOPβ€’11mo ago
that works everywhere yes, eventually
HikariKnight
HikariKnightβ€’11mo ago
now that is exciting for me
bketelsen
bketelsenOPβ€’11mo ago
it's back to system services starting at boot
HikariKnight
HikariKnightβ€’11mo ago
i will make sure to test if it breaks gpu passthrough
bketelsen
bketelsenOPβ€’11mo ago
they won't auto start like docker
bsherman
bshermanβ€’11mo ago
what do you mean by this?
bketelsen
bketelsenOPβ€’11mo ago
i mean test in a vm and make a clean vm, then snapshot it clean so you can revert and test again otherwise you'll be wasting your days reinstalling your main desktop
j0rge
j0rgeβ€’11mo ago
@bketelsen right, so then I think the key is to automate building that immediately after an upstream release. Like, if you're going to make me distroless you gotta be fast. I mean for the vscode one
bketelsen
bketelsenOPβ€’11mo ago
we can build all of this in a GH action
j0rge
j0rgeβ€’11mo ago
right
bketelsen
bketelsenOPβ€’11mo ago
so if you run it nightly it doesn't cost any more than monthly
j0rge
j0rgeβ€’11mo ago
we're really good on builders now, we'll be fine
bketelsen
bketelsenOPβ€’11mo ago
and I'm already pulling the latest release in the vscode extension so just download and systemd-sysext refresh and you go tthe new bits i gotta roll to get the boy
j0rge
j0rgeβ€’11mo ago
like, also let's not even bother with incus rpm's unless you really want to like if you wanna go all in, I'm right there with you boss.
bketelsen
bketelsenOPβ€’11mo ago
i'm all in
j0rge
j0rgeβ€’11mo ago
oh, and it works out
bketelsen
bketelsenOPβ€’11mo ago
this is THE WAY in my mind yeah, that's a prerequisite
j0rge
j0rgeβ€’11mo ago
dx has it's own dinosaur, you're already on brand
bketelsen
bketelsenOPβ€’11mo ago
it should work
j0rge
j0rgeβ€’11mo ago
No description
j0rge
j0rgeβ€’11mo ago
that's karl's whole schtick oh also @bsherman - also I think having a different dinosaur with systext for a while to see how they feel. We did get plenty of dinosaurs for a reason.
bsherman
bshermanβ€’11mo ago
you talk to the dinos and see how they feel about it πŸ˜‰
j0rge
j0rgeβ€’11mo ago
that way we can keep bluefin as bluefin, and this is more a potentially new dromeosaur that's in a different environment now. and I'm also really liking it from a network perspective, we gotta get these core images down -dx is approaching 2GB a day, we're at 1.8 and if we can make most of that go away it'd be a nice improvement until container diffs
bsherman
bshermanβ€’11mo ago
you are actually touching on another topic here... but one we can start to look at after we get our own repo... if we build an image, but no packages changed... why should that image get published?
j0rge
j0rgeβ€’11mo ago
Would 10000% love that
bsherman
bshermanβ€’11mo ago
right now, we do publish it... and it gets downloaded, and uses that 1.8GB ... but the only real change is there was some metadata differences that mean nothing to me our concept is close to "done"... but making things work the way they should on in CI and streamlining images/downloads, not even close πŸ˜‰
j0rge
j0rgeβ€’11mo ago
yeah I dunno what's going on with these scripts can't most of these be actions?
bsherman
bshermanβ€’11mo ago
actions aren't magic
j0rge
j0rgeβ€’11mo ago
they are if brian is writing them! actually, sec, I think flatcar might have them
M2
M2β€’11mo ago
Why publish daily on this?
bsherman
bshermanβ€’11mo ago
the problem isn't action vs script... it's identifying the problem and finding a solution i derailed us into size of our OCI images, i don't think there is discussion of daily publish of the sysext stuff
bsherman
bshermanβ€’11mo ago
i just want to stop i'm getting committed to too much
j0rge
j0rgeβ€’11mo ago
oh hahaha. don't worry, bk has a brand new kid, when he says 30 minutes that's like 30 days to the normal universe.
bsherman
bshermanβ€’11mo ago
but i will say... it's a LOT easier to build a streamlined flatcar or coreos image than a desktop...
j0rge
j0rgeβ€’11mo ago
this is why I still have my flatcar server, there's just no reason to ever change it, it's just part of that device now. it's like this nice NUC for it's time with good specs and storage, flatcar + containers, everything is autoupdating, etc. It's perfect for what it was built for. and baby sitting my ubuntu server is way worse
bketelsen
bketelsenOPβ€’11mo ago
GitHub
bluefin-dx tracking issue Β· Issue #2 Β· ublue-os/sysext
in order to serve as a replacement for the -dx experience we need the following included in a single sysext: docker gcc & general build toolset lxd incus vscode jetbrains kcli docker-compose di...
bketelsen
bketelsenOPβ€’11mo ago
comment with changes/additions/notes @M2 did I capture your login / gdm thing correctly?
M2
M2β€’11mo ago
Login (from VT) does not pass environment variables sourced from login shell to systemd user session. GDM does this. So if we are placing something to set a path in /etc/profile.d does systemd user session need to know this path or not?
bketelsen
bketelsenOPβ€’11mo ago
good question! Can you comment (even cut and paste from here) on the issue so we track it?
M2
M2β€’11mo ago
Done. As I think about it, I don't expect it to be a problem. I'm just curious for things like incus/LXD/docker unprivileged.
bketelsen
bketelsenOPβ€’11mo ago
just pushed a new config with neovim, vscode & docker enabled. just build-meta -> meta.raw requires a systemctl daemon-reload to start the docker socket/service but that'll resolve in F40. Not bad! friends, this isn't just possible, it's awesome sysext files in /usr/etc don't get copied into /etc - that must happen earlier in the boot process. modifying the finalization script now to create a confext (configuration extension) if there are files in /etc Done. But my metadata is wrong. Confext isn’t loading it. This is exciting. metadata fixed added /usr/bluefin/bin to the path in the confext and /usr/bluefin/share to the XDG_DATA_DIRS so application desktop files are loaded
bsherman
bshermanβ€’11mo ago
🍿 πŸ˜„
bketelsen
bketelsenOPβ€’11mo ago
neovim's .desktop file in /usr/bluefin/share/... now appears!
No description
bketelsen
bketelsenOPβ€’11mo ago
probably need to find a list of other paths - MANPATH for example
bketelsen
bketelsenOPβ€’11mo ago
and our good friend vscode now lives in /usr/bluefin
No description
bsherman
bshermanβ€’11mo ago
πŸ”₯
bketelsen
bketelsenOPβ€’11mo ago
first VM brick of the day ain't mad. that's why we test in vm's πŸ™‚ hmm systemd-confext mounts your configuration extensions over /etc which makes /etc/ read-only until you unmount that'll be a surprise pain point
bsherman
bshermanβ€’11mo ago
to say the least... the confext stuff is not core to some of the stuff that was discussed right? neovim/vscode/et al but it would matter for docker/incus/libvirt right?
bketelsen
bketelsenOPβ€’11mo ago
that's roughly correct for vscode/neovim the confext is adding paths in /etc that could be added by another mechanism
bsherman
bshermanβ€’11mo ago
oh hmm
bketelsen
bketelsenOPβ€’11mo ago
it's adding /etc/profile.d/99-ublue.sh:
export PATH="$PATH:/usr/bluefin/bin"
XDG_DATA_DIRS="/usr/bluefin/share:$XDG_DATA_DIRS"
export XDG_DATA_DIRS
export PATH="$PATH:/usr/bluefin/bin"
XDG_DATA_DIRS="/usr/bluefin/share:$XDG_DATA_DIRS"
export XDG_DATA_DIRS
bsherman
bshermanβ€’11mo ago
right
bketelsen
bketelsenOPβ€’11mo ago
and both XDG and PATH will ignore missing directories so we could add that to the base image to resolve this and/or wrap them in a directory test check
bsherman
bshermanβ€’11mo ago
exactly
bketelsen
bketelsenOPβ€’11mo ago
test -d /usr/bluefin ... so it's not a roadblock, just an inconvenience to note
bsherman
bshermanβ€’11mo ago
but it still seems at least a partial roadblock to docker/libvirtd in sysext, unless i'm missing something... i know you are building with a /usr/bluefin prefix, so that could default to them using /usr/bluefin/etc... but all guides for those tools will talk about configuring /etc/docker and /etc/libvirtd ... so ... is it reasonable to bind mount /etc/bluefin/etc/docker to /etc/docker , for example? or is this a systemd-confext issue?
bketelsen
bketelsenOPβ€’11mo ago
I think the right answer today is "I don't know"
bsherman
bshermanβ€’11mo ago
fair
bketelsen
bketelsenOPβ€’11mo ago
thursday lunch update: I have vscode, neovim, and docker installing in the "meta" sysext. Everything works as expected with one caveat - the confext thing probably wont' work for us because it forces /etc/ into read-only mode. We'll add the /usr/bluefin paths elsewhere. Separately, I used mkosi to package up some fedora packages as a sysext. This doesn't seem to be possible without mkosi, but it's really slick. Create an empty base image, add a layer on top of it with the packages you want to install. mkosi diffs the two layers and ships the diff as a sysext. I used that to install the development-tools meta package. mkosi automatically signs extensions, so I had to change the systemd-sysext command used to add an image-policy that would accept the signed disk image generated by mkosi. Documented this in the as-yet-unpublished mkosi repo I have here. the generated sysexts create appropriate desktop files for neovim and vscode, too.
bketelsen
bketelsenOPβ€’11mo ago
bketelsen
bketelsenOPβ€’11mo ago
and here's a demo of docker, showing the daemon-reload required to enable it
bketelsen
bketelsenOPβ€’11mo ago
bketelsen
bketelsenOPβ€’11mo ago
I added a withmkosi folder to the sysext repo that has the configs used to make the development tools extension. that's a really nice way to add a sysext with things that exist in the fedora repos. remember me bitching about confexts making /etc/ read-only. Apparently my coworkers have been busy:
bketelsen
bketelsenOPβ€’11mo ago
and a PR to implement the spec https://github.com/systemd/systemd/pull/31000
GitHub
WIP: systemd-sysext: Implement optional mutability for extensions b...
WIP: needs docs better commits Publishing early for visibility. The PR implements the minimal parts of the proposed update to the Extension Images specifications - uapi-group/specifications#78 - ...
M2
M2β€’11mo ago
Since confexts are part of a squashfs, if we do some sort of bind mount to place them where people expect. Wouldn't those bind mounts be read-only due to squashfs?
bketelsen
bketelsenOPβ€’11mo ago
they're read-only no matter what file system or format you use
Want results from more Discord servers?
Add your server