Universal Blue sysext with Nix packages
GitHub
Proposal: Changing the way these overlays work to integrate nix int...
Sorry for always talking about nix, but it really could be quite useful for this project. Like, I am working on an architecture for this project on my personal fork that uses the nix store for thes...
426 Replies
This thread was made for indexing this topic
good idea
apparently bind-mounts are going to be the best way to handle this
just like snaps work, lol
im on nixos so this works a bit differently but look at the loopback device right there lol
guys this WORKS
LES GOOOOOOOOOOOOO
IT ACTUALLY WORKS
@bketelsen can you help me out with something?
literally the only thing missing of this puzzle is just the systemd-sysext itself
i really dont know how to add the proper metadata to the system extension
could you send me one of yours or something?
like, the symlinks are working properly and everything!
its just the sysext now :(
you need this file inside your squashfs
GitHub
sysext/sysext.sh at main · ublue-os/sysext
EXPERIMENTAL: ublue systemd-sysext. Contribute to ublue-os/sysext development by creating an account on GitHub.
and it needs to match the name of your squashfs
so mynixthing.raw should have /usr/lib/extension-release.d/mynixthing.sysext as the name
oh.
so thats the issue...
its just the file name....
ill fix it
jesus christ it actually worked
@bketelsen
it really did work!!!!!!!!!!!!!!!
So! The PR is almost done! Now layering stuff is as easy as running:
It just isnt usable yet because the built images arent compatible with systemd-sysext for some reason??? It just complains about it.
You can ignore the warning with the
--force flag on systemd-sysext but stuff like ssh sessions and whatever else apparently just crap themselves out and dont work anymore???
I guess thats pretty much it!
I really dont know how to fix that, tho, although I'm pretty sure there is something to do with file permissions on /usr/share
Made a PR on GH with all the changes!GitHub
feat: nix-based system extensions by tulilirockz · Pull Request #4 ...
This PR implements #3!
Here is a graph explaining what this PR does:
cc: @bketelsen
(theres also a graph on the PR view)
that's insane! I'll play with it tomorrow!
please use it on a VM for now tho!!!
it gonna break your current user session for now
im trying to troubleshoot the permission issues
you can add any packages from nixpkgs on
config.json
ill eventually add flake / flakehub support too
Well! Not anymore!
you can try it out on your own system now, hopefully
it shouldnt break anything
at the worst case you can just reboot
but just make sure SELinux is permissive first (else it will actually break stuff on runtime)for
setup-nix-mount and mount-store-squashfs I had to add sudo to most of the lines to make them work.
@tuliliw initial impression: this is absolutely amazing. just wow
my thoughts:
* it makes for a strange experience having to mount the nix store in addition to the sysext.
discussion about that here - read the whole threadNixOS Discourse
Should the default nix store location be below /usr?
When I’m on Fedora and want to use a sysext containing a nix store, I can’t do that with a nix store under /nix, because systemd-sysext will only extend the /usr and /opt folders. Correct. I wasn’t aware of that. Do you happen to have any pointers to examples on how to use dm-verity with nix? I’m especially interested in having a ephemeral s...
it would feel better from a user's perspective to add the squashfs inside the sysext and add an "enable" script or one-shot systemd unit
i don't know if that's possible
* the binaries output from the flake are placed in /usr/bin, which will mask/overlay existing binaries in the host. It's better to put them in a different directory - I've been using /usr/ublue/bin, which forces the user to add that to their PATH.
I haven't read all the flake code yet, but I'm assuming you're not doing anything about config files (/etc/defaults/docker kind of things) or systemd units... if we move forward with this nix route both of those need a solution.
yup! the biggest problem with the nix approach is this /etc files thing. like with the other approaches, we do need to copy etc files for every package we want to install, and that would be an issue
the nix packages themselves dont contain anything under etc because every etc related thing is generally declared through nixos options
that makes sense
we could do a very janky approach tho: making the user build a nixos configuration on a separate nix file, build the nixos system and extract everything from it's etc and dump it to the systemd-confext layer
thats really the only way i think that could work
the systemd-confext approach is already janky tho. it makes /etc/ read-only which will trip people up a lot, and require them to unmerge the confext to make changes
since we have a bit of freedom in our implementation, we could add an /etc/ tree under a known directory inside the sysext... like /usr/extension-etc.d/hello-layer.sysext/etc/... then put any configs in there and have a host systemd unit that reconciles the /usr/extension-etc with host /etc on a trigger or timer?
that approach takes a little of the magic out of the sysext experience tho
btw - this is great brainstorming. thinking through all the permutations will help this succeed even if the target audience is relatively small
(im not answering rn because im having lunch brb real quick)
(ill reply to everything soon)
and for systemd units it's totally valid to put them in /usr/lib/systemd/system/myunit in the sysext. like this: https://github.com/flatcar/sysext-bakery/blob/main/create_kubernetes_sysext.sh#L46
GitHub
sysext-bakery/create_kubernetes_sysext.sh at main · flatcar/sysext-...
Recipes for baking systemd-sysext images. Contribute to flatcar/sysext-bakery development by creating an account on GitHub.
with systemd v255, systemd will do a daemon-reload if the extension has
EXTENSION_RELOAD_MANAGER=1 in the extension-release.d config file
that should come to us in F40 which isn't far off
if we plan for that target, we can write systemd units that don't rely on /etc/ at all like this one:GitHub
sysext-bakery/create_docker_sysext.sh at main · flatcar/sysext-bakery
Recipes for baking systemd-sysext images. Contribute to flatcar/sysext-bakery development by creating an account on GitHub.
take your time lol
hiii im back
oh this is just awesome, not depending on /etc on those layers would just be amazing
its just that in the end there will need to be a
/nix directory on the root because nix links all their binaries to that, sadly
overlaying stuff manually ontop of it would be best because in the end it would be necessary i think
idk how systemd-sysext works tho, it would be awesome if it could merge multiple nix stores on /usr, then we could just link that to /nix/store and we would be good to go!
both approaches should work nonetheless
its just that maybe there could be incompatibilities with people that already have nix installed
since this is very experimental stuff we most likely should target systems with systemd 255+
then we could worry about stuff with older systemd versions, but for now, whatever, we just need to get this working
in the end the user will need to add at least one folder to their path,we could so something like this:
- each layer has its own /usr/extensions.d/{LAYER}/path that has their binaries there (instead of /usr/bin)
- our utility would symlink everything from those /path folders to /run/extensions.d/path
- the user would add /run/extensions.d/path to their PATH
the only problem would be if there are the same binaries under the same extension, which we could solve by having a priority system
eek
or idk just use the order that they appear on the yaml file?
🤔
last symlink wins 🙂 Let's not build priorities (or dependent extensions) until they're needed
makes sense 👍
in my mind, the joy of using sysexts would be dropping them into a folder and immediately getting a magical new experience. the less tooling we need to write to make that happen the happier I'll be
now we could resolve ALL of the issues we've discussed so far by having a daemon on the host that monitors sysext loading/unloading and takes an action (ignore security implications here for now). for example if /usr/extensions.d/{layer}/ has an
init.sh the daemon could run that after loading the extension.
that init.sh file could create directories, mount things on the host, reload systemd, copy files, etc
but... the security implications are insane. You'd be giving every extension root on your machineyeah LOL
like
i think the best thing would be to have like a cli tool for managing all that
usysext for example idk
then the user would just:
usysext add PACKAGE --layer hello-layerI told @j0rge i wanted to make one called
extrathen it would manage everything by itself
this isn't a bad idea if we can't get what we need out of the operating system tools as they exist now
its just that like
we can do everything through just, most likely
but it would be much more of a hassle for both us and the users to manage everything
there could be both the imperative approach, by using the cli directly, and the declarative, by making a file like that
ok let's riff on this for a bit
for any sysext, whether it's nix or not, we need to:
* manage host paths
* manage systemd units
* create directories (tmpfiles.d kind of stuff)
for host paths, we can add /etc/usysext/paths.d and symlink files into there from the sysext
for unit files we could do something similar at /etc/usysext/systemd/
and again for directories /etc/usysext/tmpfiles.d
and... this might be interesting
the /usr/lib/extension-release.d/extension-release.NAME file is a subset of /etc/os-release which is just key/value pairs
and there is intentionally no limitation on adding your own keys
we could put all of this metadata in that file and just read it from our management app to act on it
three or 4 paragraphs down - here: https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
we add
EXTENSION_PATHS=/usr/something/bin
EXTENSION_UNITS_DIR=/usr/something/systemd/system
EXTENSION_TMPFILES=/usr/something/tmpfiles.d
maybe EXTENSION_INIT=/usr/something/init/pwnyourworld.sh
we can sign extensions too with the same key we use for images
which is guaranteed to work on all ublue systems already
and cosign should be installed
ok here's a random moment of fun
the hello-layer sysext prevents sshd from running
i rebooted and got this
Jan 25 16:15:30 beast sshd[8007]: fatal: chroot("/usr/share/empty.sshd"): Permission denied [preauth]
it broke virsh too@bketelsen make sure your layer is as updated as possible on the latest commit + SELinux is permissive
it used to break everything, ssh, whatever, because it made
/usr/{share,lib,bin,} modes 0555 instead of 0755
the latest commit fixes that, hopefully
and SELinux labels also disappear because of the layer, cant fix that for now because of a bug in the nixpkgs semanage package
i was testing the system extensions and my system just broke LOL
it literally wasnt able to boot anymore
at least it was on a VM so its good
my system just died and cant boot anymore because my pc turned off suddenly
so development on this will be halted for some hours
@bketelsen since im going to sleep rn and thus i cant work on this rn, i made a really simple graph of what i want the CLI to look like. What do you think about this achitecture?
someone would call bext like
bext layer get-propery signing-key
bext add PACKAGE --config="/path/to/config.json" --lib-flake github:ublue-os/sysext
someone could also just fork the main lib flake for making custom layer layouts of their own by modifying the base script
bext add --flake github:laixintao/flameshow PACKAGE or SCRIPT --config="/path/to/config.json"
im showing some "advanced" usages of the hypotetical cli but in the best case scenario the user would write something like:
Another example with --user flags / git cloning:
Honestly im just brainstorming but this CLI could be super helpful
Im going to sleep rn! Good night everyone!
bruh i just cut the build time for these extensions by 20 seconds on my ssd
at least a minute or so on a HDD
the old build script squashed and unsquashed a bunch of folders and idk it was just really confusing
now its super fast i love it
ill commit soon
ill make it so the build script is a derivation itself so it will take even less time than that to work
making it a derivation also makes it possible to use the nix sandbox instead of relying on the host filesystem itself, which can be quite unsafe (e.g.: "rm -rf /")
also allows for caching and everythingdone!
the build script is literally just this now
beautiful stuff!
btw I broke my dev machine so I've stalled a little until I get it bakc online
oh lol ok
im slowly building the go CLI
nice. Imagine a go version of distrobox
hi guys
just an update
im still making the cli
its gonna be done till next week at max its just a big thing
ive been quite busy this week so its hard to work on this but its looking pretty sweet!
the thing is that like
i wonder why would i do all this tho? its just nix with extra steps
i guess the best usability improvement with this would be that distro maintainers could package their own overlays ontop of their images and make dx integrated on the main images
like idk bazzite could ship /usr/extensions/bazzite/dx.sysext.raw
then activating dx would be a matter of
ujust toggle-dx
yeah actually makes sense sorry i didnt think this through im gonna keep on making everythingwait
/usr/extensions?I’m in the car for about an hour then I can look
yup! something like that!
its just a little suggestion so whatever idk what would be the proper directory
I would like to have nix itself be installed as a rpm so it can be updated along the system and not have to worry about accidently deleting nix
that would be awesome
its just that nix can install itself
itll be very weird having both the rpm-managed nix and the nix-managed nix
we would just use the rpm managed nix. I wonder if each user could have its own nix store?
of course! nix already has a solution for that called nix-profile(s)
but you could use bubblewrap to make a
nix-user-chroot-like environment for a nix application
by just bind-mounting $STORE over to /nix/store in your sandbox@dnkmmr
this project basically just helps distro builders to include dx or other kind of extensions straight onto their main images
without having to make any alternative image
how did you make ls look so good?
its the nushell's integrated ls command
you really should try it out
you can use object-based commands like powershell on it
gonna install the nix package
I wonder if we can get gpu acceleration working for nix without the need for wrappers?
sadly i dont think so?
there is a way to bundle nix packages in order to make them run with a specific thing inside of them
like
GitHub
GitHub - NixOS/bundlers
Contribute to NixOS/bundlers development by creating an account on GitHub.
generates binaries that have an entire
/nix/store inside of it with all the dependencies and stuff and the main binary just run nix-user-chroot and thats it!almost finishing the main commands! everything is looking pretty good so far
the only things missing now are:
- refreshing/mounting/unmounting the nix store along with the systemd-sysexts
- service for automounting the nix store on boot (and a way for installing it)
- prettier cli prompts with
go-pretty (stuff like progress bars)
- a better build command that includes metadata and a bunch of stuff straight onto the images
and thats it!wow that is serious progress!
its funny how far ive come with this pr in just 10 days
amazing stuff!
im making the last major command,
build
this will be particularly annoying to make
So guys!
All the layer-manipulation-related commands are done!
The only thing missing to actually finish all this is just the nix store mounting and prettifying stuff
Even the infamous build command is pretty much done
Ill just have to remember to change the default flake to ublue-os/sysext instead of my fork whenever this is doneis it all in the PR branch? I can check it out tomorrow morning...
yup!
i just made the mounting command like right now
now ill test out literally everything on a vm to see if everything is working
logging + prettifying things in general is missing tho
ill also make a nix package so that its easier to test
lookin' pretty!
also, took exactly 1m300ms to build the entire thing, pretty cool! with the build cache that will take a lot less whenever i implement that
oh that's fast, nice
well.
IT. IS. DONE
i literally JUST finished fixing some stuff on the mounting commands
the CLI is functionally done and anyone can just use it now
its in an alpha state because there may be many bugs still
but it is usable
(still advised to use it on a VM with SELinux disabled!)
What's the benefit of this over straight Nix?
Is it just that the binaries and libraries are in
/usr/{bin,lib} rather than /nix/var/nix/profiles/*/{bin,lib}?this doesn't appear to require nix on the host. just a bind-mount for the nix store
@tulip can you make a demo video? no audio is necessary, but I'd like to see how you use it.
this is some ninja level shit
mostly not requiring nix on the host, and a similar distribution model to what we do with OCI images, like, we can just distribute any system extension and everyone will have a consistent environment. basically we could replace the -dx images with these if this project actually succeeds
alright! i just cant do it right now because im a bit busy, but ill make a little video showing it off
but seeing that you had trouble figuring out how to use it indicates that something is wrong with the UX
no i haven't tried yet. I was reading the code in the PR. mostly I don't see an example config file, that'd be useful
oh! the example config file is in the code, its just a struct that gets marshalled and written to a file
here it is: on the "defaultConfiguration" variable
plus it is also Atomic™️ since any layers wont be applied without them actually working first (thats something we can also add to our README/branding)
also ive been quite busy this week so i havent worked on this
but ill be working on this tomorrow and friday
just a heads up because i really wanna get this done
its such a cool project
lol
I was watching some stuff from last year's nixcon
And I just found out that "Layered Nix Store(s)" exist?
The example that the guy made on his presentation was like to provide a container with a 16TB cloud-storage lowerdir for its
/nix/store path so that it will think it has all that storage and packages but actually just has to "download" (more like import?) them
Maybe that could be an useful idea for the bext build build cache, like, having a flag named --save-run-to-cache which would merge the default container store with a cached store on the host
This is all just because building stuff takes a long time, not even because it is like particularly badly optimized, its just a lot of stuff!
Like, fetching the entire Docker / Containerd / Moby runtime, Podman, Incus, Qemu, whatever else all the time you build an image is a very expensive and slow thing
Maybe that could be an awesome thing to add
Im currently just cleaning up some bugs, and doing the "AddToPath" command so that people can just add the /tmp/extensions.d/bin dir to the target shell's rcs@tulip in here!
ok how do I run something?
i sent some instructions on #💾ublue-dev
but like
if you wanna create your extension from scratch it would be something like:
(once you get the binary)
@j0rge
now i just need to add logging and create a package so that this wont be annoying to compile from scratch
ok so this creates a config.json
and then I edit it in there and add whatever nix package I need
yup!
then I just add that.
yep!
ok where do I get bext from?
i can just send you a static binary now?
you could build it with go on a distrobox
but ill just send you a binary if possible
attach it here it should be small
just like please dont run this on your host for now because it may break stuff and i dont wanna break your pc
im building it
remember: deactivate SELinux permanently when running this for now because it is gonna break something
(run on a VM ideally)
/etc/selinux/config -> SELINUX=permissive
@j0rge
@j0rge just found a super dumb bug: the progress bar gets stuck on 80% but the build finishes
so just know that thats a thing
ack
ok so gonna vm this
just for now tho, it is a super dumb upstream bug
ill try to manually patch it whenever i finish up this PR
but still, annoying
also proprietary stuff is not enabled yet but fixing that is a matter of just modifying a single line
nod
:( really no way this -> static <- binary jsut didnt work
lol computers
try running from source then it should work on a distrobox
git clone https://github.com/tulilirockz/sysext bext
sudo dnf install -y device-mapper-devel gpgme
go build -o bextWe need a self updating nix static binary
@dnkmmr can i ask you something this might help out with the whole ".desktop" issue we were having with this thing
i remember you made some nix install scripts
.desktop issue?
and they added their
.desktop files to XDG_DATA_DIRS
yeah .desktop files arent on any sysext for now because i havent figured out how to add them to those
do you know where does nix puts their desktop files to?
i remember it was something like /nix/var/nix/profiles/*/share
but i literally dont have any profile there
the big problem that I have for nix is that nix itself installs itself on the default profile (basically installing flatpak as a flatpak)
yup that is correct
getting some ioctl btrfs compilation errors, but I'll leave them vm around for a while and dig in when I get more time
thats super ultra weird
its it something with the build?
i havent ever had any issues with that
it's a dep
can you send me the errors if you can?
OH
i know what it is
its just that your thing is missing
btrfs-progschecking
i forgot to add that to the dep list
lol sorry
gpgme-devel too
really? i thought it was just gpgme
its that the
build command uses some podman bindings, and they require some C stuffI needed devel to pass that oen part
well
gonna add that to the README
even with
btrfs-progs?yeah
thats so weird jesus christ
maybe
btrfs-progs-devel?that's it!
yo!!!!!!!!!!
where's it put the binary?
it builds the binary as the directory name
there should be either a
sysext or a bext binary on $PWD
build the thing inside a distrobox then exit the distrobox because it requires host access (when building/deploying sysexts), toooh ok
looks like it built but can't find the binary
i dont even know what to say like what is happening LOL that never even went under my radar
are you running
go build -o bext?yep
are those errors or output?
oh you cloned the repo in the few seconds between me removing something and fixing this
git pull the newest thing now sorry
go gets super mad at you not using imported libraries, so yeah, errorsup to date
jesus christ what is happening
yknow what
to stop wasting your time
maybe it would be better that i just make the RPM package first
and you layer it onto your vm
that might be better for now
sorry jorge really
lol it's ok we're always trying new stuff
im going to make a prototype today, test it out, and send you the built rpm with the proper things and everything
yeah no worries, don't sweat it!
I learned the horrors of the golang pacckages in fedora
LOL yeah
thats such a mess
did someone seriously do all of this?
on purpose?
these errors?
no these packages lol
oooooooooh
oh god theres so many things
the names too lol
is there some sort of error?
no that's just a list of packages
looked like a log of an error because of the red text
buiding some packages with melange!
you know how we wanted an example and we picked "emacs"?
maybe the play should be bluefin-cli as a sysext
that
would be an amazing idea?
especially considering theyre all just cli applications
oh god this is the best
hell yeah lets do this!
we can even like add hooks for the sysexts if necessary
(e.g.: run topgrade on install, generate X file, do Y thing [...])
It worked out!
The only thing missing is just a RPM package now
cc @EyeCantCU
yeah so I'm thinking, just bolt it onto an existing system = everything you'd ever need decoupled from the host os but with access to everything
then for us it becomes "we give you this CLI experience, use it as a container or as a sysext, whatever works for you."
this is just amazing not going to lie
I need to step out tonight but post pics!
okok!
making the RPM thing
ill also make the demo video with some bluefin-cli packages
because I think what we also do is make an action generate the raw file regularly
but not sure how updates happen to extensions, is there like a service that handles that?
i mean
the extensions are quite literally just squashfs files
we coould rebuild them locally and that wouldnt be so much of an issue
the config.json gets copied to the packed system extension, so its just a matter of running
bext build /usr/extensions.d/$EXTENSION/metadata.json -o /tmp/bext-cache-$TIME/ then adding the result to the cache and activating the image againI was thinking an action would just build one on the spot right in the repo
ye! could also work
if the user really wants to rebuild it locally thats an option too
shouldnt be an issue
indeed, and useful for folks too
so they can modify without needing github
yup!
It's beautiful
I’m so down to see this
the RPM thing is almost done
im still trying to compile this
whenever i get this done ill test it out on a VM and thats it!
@tulip when you're around can you walk me through testing
bext?oh yeah!
im around right now! lol
let's do it!
do you want to go on a VC or through text here?
text please, I'm multitasking while I do internal required training at work 🙂
oh lol okok no prob
I have your branch.
that's as far as I went
you can either build the packages or just the binary itself directly
i havent been able to install the rpm package properly through rpm-ostree due to some selinux issues for some reason
so i guess the binary would be better rn
ok so I'll drop into a container with go and build
it
make a fedora distrobox with these packages:
device-mapper-devel gpgme-devel btrfs-progs-devel (and also go)
then you should be able to compile it fine
oh there is something very funky going on with the build progress bars but it should build
well
did it work?
ok I added a
build-devbox.ini with those packages while I was at it. Built it go build -o bext
that's donegood idea! i made a devbox just for building the rpm but makes sense indeed!
okok
so
now you can initialize a new image with
bext layer init
it should make a config.json file in $PWD by defaultok to do that in the container?
actually i have no idea ive always ran that on the host so maybe that wont work
should I copy bext over to my test vm?
yep!
i'll do that lol
its just that bext connects to /run/user/X/podman/podman.sock
so if your container has access to podman it should work
but i guess it wont
the build should be completely safe to run on any system
ok I have a config.json
its just the
bext mount extensions command that isnt
okok
you can add whatever you want there
(also do you think it would be better to have yaml support?)
then run bext layer build /path/to/config.jsonjson doesn't have indenting issues. my vote would be to worry about yaml later
should create a file named
$LAYER_NAME.sysext.raw in $PWD also
okokoh I have to enable the podman socket
do you remember how to do that?
ummmm
its
systemctl enable --now --user podman.socket
but try running it without the socket being enabled
i havent tested that out
i wonder what will happengot it
it fails without the socket
okok
should also work with the system-wide socket
ok now i have example.sysext.raw
(and that was faaaast!)
wow that built so fast for you
yeah jesus christ how
can you stat the file?
stat -h example.sysext.raw-r--r--r-- 1 bjk bjk 98394112 Feb 16 10:20 example.sysext.raw
maybe it literally didnt work?
oh
it is big enough that maybe it actually worked
what is in your config.json?
just neovim
cool!
so it should have actually worked! cool!
so
you can check out what are the sysext's contents with
sudo mount -t squashfs -o loop example.sysext.raw $EMPTY_PATHok just to prove it isn't there:
okok
so
empty_path is just a folder that you may create
lol
mkdir testbahaha
the nvim symlink should be under
test/usr/extensions.d/example/bin/nvim
(i hope it worked, fingers crossed!)ok there's a bunch of symlinks to nix store in there (that show broken because there's no nix store mounted)
yeye
is there a nvim in there?
yup
great!!!! so it worked!
nvim, nvim-python3 nvim-ruby
so far so good!
ok now we are gonna add the layer to the cache and activate it
(just unmount the test layer thing first just because maybe that might break something)
sudo umount ./testjust did that
then:
okok
bext layer add example.sysext.rawyeye so its right
you can change the cache path but ideally that should be system-wide
so run that but with sudo
sudo !!did it - had to give full path to bext because it's not in root path... done
should create
/var/cache/extensions/(? idk)/example/BLOB
okok
now you just need to bext layer activate examplegreat!
as sudo?
so its working great so far
yeah
it will symlink the
current_blob to /var/lib/extensions/done
then the last part is really just to mount everything up
ls -la /var/cache/extensions/blobs/example/
total 96092
drwxr-xr-x 1 root root 152 Feb 16 10:27 .
drwxr-xr-x 1 root root 14 Feb 16 10:27 ..
lrwxrwxrwx 1 root root 100 Feb 16 10:27 current_blob -> /var/cache/extensions/blobs/example/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
-rw-r--r-- 1 root root 98394112 Feb 16 10:27 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
~/bexttest
bext mount layer --refresh
bext mount store --refresh
bext mount path --refresh
bext add-to-path
thats it i think
in this order in particular btw because the store cant be mounted without having the layers mounted firstbext mount layer doesn't exist
oh
bext mount extensionsbext mount [extensions, path, store]
yeye
weird?
oops
i did it out of order
oh no i didn't
well lets see if it works
hmmmmmmm
but I don't have nix on this host
try running this:
it should just create the store folder
ok done
and the mount store worked
ooooooooooh i know what happened
i called
os.Mkdir instead of os.MkdirAll on the mounting process
well
so it worked?
did the bext mount path also work?mount path didn't work
what did it say?
maybe another mkdirall issue?
hmmmmmmmmm
i wonder what happened
yeah I don't have /tmp/extensions.d
can you try debug logging?
bext mount path --loglevel debug
oh yeah it should be that
maybe a mkdirall issueok mount path worked
great!
now
bext add-to-path bash
(i should make that automatic without needing the user to actually type the shell name but still)great!!!!
now refresh your bash shell and it shouuuuld be fine!
brb
it did work btw
neovim worked?
y
also im very surprised by how incredibly fast your layer built
my computer takes at least a minute building any layer
can you also check if
/usr/share/applications/neovim* exists?yes it's symlinked in!
YO
NO WAY
AHAHHA LES GO
i really thought these didnt work!!!!!!!!!!
thats great!!!!!!!!!!!!!!!
well, now we can just use that to install steam
LOL
anything really
WOW really this is even bigger than i thought
sooooooooooooooooooooooo
thats it!!!!!!!!!
one word
the only thing that i havent done yet is the automount service so that
/nix/store gets automatically mounted on bootWOW
yea!!!!!!!!!!!!!!
this is so amazing
this is insanely amazing
yeah!!!!!!!!!!!!!!!!
@j0rge check this out!!!!
he's on a plane to Mexico I think
lol okok hes eventually going to see this
this is so cool oh god
so my notes:
1. there are some mkdir issues
2. a few commands are missing args from the help text
yeah im literally fixing right now all these issues we got
(really?)
the command needed an arg, but its' not in the help string
OH yeah that is something that i didnt consider
layer activate
HAHAH hey jorge!
I'm listening to ghost celebrating!!
change it to
layer activate [layername] or whateverokok
add a minArgs = 1 to the command definition
God Bless America
this is sooooooooo good
okok!!!
that is such a sweet thing like oh god this might change some stuff around
well
is there something else?
i was going to ask if it survives reboots
but I see the mount unit in the source
there is the unit butttt i dont know if it works?
shouuuld work fine but still
Is this neovim?
yeah
my go-to first test since it's pretty self contained
indeed
initial thoughts:
1. Seriously, amazing job on this. WOW
2. we should make a repo with prebuilt configs and a GH action that builds & signs the raw files
3. add a
bext get command that pulls the raw files, checks the signature
4. a ton of documentation is needed
5. WOWhow would we sign them, though? would it be a good idea to add the signing thing straight into the image?
LOL yeah
cosign? I think we already have all the plumbing needed in the bluefin org
and bluefin probably already has the pubkey installed
oooooooooooh
Yep
Should sign like any other artifact
yeah!!!! so it shouldnt be much harder than signing a container
yeah!!!
flipping wow
this is wonderful im happy that this actually works as it does
distributed nix lol
nixless nix
lol yeah!
it's like serverless and distroless
also like thers something that i really want to do
its just that like all builds require every package to be fetched at every run
there isnt any cache implemented yet for this
so add a local cache?
i dont exactly know how to do that
like
ive tested two approaches
either 1: make a tarball of the old container and paste it into the new container
determinate systems has a magic nix cache thing
(doesnt work because of impurities with /bin/sh)
yo what im going to check this out
I can drag them in, they're awesome and would probably help us build the gh action
🙂
yo wow no way???? like i didnt know that that existed
i was thinking about mounting a
/nix store and using chroot-nix-stores to try to implement a build cache
but cool!!!!!!!!!!!!!!
also yeah documentation is pretty much required for a project like this
havent written anything because the approaches that i was testing were so volatile that after like 24 hours the documentation would be worthless
well, working on fixing all the stuff we talked about!
there are also some places where concurrency would help out a lot
nailed it
hahah lets go!!!!!!!!!!!!!!
i told him we were going to fix the last few bugs and get documentation together, then they'll point us in the right direction to get the nix cache stuff working
alright!!!
also
you know that ummmmm
that
nix bundle thing that we were talking about some time ago?
this is very experimental but maybe we could wrap all those symlinks to scripts that run bwrap with a nix store on a non-root pathyeah. I didn't end up loving that very much
should be leagues faster than
nix-bundle but still work just as wellv2
basically would be something like
nix-user-chroot or nix-portablelet's ship this beautiful beast first
ye!!!
i'm so amazed
you really nailed this, great work
but all the things that this does should be also able to be run as a non-root user
like entirely, up to the overlays and everything
https://github.com/containers/fuse-overlayfs and https://bindfs.org/ should make that possible
thank you!!!
i'm in shock at how much potential this has
also!! you can check out the layer metadata by running
bext get-property layer_name, too!
yeah???? like oh god
this is amazing
it should also work everywhere that systemd-sysext works
actuallyOMG
it's beautiful
we can also implement systemd-less suport by just layering the layers ourselves
lol
so the only limit for this program's portability are just where we get the binaries from
should work with anything, like freebsd binaries for example
we just really need overlayfs or something similar to that in order for this program to work
LOL thats so cool
GitHub
GitHub - ublue-os/bext: Build and manage systemd-sysexts
Build and manage systemd-sysexts. Contribute to ublue-os/bext development by creating an account on GitHub.
created
bext repo
so there won't be confusionokok makes sense
sooo can we just upload stuff to that instead?
technically the program is already functional, it just has a bunch of (relatively minor) bugs
yes let's PR against the new repo instead of the sysext one, since the history is so unrelated and the name is confusing
ok!!!
gonna send a PR to that one
@j0rge the final shape
and when f40 lands (and on other distros, systemd 255 lands) we can build in support for systemd services
so we can layer things like mysql or redis, or whatever
we need a systemd unit to create
/nix/store - which already exists in the ublue tree somewhere
and that needs to run before the bext-mount servicebext-mount should also create the nix store by itself so its ok
and we should have nix be installed as an rpm
we don't need nix for this
!!!
lol maybe someone will do that eventually but this doesnt require nix indeed
then how will it mount nix packages?
magic
read the backscroll here
it's all explained
tldr: mounts /nix/store from mounted systemd-sysext files (/usr/store directory in them)
/usr/store ?
bruh @bketelsen with the bwrap approach we literally can do the dreaded /usr merge for nix! lol
literally just thought about that
baby steps @tulip
ship it first
yep!
im just thinking about stuff that we could do in the future
could be amazing
theres so much stuff we can do
it's already amazing
ye!!!
yeah! we build a nix derivation with all the packages specified in a
config.json file, then copy the resulting /nix/store paths onto a squashfs image and mount that to the hostso we make a new repo for sysext recipes, then we can copy flatcar's gh action for releasing them: https://github.com/flatcar/sysext-bakery/blob/main/.github/workflows/release.yaml
GitHub
sysext-bakery/.github/workflows/release.yaml at main · flatcar/syse...
Recipes for baking systemd-sysext images. Contribute to flatcar/sysext-bakery development by creating an account on GitHub.
@bketelsen can you help me out with this?
idk how to pr to the new repo because there isnt anything in commom between them??
weird
git remote rename origin old-origin
git remote add origin https://github.com/ublue-os/bext
GitHub
GitHub - ublue-os/bext: Build and manage systemd-sysexts
Build and manage systemd-sysexts. Contribute to ublue-os/bext development by creating an account on GitHub.
git push -u origin $yourbranchname
i think that will work
it won't be a PR it'll be a new branch
if it doesn't work I'll just copy the files from the exiting repo to the new one and you can pr any local changes
oh do i have perms for that repo?
i thought i didnt
idk
gonna try lemme see
it won't work I don't think
yea i dont have perms lol
let me add the current stuff and you can check it out then pr
okok
when @j0rge is back we can get some help fixing up your permissions. I dont have that level of mojo
sorry
yeah dont worry lol its fine
it's going to make the committer me which isn't ideal since you wrote all this 😦
i'll do the coauthor
honestly i wouldnt mind but i guess that wouldnt be so cool
is that ok?
ok!!
yeah dw
what do you want
NAME <email> to be?usually i use:
Ping an approver
changed the module name to github.com/ublue-os/bext
changed all the code to use the new module name
alright!
also we should change the flake that the
build command points to
that, to
github:ublue-os/bextdone
great!
also this is so extensible jesus christ people can just have other types of build processes
just gotta merge it and we're done moving!
great stuff :)
can you thumbs up comment on the PR in lieu of a PR approval?
so we don't look like a bunch of cowboys
oh yeah LOL ok
ok merged
now you can PR anything
sweet!
@bketelsen i have a bunch of ideas and stuff that we could do
should I make separate issues for everything or should i include them all on a single issue?
separate them if they arent' related, I think
theyre all not related
ok ok
oops
Having too many is better than one monolith
alright!
woops! hahah that url looks nice
it was a good $13 investment
i got the "rockz" part from my github username because of the github org for sodalite (
sodaliterocks)
so we're going to host the docs on that domain?before go had built-in package management I prototyped one called
go rocksthats sweet! i thought that it was just an idea or something, but you actually bought it LOL
yup 🙂
really? cool! i never heard of it
i never released it. It became
athens with Aaron Schelshinger's helpoh! looks like a pretty cool project ngl
also like ive never really had much experience with go before this project so maybe there are some things in the code that are not particularly ideomatic
most likely going to need some cleanup
my first glance - it looked great
You picked the right person to learn from
yeah hahaha ive seen some of brian's presentations
honestly im flattered to see you guys happy about this project
really hoping this will help out
you should be really proud of this, you went from "hand-wavy idea" to great implementation in very little time.
hhhahha thank you so much like really you gave me the idea honestly but im indeed very proud of this!!!
😈
@tulip caution! github.com/mitchelh/osext has been removed
i did a go mod tidy and now I can't build
because containerd indirectly depends on it
honestly i dont know what happened to the dependencies
i was trying out some stuff (like libpod, docker bindings...) and the go.mod got super bloated
even though the podman bindings shouldnt take so many dependencies
maybe it would be better to delete the current go.mod and re-add the dependencies one by one
podman pulls all those deps
the go tool removes unused ones when you
go mod tidywow thats weird
podman is a pig
yeah lol
GitHub
feat: goreleaser config by bketelsen · Pull Request #12 · ublue-os/...
add configuration for goreleaser
for now, don't
go mod tidy or update any deps
can you thumbs up this?yup!!
first step at CI
GitHub
fix: Update go.yml · ublue-os/bext@7b9a06e
Build and manage systemd-sysexts. Contribute to ublue-os/bext development by creating an account on GitHub.
writing a warning for the curious folks
yoooooo the build worked!
now with a status badge 🙂
added all my ideas as issues :)
im adding very strict linters and formatters
should i even do that tho?
that might make building code very annoying
but ill make the PR anyways
its just that its so annoying to see end users getting bugs... i hope these will ensure some runtime safety
loosening is always way easier than tightening up later, it's a logical choice
yep!
Yes!
just finished my final pr for today
could someone approve them? would be awesome
GitHub
Pull requests · ublue-os/bext
Build and manage systemd-sysexts. Contribute to ublue-os/bext development by creating an account on GitHub.
whenever all the currently open issues are done i personally will consider this project funcionally done
LOL
@j0rge sorry for asking but can i have approving perms for ublue-os/bext? its just that there are some PRs that would be awesome to add like the linting stuff and whatever else that are quite weird to get reviewed without having the "write -> commit -> test CI" loop
maybe i can try to do those things locally if you dont wanna do that, no problem, really
I'm afk but will handle it asap
@tulip for the meantime I've approved the workflows. I can give the PRs a look over here in a bit. Apologies for my inactivity. Valve to our dishwasher broke and ruined the kitchen (we opened a help request over a month ago for issues with the dishwasher and the complex never looked into it after we tried getting a hold of them multiple times so... yeah)
hii guys
just a update about this project
im not working on it right now because this week im super busy with lots of college things and whatever else
ill be messing around with this a lot more next week
it is not abandoned or anything
im aiming to chose at least most of the current issues to consider this project in a good state
we can also make the docs website by cloning stuff from getfleek.dev and changing the branding
Blincus.dev is another one I did that we could steal