139 Replies
Alright, starting to look into how we can do initramfs better
this interests you @j0rge since you get your rainbow dinosaur for free
Basically, we need a way to grab the initramfs.img file and modify it
as an action or as part of the containerfile
Looking into a few ways to do this, one concept I have is to mount the .img file and modify it directly
are we thinking this would be easier to do as part of the ISO build process?
or in the container itself?
container itself
this saves the 20+ minutes on bazzite
and will help nvidia users if we can load the nvidia driver into initramfs
without the 20 minute wait
also useful for branding (since we can load our watermark in without rebuilding initramfs)
and useful for encryption (Since we can add the tpm/other modules)
would that get rid of the need for using mokutil to load the key?
that would not, that's motherboard -> loaded image
outside of initramfs
but it would mean functional encryption with just initramfs-etc
and not the full 20+ minute initramfs feature of ostree
lsinitrd is handy for this, it's able to list everything in the initramfs.img file
uses cpio to do it, so it shouldn't be too hard to copy from that, extract with cpio, and recreate when done making changesyou think unpack, inject, repack would be easier than running dracut as part of the build process?
that already happens in bazzite when we install the kernel, but it doesn't pick up our changes
worth trying a manual run
Where do you all do the kernel stuff right now?
near top of our containerfile, installing a new kernel does initramfs
https://github.com/ublue-os/bazzite/actions/runs/8641069139/job/23690555718#step:11:683
it fails to write the image to /boot
so it does, haven't bothered w/ it due to us always rebuilding
so if we can fix that to write to the proper location and manual-execute
that may be enough
https://github.com/ublue-os/bazzite/actions/runs/8641069139/job/23690555718#step:11:758
Seems to have a write location in /tmp while building
that tmp location is where it's doing the copy for microcode in as well.
I'm unsure why boot isn't writeable
that initramfs file should live in /usr/lib/modules
so we could copy it there
also complaining about /root (might need to do a bind mount of /var/root to /root. But I think that is just build log of dracut
yepp you got it. We need to force the location to /lib/modules/$kver/initramfs.img
sha256sum matches on boot and that
found inside of ostree deploy as wel
looks like it would need a manual run since when called as dracut, it will immediately drop privileges to prevent overwriting
yeah that's fine, we'd have to manual run anyway for nvidia builds
and it looks like adding the modules we need is trivial now that we use dracut
nice, this is a W
Looks like we just use a specified /tmp/dracut for build location. If we manually call as root it will still autocopy to the correct to kernel version as well!
so here is my quick idea:
could probably even toss this in libexec in main
so everything that builds from it has it available to use
well we won't have AKMODS variable....
ah true, in that case as a file in main that can be grabbed
I do that for nvidia drivers by pulling the script from HWE
or also in HWE
only reason I wouldn't want to put this into libexec is that clients should be using rpm-ostree initramfs at that point or initramfs-etc
since we are going around ostree here
maybe?
would definitely be curious on what happens in a VM
Uhmm rpm-ostree wrapped is not default..... So guess that needs to be added to bluefin
Oh it's that cliwrap
rpm-ostree cliwrap install-to-root / is needed... which then gives us real dracut
while bluefin has real dracut already@Kyle Gospo would we want to figure this out before spinning new ISOs for Bazzite?
or is flatpak support enough to justify new ISOs?
would be awfully nice to have this in to make the install experience that much better.
GitHub
kernel-signer/sign-kernel.sh at main · EyeCantCU/kernel-signer
Contribute to EyeCantCU/kernel-signer development by creating an account on GitHub.
this has a good generic way to get the current kernel
and then I think we just make it so --add can have additional params passed into it, and toss this into HWE
then anyone can grab it & execute it
bazzite would need to run it 2x, once in the base build and then again at the end of the nvidia build
for us, I think we should use /usr/lib/dracut.conf.d files or /etc/dracut.conf.d since we control the image.
that's fair
Okay, once I'm done with my lunch walk here I'm going to go ahead and give this a try
Did you spend any more time on this after this convo @M2 ?
Just want to make sure we're not wasting effort doing the same thing
nope doing work work
just made the script and saw that the sha256sums matched in /tmp/dracut and /lib/modules/$kver/initramfs.img
and then on my machine that matched what I had in /boot/
Awesome, I'll play with it then
Ty!
@M2 @j0rge Bazzite testing built w/ that script
🙂
initramfs generation was successful
just need to verify it made it to the right place
That also means you can have your branding now without the headache @j0rge
yeah, raven bird ftw
are we sure we wanna do this for F40?
Please don't say "but we have an extra week"
we can add at any time, it's just part of the image
no rush
k
I'm doing it tho, the old shit took way too long
and makes a lot of tickets
I agree
people think it's broken
yeah I'm down, just saying, you know what work needs to be done, don't crush yourself
For the people who don't know what initramfs is, what will this allow us to do?
add modules loaded at boot time (nvidia drivers, tpm, etc)
brand plymouth
Are you gonna test with an ISO? I can help @Kyle Gospo
Oh cool! So no just script to setup Nvidia?
yep
I wanna see how much faster it really is >:)
You basically won't need to use a locally built initramfs.
Enable support for tpm/fido so only need to do cryptenroll.
Nvidia can do early KMS and not fallback to text mode on boot.
And no wait for initramfs setup on boot.
@M2 dracut: Creating initramfs image file '/tmp/dracut' done
Did you say we didn't need to move that file?
Or we did?
I didn't move it. And that file and /usr/lib/modules/$kver/initramfs.img had same sha256sum
Sweet
You can do a sanity check with lsinitrd in a container to make sure dracutmodules are there and the Nvidia libs
Logos worked, no go on the modules though
nvm might have found what I did wrong there
since i havent used initramfs-etc, will that still require some kind of rebuild whenever a dracut.conf file is added to
/etc/dracut.conf.d/? asking since both vfio and kvmfr will be adding configs 🙂yes, same as before only much, much faster
also existing installs aren't getting touched
so if you use initramfs today, you'll keep doing so unless you explicitly turn it off yourself
and we can easily add new modules to initramfs in the future
ok so we just need to adjust bazzite-hardware-setup then so touching rebuild will account for it
maybe write a migration guide to move people over to initramfs-etc too (or an ujust if possible)
Somethings seem to work fine with initramfs-etc. Modprobe works. Install items inside of /etc worked. But dracutmodules misbehaved.
For dracut modules there is a force add as well.
well that is problematic as i know i need dracut modules working for my stuff 😅
Vfio/kvmfr are both modprobes.
Which dracutmodules do you still need?
ah nvm then i mixed up dracut modules and modprobes 😅
@Kyle Gospo you have the conf file in the wrong spot. Maybe it's corrected. But it should be in
/usr/lib/dracut/dracut.conf.d/ Additionally, we don't have dracut-clevis installed so clevis module will fail the build.
finally, we can specify a different compression scheme as well. By default it is using pigz for some reason
I see that you got the location fixed. Just removed clevis. Will probably consider including that on bluefin if it isn't alreadyIs it worth installing it or should we not bother?
I will look into zstd
I don't think we should bother. We don't have the networking modules enabled either so tang decryption wouldn't work.
Fido2/TPM2 would be hardware enablement. pkcs11 and pcscd could also work for smartcard decryption (but seriously, who is doing that?)
For compression: it means the initramfs needs to be decompressed by the kernel. Arch defaults to zstd, Debian defaults to xz. The atomic images defaults to 1 GB in /boot. With 3 deployments, I'm only using 158 MB so, defaults might be fine. Personally just find it weird that it defaults to pigz.
To improve boot times, we could look at not compressing the initramfs
Uncompressed the thing is 169 MB. Nvidia would be an issue
zstd is 127 MB
Default is about 130 MB. Zstd compresses faster, Uncompressed is 39 MB more but should be quicker to boot
YubiKeys can do pkcs11
we have a winner!
Want to add pkcs11 and pcscd as well for YubiKey smartcard functionality?
please
Done
beautiful
👀🍿
This can be done in the HWE images
so Nvidia gets it's modules loaded during boot
That would mean having cliwrap for all of our images
But honestly would resolve the majority case for when people need to do initramfs
@Kyle Gospo have you tried these or just building?
pic is me actively running bazzite:testing on my desktop
look ma, no more
rpm-ostree initramfs
yepp, we'll need cliwrap. The dracut that is on bluefin isn't copying over to the right place compared to bazzite
so this is an added step in the build_image job after building the image but before pushing? or... a new final step in the Containerfile?
i haven't read any commits about this yet, just asking for clarity
One additional script you run in your container file at any point
Was having a look at this. How does the built file go from
/tmp/dracut to the live system? Tmp is cleared out, so this is confusing me a bit
https://github.com/ublue-os/bazzite/blob/main/scripts%2Finitramfs.sh#L11GitHub
bazzite/scripts/initramfs.sh at main · ublue-os/bazzite
Bazzite is a custom image built upon Fedora Atomic Desktops that brings the best of Linux gaming to all of your devices - including your favorite handheld. - ublue-os/bazzite
I need to look at the code generating this, but what appears to be happening is it tries to put it into a folder that doesn't exist at container build time, so tossing it into temp means it's successful, and then it also places it into the system folders where ostree actually makes use of it
Granted this could change, but either way we have what we need here
We also would need to enable CLI wrap for the other images.
It is definitely interesting that this works though.
Without clip wrap, I didn't see the initramfs getting copied correctly.
@M2 This may be safer
QUALIFIED_KERNEL="$(rpm -qa | grep -P 'kernel-(|'"$KERNEL_SUFFIX"'-)(\d+\.\d+\.\d+)' | sed -E 's/kernel-(|'"$KERNEL_SUFFIX"'-)//')"
/usr/libexec/rpm-ostree/wrapped/dracut --no-hostonly --kver "$QUALIFIED_KERNEL" --reproducible -v --add ostree -f "/lib/modules/$QUALIFIED_KERNEL/initramfs.img"
assuming it buildsif that seems to work, maybe we don't have to enable cliwrap and can use dracut directly on everything else.
didn't complain about it 🙂
If you run this once (e.g. in an upstream image), are you then able to drop in the configuration in dracut.conf.d in downstream images, or would you need to rerun the dracut in every image that touches the config?
re-run in anything that touches a config
but with the exception of Nvidia HWE all that really could use this are downstream to begin with
IE: Bazzite & Bluefin
Nvidia HWE is just nice to have because you want your Nvidia drivers at boot time
@Kyle Gospo Are we thinking that these changes (and ISO changes) will be merged into main by middle of week next week? I'm just trying to plan my presentation next weekend.
hope so
GitHub
feat: Add initramfs generation by KyleGospo · Pull Request #1127 · ...
TODO: Copy files from system_files/nvidia/shared/* into the image only for Nvidia builds
If this was in config we would enable decryption methods and Nvidia.
Bluefin/Aurora/Bazzite would then be rebuilding for branding the initramfs
@j0rge ^
what's the tldr on this?
I don't know much about initramfs
Can you send me that dark dinosaur you wanted @ boot?
tldr is now nvidia can load drivers at boot, and we can support unlock methods like TPM & Yubi Keys
sec
and you can have your branding without the downsides
no 20 minute wait on first boot
is the only purpose branding? what else does it do?
whenever something needs to be modprobed at boot it doesnt take a full rebuild of initramfs to enable it
k
is it supposed to still be draft?
yeah, there's a todo in there for nvidia
oh I see it
since those images need extra files added to them so they try to load the driver
priorities!
don't make it too big, smaller is better, subtle.
it's quite tiny
baby murder chicken
Bluefin
That's looking good. We can probably just put a cleanup of Nvidia stuff inside the initramfs script.
Just delete the files then run
Sounds great
Oh I see, it builds it as part of the image instead on boot
Yep, no more 20 minute wait
Now we get our branding and novideo users get a better experience for free
do you have this in bazzite already?
and enables us to use initramfs-etc which is way faster when we need to tack on other things (ex: vfio and kvmfr and other stuff)
In testing
Yeah this kinda was figured out over the past 48 hours
@Kyle Gospo before merging in the initramfs on bluefin
I want to make sure that I think we need to look at the bluefin hardware setup script as well. Make sure kargs aren't being set since Nvidia modprobe arguments are now handled by initramfs
@Kyle Gospo if you can check that using modprobe for amd legacy gpu devices works (I believe this is how it is setup in Bazzite) I think initramfs is good for merging
https://github.com/ublue-os/config/issues/250 putting this in here
I just added it to the hardware setup. I've not noticed needing that with my Nvidia Machine
@M2 looks great to me, approving
or wait I can't, it's mine lol
I will approve, I just want a double check on all the stuff I did
yup, looks great to me
nothing stands out as wrong
okay did you see the simple drm stuff being talked about?
It has a very different karg. One that is actually for the kernel and not being passed to modprobe
@Kyle Gospo @M2 thank you for making the goth chicken a reality.
@j0rge you'll need to approve.... Apparently github is smart enough to recognize I've added a ton of stuff to it
weirldy, the button came back after you hit approve..... So 3 approves
we had a required check for aurora even though I was only modding bluefin files so it never passed, will investigate later
merged it for now and then this one will autorebuild everything anyway
Yeah, we need to improve the logic on these automerge checks. I have it so workflows will not run if the file changes are only in the other one's file path. Aurora changes don't effect Bluefin changes.
Might simply need a step in Aurora and Bluefin to exit (and skip future steps) when an unrelated change is made, rather than having the logic via triggers.
The workflow will then run (and satisfy the required workflows) but not waste time building the wrong image.
That might effect build badges
I would expect the badges to only look at runs on
main? These workflows are ran on PRs, so I don't think it would change badges.
Builds on main are all scheduled (and manual) ones, so this logic to skip builds will not be used.
After the initramfs change, I finally have my LUKS graphical prompt screen back! 🎉
Once rolled out to Bazzite, it should resolve this issue
https://github.com/ublue-os/bazzite/issues/831And reduce setting kargs. And we can steer people to using cryptenroll for luks decryption with tpm2, fido, or pkcs11
With no local initramfs, I have a branded Plymouth and tpm2 unlock works
ok so since i never used initramfs-etc, is it all automatic when we add conf files to
/etc/dracut.conf.d/ or do we have to issue a rpm-ostree initramfs-etc --track="/etc/dracut.conf.d/foo.conf"?
or would it just be easier to do rpm-ostree initramfs-etc --track="/etc/dracut.conf.d" in bazzite-hardware-setup and just issue a rpm-ostree initramfs-etc --force-sync whenever a conf file has been added?
since i have some guides i will have to update/rewrite now 🙂need to manually specify tracking
for the folder
/etc/dracut.conf.d and /etc/modprobe.d
yeah so we should really do that by default then so the user just need to think about
rpm-ostree initramfs-etc --force-sync for simplicity
will play a bit around with it next weekwhile it's fast, unless a person needs it, nothing actually gets added
vfio and kvmfr will need it
if someone needs to add a modprobe file they need it (which is the more likely scenario)
If they have their own initramfs initramfs-etc is not compatible
Only downside for turning on by default
atm they are 100% dracut files but they touch the now obsolete rebuild file
the documentation on discourse about initramfs is also a bit outdated referencing the same file too (but i edited it beforehand to remove all the custom stuff and only describe how to add dracut configs and modprobe)
That's good, but if a user is using clevis that won't work with etc
Probably should just add it
We should add it in config or main
i just opened a PR
Very nice work on this one!
I think that
.conf files must be specified
that folders are ignored
but I could be wrongYeah, I hesitate to recommend a default. initramfs can do things that initramfs-etc cannot do.
correction: only files are supported, extension doesn't matter
for initramfs-etc
--trackthats all i need to know then, just have to set the commands to track the files, force sync, untrack files and force sync again
Mine does that automatically, if you wish to look
Although, I use force sync only when
rebuild file is "touched"
I'm not sure if force sync is needed for tracking & untracking files
But my script can be easily modified to include force sync there if neededok if force sync is only needed when rebuilding then even better 😄
@M2 Are you sure that this modprobe for NVK is working?
options nouveau config=NvGspRm=1
Format looks odd to me.
Based on modprobe docs, something like this is more correct:
options driver option=valuethat line is from me
and it's correct as odd as it looks