BlueBuild•3mo ago

Building custom ARM64 images?

Hello! Is it possible to build arm64 images with Blue-Build? If so, what should be changed in the build recipe or the GitHub workflow, based on the Blue-Build template? I found this PR (https://github.com/blue-build/cli/pull/191) from the blue-build/cli repo mentioning ARM, but as far as I can see it's only adding support for building an ARM64 binary for the cli tool. Or is it better to try building from scratch by writing a Containerfile? The docs about this topic aren't finished yet unfortunately (https://blue-build.org/how-to/from-scratch/), but this link seems to have some useful information: https://github.com/ublue-os/website/pull/688/files As far as I can see, there are no prebuilt Ublue OCI images for ARM64 either so I'd have to start with Fedora Silverblue or similar as a base, correct? I'm happy for any advice, thank you!

GitHub

feat: Add arm support by gmpinder · Pull Request #191 · blue-build/...

This adds support for ARM based images and binaries.

GitHub

feat: syncing page and minimal setup page for tinkerer's guide by x...

Tackles #331 while adding some syncing instructions. The minimal setup stuff has been tested here: https://github.com/xynydev/test-minimal-oci

73 Replies

Luke Skywunker•3mo ago

You're pretty much on point so far. The basic requirements would be to run the bulid on an arm64 runner (self-hosted or otherwise) and use an arm64 base image The modules are written in bash and as you've found, the CLI is being compiled for ARM so you can install the binary and use it immediately on an arm based machine We've just not done any of that ourselves

NargaOP•3mo ago

Thanks! I used the blue-build cli to build a basic Containerfile and will try it with podman build --arch am64 first, on an x86 machine using qemu. Let's see if that can produce anything and then I'll need to go from there. One of the main culprits for ARM is the bootloader currently since usually ARM SBCs like Raspberry Pi and others don't support UEFI.

Luke Skywunker•3mo ago

Maybe adding a platform arg would be useful 🤔

NargaOP•3mo ago

Add a platform or arch arg to the bluebuild cli you mean? That would definitely make some things easier! Probably shouldn't be too difficult to implement. But for now, just using the bluebuild cli to generate a Containerfile is also pretty nice.

Luke Skywunker•3mo ago

It'd be real easy to add Think I might do that after I'm done with this bug

NargaOP•3mo ago

That will be great!

Luke Skywunker•3mo ago

I think I've got it working Some packages might fail to install lol I should probably also find a way to have the architecture easily available as an env var during building

NargaOP•3mo ago

Sounds good! Is there a reason you chose to use --platform instead of --arch? (linux/arm64 versus just arm64) Is this considered best practice? If so, I'll try to use --platform in my test builds as well

tulip•3mo ago

❯ podman build --help | grep arch
      --arch string                                  set the ARCH of the image to the provided value instead of the architecture of the host (default "amd64")
      --dns-search strings                           set custom DNS search domains
      --platform OS/ARCH[/VARIANT]                   set the OS/ARCH[/VARIANT] of the image to the provided value instead of the current operating system and architecture of the host (for example "linux/arm") (default [linux/amd64])

❯ podman build --help | grep arch
      --arch string                                  set the ARCH of the image to the provided value instead of the architecture of the host (default "amd64")
      --dns-search strings                           set custom DNS search domains
      --platform OS/ARCH[/VARIANT]                   set the OS/ARCH[/VARIANT] of the image to the provided value instead of the current operating system and architecture of the host (for example "linux/arm") (default [linux/amd64])

should probably be the exact same? --arch apparently lets you crossbuild images

❯ podman run --platform linux/aarch64 --rm hello-world
exec /usr/local/bin/podman_hello_world: exec format error

❯ podman run --arch aarch64 --rm hello-world
exec /usr/local/bin/podman_hello_world: exec format error

❯ podman run --platform linux/aarch64 --rm hello-world
exec /usr/local/bin/podman_hello_world: exec format error

❯ podman run --arch aarch64 --rm hello-world
exec /usr/local/bin/podman_hello_world: exec format error

NargaOP•3mo ago

Well I guess --platform linux/aarch64 is the same as --arch aarch64 if your host is Linux 😄 The step I'm currently struggling with is generating an image file of the rootfs (iso/img or even a rootfs tar file) from the built OCI image. Any tips on which tool might be usable for this task? build-container-installer (https://github.com/JasonN3/build-container-installer) only seems to work on x86 and not for building an arm64 ISO.

tulip•3mo ago

you know what? --platform linux/* could be the best thing to use due to it being technically portable between OSes like, you could just use a single command for building a linux container in both freeBSD and linux :p super dumb thing but still kinda interesting if you want to build a rootfs image you can do podman run --name $CONTAINER localhost:$BUILT_IMAGE /bin/false then podman save $CONTAINER > test.tar and push it into github artifacts building an ISO shouild be actually pretty easy! check out this thing:

tulip•3mo ago

https://github.com/atomic-studio-org/Atomic-Studio/blob/main/.github/workflows/build-iso.yml

GitHub

Atomic-Studio/.github/workflows/build-iso.yml at main · atomic-stud...

Operating system based on Fedora Atomic meant for content creators and artists - atomic-studio-org/Atomic-Studio

tulip•3mo ago

oh aw this thing died, didnt know cant you set ARCH to aarch64?

NargaOP•3mo ago

Very cool, I didn't know that, thanks! I'll miss out on all the automated goodness that the ISO builders have, like signing/verification and stuff. But the advantage is that if I only have the rootfs I can partition the img file myself. The builders typically partition with EFI and grub, which isn't really a thing on ARM (apart from some exceptions). Instead there is U-Boot as bootloader. Yeah this also is mostly build-container-installer 😅 I can, but it doesn't work. https://github.com/JasonN3/build-container-installer/issues/141

tulip•3mo ago

i thought it was still being maintained :( oooooh

tulip•3mo ago

tulip•3mo ago

fedora doesnt have the damn cdboot thing for grub-efi on aarch64 thats why it doesnt work...

NargaOP•3mo ago

😦 Well every project is a time and work effort. If the upstream project is "good enough", then it's good to use the time and energy elsewhere imo 😄 Mmm yeah seems like it! Thanks for your effort! I wish I could just disable the installation of grub and efi related stuff, since I'd need to remove that afterwards anyway 😅

tulip•3mo ago

it seems that at least previously there were some packages for that though its kinda weird that they just removed it from fedora? wdym? are you installing systemd-boot?

NargaOP•3mo ago

Maybe it was there in Fedora 39 but not in 40? Distros change over time and add or remove packages, that's pretty normal I believe.

tulip•3mo ago

honestly, no idea. i checked out the places that have that specific package and it seems like oracle linux has a grub2-efi-aa64-cdboot package in its repos it just contains two EFI files though thats it

NargaOP•3mo ago

No, ARM usually doesn't work with EFI, which means grub and systemd-boot is pretty much useless. Normally U-Boot (https://github.com/u-boot/u-boot) is used as the bootloader, which can directly load the kernel and the specific device's devicetree (or extlinux, in theory it can even load grub but that'd be just an extra layer that's not needed if you can load the kernel directly)

GitHub

GitHub - u-boot/u-boot: "Das U-Boot" Source Tree

"Das U-Boot" Source Tree. Contribute to u-boot/u-boot development by creating an account on GitHub.

tulip•3mo ago

oh! yeyeyeah! have you used systemd-repart at some point? (or heard of it) probably generating the rootfs tarball, removing all the grub things and installing systemd-repart should be your solution for the img approach

NargaOP•3mo ago

Yeah I think I tried to use it at some point, but failed to understand how to use it in the initramfs phase. I tried to use it to extend a partition and its filesystem iirc (if it's the tool which I think it is)

tulip•3mo ago

yeah its the thing that extends the partitions on first-boot i guess you are building this image to just work on a single device right?

NargaOP•3mo ago

Yeah, but have you tried to extend a rootfs filesystem with it that can't be extended while mounted? If you know how to do that, please let me know! I think I spent over 10 hours before giving up 😅 (Of course you can't simply unmount the rootfs on the running system, so likely the only option would be initramfs while the rootfs is not yet mounted. There is some systemd doc I believe that claims that systemd can run during initramfs but I had no idea how to achieve that)

tulip•3mo ago

you should be able to include the systemd tag in dracut.conf itll just use systemd on the initramfs from there

tulip•3mo ago

https://wiki.archlinux.org/title/Dracut

Dracut

tulip•3mo ago

oh sorry jesus christ my keyboard

add_dracutmodules += " systemd "

add_dracutmodules += " systemd "

yeeeeah

NargaOP•3mo ago

Yes, the image can only be built for one specific device since the bootloader needs to have the device's devicetree hardcoded (so it knows which device it is running on) Cool, thanks! The system I was working on didn't have dracut unfortunately and switching to it is quite a process. Especially on Debian/Ubuntu where software packages are lacking behind a lot 😅

tulip•3mo ago

oh yea :p makes sense honestly

NargaOP•3mo ago

I don't quite understand what you meant by this though. How can systemd-repart help with img generation? Or did you mean use it to extend the rootfs partition on boot?

tulip•3mo ago

but still like, you could make a script run on the github actions runner that just builds your thing for you like this:

# include the systemd-repart, dracut configuration, uboot-tools and devicetree in the base image (also generate the initramfs from there) 

apt install -y util-linux bubblewrap podman

IMAGE=ghcr.io/youruser/yourimage:latest
CONTAINER=tmpcontainer
DISKIMAGE=diskimage.img
podman run --name $CONTAINER $IMAGE /bin/true
podman export $CONTAINER -o export.tar
mkdir -p rootfs/boot 
tar -xvf export.tar -C rootfs
qemu-img create -f raw $DISKIMAGE $(du -sh export.tar | cut -d" " -f1)
rm export.tar
sfdisk $DISKIMAGE <<EOF
label: gpt
unit: sectors
first-lba: 64
name=spl, start=64, size=16320, type=8DA63339-0007-60C0-C436-083AC8230908
name=uboot, start=16384, size=16384, type=8DA63339-0007-60C0-C436-083AC8230908
name=boot, start=32768, size=2GiB, type=BC13C2FF-59E6-4262-A352-B275FD6F7172, bootable, attrs="LegacyBIOSBootable"
name=root
EOF
losetup -fP $DISKIMAGE
mkfs.vfat -L BOOT /dev/loop(IDK YET)p(3 - for boot partition)
mkfs.btrfs /dev/loop(IDK)p(4 - for root partition)
mount /dev/loop(IDK)p4 /mnt
mount /dev/loop(IDK)p3 /mnt/boot
rsync -avhx rootfs/* /mnt
bwrap --dev-bind /dev /dev --proc /proc --bind /mnt / /bin/sh -c "update-uboot ; genfstab -t PARTLABEL / > /etc/fstab ; exit"
umount /mnt/boot
umount /mnt
losetup -d /dev/loop(IDK)

# then publish the artifacts to github

# include the systemd-repart, dracut configuration, uboot-tools and devicetree in the base image (also generate the initramfs from there) 

apt install -y util-linux bubblewrap podman

IMAGE=ghcr.io/youruser/yourimage:latest
CONTAINER=tmpcontainer
DISKIMAGE=diskimage.img
podman run --name $CONTAINER $IMAGE /bin/true
podman export $CONTAINER -o export.tar
mkdir -p rootfs/boot 
tar -xvf export.tar -C rootfs
qemu-img create -f raw $DISKIMAGE $(du -sh export.tar | cut -d" " -f1)
rm export.tar
sfdisk $DISKIMAGE <<EOF
label: gpt
unit: sectors
first-lba: 64
name=spl, start=64, size=16320, type=8DA63339-0007-60C0-C436-083AC8230908
name=uboot, start=16384, size=16384, type=8DA63339-0007-60C0-C436-083AC8230908
name=boot, start=32768, size=2GiB, type=BC13C2FF-59E6-4262-A352-B275FD6F7172, bootable, attrs="LegacyBIOSBootable"
name=root
EOF
losetup -fP $DISKIMAGE
mkfs.vfat -L BOOT /dev/loop(IDK YET)p(3 - for boot partition)
mkfs.btrfs /dev/loop(IDK)p(4 - for root partition)
mount /dev/loop(IDK)p4 /mnt
mount /dev/loop(IDK)p3 /mnt/boot
rsync -avhx rootfs/* /mnt
bwrap --dev-bind /dev /dev --proc /proc --bind /mnt / /bin/sh -c "update-uboot ; genfstab -t PARTLABEL / > /etc/fstab ; exit"
umount /mnt/boot
umount /mnt
losetup -d /dev/loop(IDK)

# then publish the artifacts to github

@Narga this is just a rough sketch but it could actually work when worked upon

NargaOP•3mo ago

Thanks a lot, this looks great! I'll try to adapt this later. I already tried to build a custom RPM package for the custom kernel which is needed. In the RPM package I have instructions to remove all the default kernel packages, but I still need to figure out which kernel files I need to copy into the final image to make it work. I never made an RPM package before, so these .spec files are actually quite intimidating for me 😂

tulip•3mo ago

no need to suffer, just use rpkg for building the spec file its so much easier, like, really

tulip•3mo ago

https://github.com/atomic-studio-org/Theming

GitHub

GitHub - atomic-studio-org/Theming: Theming for Atomic Studio

Theming for Atomic Studio. Contribute to atomic-studio-org/Theming development by creating an account on GitHub.

tulip•3mo ago

check out rpkg*, src/**/**/*.rpkg.spec and Containerfile but yeah .spec files really suck

Luke Skywunker•3mo ago

Mostly to ensure it's building on a linux platform I've still got some things to work out in the PR, but it should be good soon

NargaOP•3mo ago

Rpkg? Another thing I need to have a look at. Thanks for pointing me to it 🙂 I had a look at the Fedora kernel spec and it looks humongously complex. I'll keep it simple 😅

NargaOP•3mo ago

Progress, yay 😄 There's a whole heap of things to do still, though.

tulip•3mo ago

very nice!

Luke Skywunker•3mo ago

Ok I've got it working I'm going to try to get this out in a patch either today or tomorrow

NargaOP•3mo ago

Awesome, thanks! That's very cool. I just saw the new release is already out 💜 What do you think about being able to set the arch/platform in the recipe?

Luke Skywunker•3mo ago

That could be worth a shot @dev what do you think?

RoyalOughtness•3mo ago

trying to catch up on this thread, this is big news what upstream images are available for arm? or is this support for composing arm images from scratch either way good stuff!

Luke Skywunker•3mo ago

It seems the fedora atomic images already have arm version being created I'm just making it easier to build arm images via bluebuild As far as I know, Ublue hasn't created any official arm images

xyny•3mo ago

would that change containerfile templating as well or just the build process? im not sure if it would be a better fit in the recipe or as a cli arg especially if building for multiple arches is a viable usecase

Luke Skywunker•3mo ago

So doing it in the recipe would allow me to build both images and put them together with a single manifest so that they are a multi-architecture image. These would take way longer to build because arm just takes a while but we do the same thing that work all the time for our images because our devs are on both x86 and arm To support this, I would need to continue work on trying to do multi-recipe builds using compose files. That would probably be the best way to go about this. I'd still have to test to make sure that that would be possible, but other than that, we can totally do a multi-architecture image deal. But to answer your question, it's just the build process. The container file would stay the same. It would just be built using two different platform arguments. We could then use the CLI arg to override the list of architectures to build.

xyny•3mo ago

Do architectures need to be separated by tags or is that handled by the OCI standard? If it's done natively I'd be pretty much on board with listing arches in the recipe

Luke Skywunker•3mo ago

They can use the same tag. Podman and docker has the ability to take two images and join them under a single manifest and tag them as a single image. When a user pulls the image, it will attempt to pull the architecture that matches the user's machine It's intended use is for multi-arch images So it's a standard pattern

Luke Skywunker•3mo ago

If you go here https://quay.io/repository/fedora/fedora-kinoite?tab=tags you can see that some tags have multiple architectures

Quay

Quay is the best place to build, store, and distribute your containers. Public repositories are always free.

RoyalOughtness•2mo ago

got it, wouldn't this make it possible though to use bluebuild to recreate ublue-os/main and ublue-os/hwe but for arm and using bluebuild?

Luke Skywunker•2mo ago

Yeah totally

RoyalOughtness•2mo ago

😮 interesting

Luke Skywunker•2mo ago

You would have to translate their containerfiles over to a recipe Hell you could use the containerfile module for an easier transfer

RoyalOughtness•2mo ago

I'd rather do the former tbh to this day the scope of ublue-os/main remains unclear to me and if I'm going to make a copy of it to enable arm for secureblue, I might as well make the scope clear and build it for x86 too

Luke Skywunker•2mo ago

I'm glad there seems to be interest in this. I think this would be a great new feature to add

RoyalOughtness•2mo ago

which feature, arm support?

Luke Skywunker•2mo ago

I might end up using this to build myself an image for my VM for work

RoyalOughtness•2mo ago

or being able to use fedora images?

Luke Skywunker•2mo ago

Multi-arch recipes

RoyalOughtness•2mo ago

ohh yeah would be cool