Is it suitable to create server images?
I want to create a server with a web server and some other services, would blue build a good option or would it be better to use another tool?
25 Replies
i think it would be the best to just install ucore and run everything in docker containers
But I want to customise some host stuff like certificates
If im correct certificates are installed to /etc and thus should be changeable even on an immutable distros
I would recommend building your own images if you have more than one server, so you only have to do things once. But if you only have one server it is probably more work maintaining your image than just using ucore and changing a few things
Yes, I will have multiple servers
Multiple physical servers?
Yes
I want to bu this minipc( https://slimbook.com/en/zero), install my services and distribute them
also provide the image freely so someone that has a pc can install it himself
Well that changes a lot, i can recommend this, but only if you have sufficient bandwidth for updates as they can be quite heavy
okay
do you tjink NixOs would be better?
That's up to you. I don't know about NixOs, but Fedora Atomic requires rebooting to update
You can build uCore images if you want. UBlue has a base image with no DE if you want to do that
BlueBuild is mostly designed for desktop image-making convenience
I like CoreOS, but there are many different solutions in the server space, with often their own ways to configure them
My own CoreOS setup is just based on the default version of CoreOS with some containers configured on top, and the butane/ignition configuration AFAIK also allows customizing 'host stuff like certificates'
But there are many other cloud-native (and more ordinary) server operating systems too
IMHO building images is not the best way to manage servers, because then one can't really include secrets, and the configuration is mostly just putting files in the filesystem... CoreOS with butane/ignition I think is really nice
That's not to say BlueBuild is not suitable, if your desired base image is bootc/ostree native container image, and you think that it's a good solution for you
It's really interesting to read this, because I'm still trying to grasp this changing landscape for image-based installs, but I had thought/assumed that building and booting bootc images was a flow more typically targeted at servers, and that it is their use on the desktop that is particularly novel.
I also thought this looked pretty cool for automatically booting back into previous image if an update has broken something on your headless server: https://github.com/fedora-iot/greenboot
GitHub
GitHub - fedora-iot/greenboot: Generic Health Checking Framework fo...
Generic Health Checking Framework for systemd. Contribute to fedora-iot/greenboot development by creating an account on GitHub.
Building bootable images a la bootc is very new, what's common on servers is straight-up containers
Sure I just thought it interesting that as a founder of bluebuild you don't seem eager to introduce a similar workflow into your home server(s)
Yeah, well, I find using and building containers the closest to the same cloud-native ethos
My main home server is a NAS and I'd rather use something stateful but robust than something I built myself on it lol
Then on the Raspberry Pis it's just testing different technologies, but it's arm, and last time I looked into that building arm images was not even anywhere near as well explored as now
Fedora CoreOS I have found to work well with a similar ethos; declarative repdroducable installs, running software in containers. As a plus it runs on arm hah
That makes sense. Presumably you could run your whatever-service-containers within a bootc-server-root-container-image though?
Sure, but I'd have to figure out another way to push in secrets than my bootc image
Yes that's an interesting challenge
Along those lines, here's how mine is set up:
- Main router set to assign static IP addresses to all RPi servers based on their MAC addresses.
- Main NAS running a docker image of nginx-unprivileged, set up to redirect from NAS_IP/fetch.ign to ServerIP/hostname_based_directory/fetch.ign based on the visitor's IP address. One redirect set up in the config per RPi.
- Multiple microSD cards, all with the same ignition configuration, set only to fetch NAS_IP/fetch.ign, which will get auto-redirected to the proper configuration for that particular device.
- The overall ignition configuration is broken up into multiple files. Things like SSH configuration, user configuration, regular backup services, restore-from-backup services for first boot, and other common stuff, is imported into the configuration for all devices. Things like specific podman quadlets for each device are imported only into the specific servers that are supposed to be running them.
If a microSD card dies, I can just pick up any of the pre-ready cards, power off an RPi, replace the card in it, and turn it back on. It'll get restored from last night's backups and off it goes.
I'm often in another city for weeks at a time, so it helps to just be able to give someone simple instructions to replace a card and turn it back on if tailscale shows any RPi is offline for a long time.
This setup sounds really neat, I'd like to try to replicate something like this including non-Pis. I've not used ignition/coreos before.
When you're updating the container-based services, how do the updated containers get pulled onto the Pis? Do you have it configured to check and pull automatically from your NAS?
Wow this sounds amazing
No, I didn't bother with a method to keep the config on each device updated through a central repository. Once everything was fully set up about 1.5 years ago, changes became rare. I might make a one line change to the config here or there based on changing needs, or add or remove a podman quadlet from a device. In that case, I manually replicate that change onto the NAS after testing directly on a server.
Each podman quadlet is set up to auto-update the respective containers using the quadlet configuration.
Each podman quadlet is also set up to call the backup service as a timer for itself, which handles backing up the container volumes.
Backups go into each device's separate directory on an NFS share on the NAS based on the hostname of each device. Old backups are deleted by the backup service automatically.
SSH keys are signed weekly automatically using scripts and timers on both ends (NAS and each device), and they are set to expire after a few months.
Butane/ignition give you the ability to create a file or folder on the device, define file contents inline in the butane/ignition or fetch the file from a link, and set its permissions. It also lets you enable or disable systemd services. I've used that to create a few scripts in
/var/usrlocal/bin, systemd services in /etc/systemd/system and podman quadlets (.container and .volume files) in /etc/containers/systemd.
Butane / ignition only runs at first boot. It does not run again.Pardon my ignorance, but I got slightly confused by your responses:
"No, I didn't bother with a method to keep the config on each device updated through a central repository"
and
"In that case, I manually replicate that change onto the NAS after testing directly on a server.
Each podman quadlet is set up to auto-update the respective containers using the quadlet configuration."
which seemed to be contradictory, because aren't the quadlets therefore auto-updating from the central NAS? 😆
Sorry for the lack of understanding, but your setup is very interesting.
Quadlet configuration is comparable to defining a docker compose file. I don't keep those in sync using a central repository. i.e., there's nothing like a git pull or scp on each device to regularly fetch the latest version of those files from the NAS. If I change something on a device and it works as intended, I manually copy that change over into the butane/ignition on the NAS for future deployments. It's possible to make a method to update all configs on the NAS and fetch them on each device regularly, but that would be more of a hassle than convenience for my use-case.
Containers, though, are pulled daily by each device in case an updated container image is available because of the line
AutoUpdate=registry in the quadlet's .container file. See AutoUpdate= in https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html for more information about this.Ah, I understand what you're saying now. Thanks for explaining!