Re: Popular Linux OS for dom0 with UEFI boot?

[zithro <slack@xxxxxxxxx>]

On 05 May 2023 19:40, Ray Barnes wrote:
> Hi.  It seems my age-old habit of using CentOS 7 as a dom0 is no longer
> sustainable.  Or at least not as far as I'm aware of.  I have not yet seen
> a solution to the issue of 'multiboot' hangs during boot when the box is
> put into UEFI.  I believe there are packages that were supposed to fix
> this, but they never actually did?
Before going on a mission to show you the possibilities for another dom0 
distro, lemme answer to that.
I have 2 physical dom0s, both debian stable, one being BIOS/CSM based, 
one UEFI based. Both AMD, but different generations : Athlon x4 760k and 
Ryzen 1700x.
Because of personal choices, the Ryzen is using BIOS, the Athlon UEFI 
(counter-intuitive right !).
On the BIOS one, grub always worked, from stretch/buster (~2018) to 
bullseye.
But on the UEFI one, grub failed on me, and had to hack it (~mid 2019, 
from file timestamps).
TBH, I've never taken the time to pinpoint the "offender", but till it 
works ... (that's dumb, it took me time to understand sharing is caring).
What worked for was to re-use an old "20_linux_xen", this way :

- keep
"multiboot	${rel_xen_dirname}/${xen_basename} placeholder ${xen_args} 
\${xen_rm_opts}"
/instead of new/
"${xen_loader} [...]" (on my UEFI platform, ${xen_loader} resolves to 
multiboot2)
- keep
"module	${rel_dirname}/${basename} placeholder 
root=${linux_root_device_thisversion} ro ${args}"
/instead of new/
"${module_loader} [...]"

- keep
"module    --nounzip   ${rel_dirname}/${initrd}"
/instead of new/
"${module_loader} [...]"

- keep
[nothing]
/instead of new/
"if ($grub_file --is-x86-multiboot2 $current_xen); then
        xen_loader="multiboot2"
        module_loader="module2"
    else
        xen_loader="multiboot"
        module_loader="module"
    fi"

I've never tried to understand the problem, so I just kept the 
old/working version.
Maybe someone kind enough will explain us the problem !
(if we both have the same problem though, unsure)
If you need the full files contents, tell me.

> In any event, I'm on the hunt for a dom0 OS, something with good support
> like Ubuntu LTS, capable of running as a dom0 with native repo packages,
> and supporting UEFI boot.  What do all of the cool kids use these days?
>
> -Ray
I'm not really a cool kid, even if I -think- I still am ! Remember, when 
you grow up, only the toys change ^^
I'm a middle-aged sysadmin by trade, and an IT enthousiast since dozens 
of years, but here are my suggestions.
This is *very* partial, but I'll try to separate what I know from my 
personal experience.
Ready for the ride ? This is not your 5min read ^^

To me, those are the available choices, listed alphabetically :
- Alpine
- Arch/Gentoo
- Debian
- Fedora
- NetBSD
- Qubes
- Slackware
- Solaris/illumos
- Suse
- XCP-ng

Again, I don't know enough about all other solutions, so other 
alternatives are viable too.
Anyone is free to comp(l)ete ;)

Considering a dom0, I only have personal experience on Debian and fedora 
(a bit, with Qubes).
Without further ado, let's begin.

--------------
Alpine
--------------
(0 XP, but ...)
- very lightweight
- security & server focused
- used a lot for containers and "small systems", so lot of feedback
- "raw" system: does not want to do everything, just the things it's 
designed for, which is being a server platform
--------------
Debian
--------------
- my personal choice for 2 dom0 on "Network-in-a-box" systems since 5 
years (1 "user like", 1 "server/bkp like"). My config supports pfsense, 
freeBSD-based freeNAS, w7 domains including a gaming host, other Debians 
ofc and various other distros (a nested Qubes, openBSD, and many test 
ditros). PCI-PT active on several domUs for various HW. One dom0 has 
been configured "à la Qubes" (before I learnt about it, so way less 
secure and "integrated").
- stability
- kinda close to unix philosophies, choice of kernel (linux/BSD)
- promotes free software, and more importantly nowadays, free 
*firmwares* (look bookworm handling of free/closed FW)
- huge community, so lot of feedback (I recently joined and posted a lot 
in debian-user ML. Nice people, happy to help)
- choice of init system
- can be used as a small/lightweight server or as a fully featured 
desktop, so you don't need to learn things twice
- upstream of a lot of other distros, particularly Raspbian (ARM) on 
which you can also use Xen as dom0, (from RasPi 4 but iirc possible on 
Rpi3 with hacks)(and Ubuntu just because it's popular, but never used 
it, and I don't like their decisions but again, opinion). It may help to 
have the same OS on desktops/laptops and ARM SBCs.
- I recently chose to be part of the Debian Xen team, and they are nice 
& dedicated people, so you're in good hands ^^ Joke aside, except 
expected minor glitches, my experience has been flawless in 5 years
(I will only develop more -can I ?- about Debian if you ask for it, to 
not pollute even more -possible ?!- with self opinions)
--------------
Fedora
--------------
- you come from CentOS, so it will look familiar (I think ?)
- Qubes dom0 is based on it, so it contributes to the Xen project, 
especially security-wise (read more in the Qubes section)
- because RedHat ... Even though I kinda hate them for systemd, 
described as theoretically useful to system mgmt even from freebsd 
developper(s), but i still fail to see how it's useful to me, creates 
more problems than it solves. The fact Lennhart got hired by MS proves a 
point: as we say in french, "qui se ressemble s'assemble" (~ who looks 
alike, like each other), but /rant off, and again, biased opinion !
--------------
Arch/Gentoo
--------------
(0 XP, but ...)
- outstanding documentation ! Gimme a Linux user who didn't solve a 
problem in its distro without reading their docs/forums, even if not 
using those distros !
- "raw" systems, close to unix philosophies
- highly and easily customizable to your needs, again thx to the docs
- Arch runs on RasPi/ARM, so can host a Xen dom0 (dunno about Gentoo). 
It may help to have the same OS on desktops/laptops and ARM SBCs.
--------------
NetBSD
--------------
(0 XP, but ...)
- because the simplicity and cleanliness of BSD systems
- stability, security
- low overhead
- can also run on ARM (so on RasPis, etc, you got it)

--------------
Qubes
--------------
Here I will consider Qubes as a desktop PLUS server system, not a 
laptop/isolated one.
For now, I'm testing Qubes as a nested dom0, to see how I could replace 
my "vanilla Debian/Xen network-in-a-box user mode host" setup by Qubes.
- Qubes is a really nice dom0 to use for a user environment, as it's 
providing a GUI directly on dom0 to manage the domUs (integration goes 
way beyond virt-manager)
- it's more "user+security-oriented", but nothing prevents you from 
using it in a mixed desktop+server mode
- supports all Xen functionnalities, even if security-wise, it's not 
recommended by the team, ie. not the usual use case
- it has some peculiarities, a bit more than your "average" OS, but once 
you grasp the paradigms, you can do what you want and it's not so hard
- nice and helpful community (I participate in it a bit)
- nice documentation, even if to grasp everything, you need to spend 
some time
- strongly security-focused (even though my use case may reduce overall 
system security), project started by a security-focused company
- strong separation between domains, secure dom0-domU and domU/domU 
exchanges
- uses "advanced" (for me) Xen capabilities, so it's also a good 
learning tool for Xen itself
- even if totally noob, you can follow a few guides and get started 
quickly, -with- network access (and then you consult the online docs 
from Qubes)
- when you know what you're doing, it can provide a quick 
"click-click-it works" experience
- opinion (srsly, again?) : it should be the next-gen OS for everyone, 
at home or at work (hey Marek, when are u switching to a Debian-based 
dom0, which can prevent from using closed source firmware ? ;) Ah the 
usability/security dilemna)
--------------
Slackware
--------------
- because it's the system I learnt Linux on, and I actively participated 
during the creation of "docs.slackware.com" ^^
- so ... documentation !
- in-system/offline documentation: you can learn GNU/Linux w/o Internet 
not only by reading the integrated docs, but most importantly by reading 
the config files ! Strange to say nowadays though, but when you only 
have a (not smart) phone at hand and try to reach the internet with 0 
linux knowledge, everything is there for you to succeed ... Priceless.
- "if u wanna learn $distro, use $distro, if u wanna learn GNU/Linux, 
use Slackware"
- Pat Volkerding, the BDFL, is a bright, knowledgeable and nice guy 
adhering to the KISS philosophy (w/o comma). Never underestimate history.
- stability, security
- "raw" system, close to unix philosophies
- very nice community, with a ring-like structure: Pat provides the 
base/ring0, his "guards" provide "easiness" (for peasants like me ^^) 
(read slackbuilds by alien and more)
- you're in charge, not the system, but nowadays ...
- package managers ! (you can laugh, apt stuff is practical but -to me- 
too much hand-holding)
- used to work on old RasPis/ARM (self-tested, Pi1B+), but 
unfortunately, AFAIK support for old ARM archs had to be dropped cause 
€€€ :(
- so again, it may help to have the same OS on desktops/laptops and ARM 
SBCs.
--------------
Solaris/illumos
--------------
- ok, that's just to rant about how Oracle killed it ...
- but good job joyent keeping it alive
(afaik, Xen dom0 does not work anymore on illumos)

--------------
Suse
--------------
(0 XP, but ...)
(I've only used Suse when it was like SLES8/9 (~early 2ooo), and only on 
servers, and unfortunately don't have much knowledge, but ...)
- Suse is actively participating in Xen developments !
- pro/personal versions, nice tools
- can be used as a small/lightweight server or as a fully featured 
desktop, so you don't need to learn things twice
--------------
XCP-ng
--------------
(== XenControlPanel-newgen ?)
Sorry but it needs a full paragraph for itself (again ?!), because this 
is the exception amongst all others, as it's not really a distro per-se 
that you install and then install Xen on top (below!), but a pre-built 
all-in-one server solution.
Considering usability only, XCP-ng can be thought of like the 
server-only version of Qubes: it also has a nice management GUI, but it 
must be accessed from a remote host, usually via a browser. It also 
provides, if u need it, Xen Orchestra, a web-based management interface 
to your XCP-ng server [farm].
Note, and sorry for Citrix devs, that I omitted Citrix/XenServer from 
the list as (last I've checked), it's not really home/enthusiastic-user 
friendly.
Apologizes and correct me if I'm wrong, but this is my experience. In 
fact, the very first Xen-based system I tried was Citrix XenServer (iirc 
before XCP-ng even existed ? at least I didn't know of it), but it 
imposed restrictions that were unacceptable for my use case, 
particularly on PCI passthrough (wanted a "Network-in-a-box" solution, 
so consolidating all my hosts into one, hence including PCI-PT for my 
gaming/multimedia machines). Did that change ?
Then came XCP-ng ! An open-source fork of XenServer, with no 
restrictions at all.
Note before my remarks, even though XCP-ng (or XenServer) can be used at 
home, those are systems fully qualified to handle a farm of dom0s ! Read 
"enterprise-ready", and even "big corps ready".
- first and foremost, XCP-ng provides a management interface "above" 
Xen. It's called XAPI (Xen Project Management API), read more there : 
"https://xenproject.org/developers/teams/xen-api/";. Please note that the 
company behind XCP-ng (vates.fr), is currently investing to rebuild the 
Xen www and wiki/docs (and as a Debian-Xen team member I even urge/spam 
them so we all can get docs as good as the software is)
- easy to use web interface: you can manage 1->n hypervisors, and like 
Qubes, you don't need to know everything about Xen to create your first 
domUs
- lightweight on the servers/hypervisors
- advanced Xen functionalities accessible via "click-click it works", 
This has so many features I can't list em all (consult the docs). As a 
vanilla Xen user, I can tell you : what I have to handle with home-made 
scripts and/or manual intervention is all handled by the GUI (the 
infamous difference between corporate-oriented software stack versus 
i-do-it-in-my-cave)
- reactive and friendly community
- good documentation
- ofc, contributes a lot to Xen "base"

------------------------------------------

Soooooooooooo, this is a way too long answer ...
But it's on a mailing list, so maybe it will help others like you, 
picking the right tool for the right needs.
Of course as I said before, this is very biased.
But everyone is free to correct me or enhance what I said.
I just felt that the mind blowing discoveries Xen brought me, as an IT 
enthousiast AND sysadmin by trade, deserved some time sharing !
Whatever your choices, happy computing !

++
zithro