I just wanted to thank all who responded to my post: the comments were
extremely thought-provoking. One thing which came out often and
confirmed my own view is that it takes a sysadmin to assess the
capabilities of a sysadmin: for management to do so, without the input
of the guys at the coal face, is potentially disastrous.

I can't add much to the description of the situation without violating
confidentiality constraints. However, I can say a few things about my
own position re. this. If it's TL:DR - apologies and no worries.

I've worked as a unix and linux sysadmin for over 25 years, and I'm
preparing to retire soon (hopefully next year. Quite frankly, given the
systemd insanity, I can't wait.). (Before the unix/linux phase, I
managed IBM S/370 and S/390 mainframes at various large concerns all
over the UK, for 10 years. I still miss mainframes - I suppose in the
same way that people miss steam locomotives.)

When it was suggested to me that a linux sysadmin might not be able to
manage a server running a non-systemd-based linux distro/release, my
first reaction was incredulity. It just seemed too preposterous even to
contemplate. However, I gradually started to wonder. If it's the case
that most linux servers are getting upgraded to systemd-based releases,
then obviously the sysadmins managing them will get more exposure to
journalctl, sysctl and the other sysadmin tools, and less to the
traditional unix/linux tools, and this would obviously be worse in the
case of sysadmins who have taken up the profession recently. (But I
still can't see that any linux sysadmin worth his/her salt would ever
lose touch with the traditional unix/linux tools and techniques
altogether.)

But in any case, how quickly ***are*** servers being upgraded to
systemd-based releases? Throughout my career, it's been my experience
that if a server is running a good and stable release of a particular
distro (such that the box stays up for years without a problem); and if
it's required 24/7 and there are no defined maintenance slots; and if
it's relied upon by hundreds of people to do their jobs and earn their
living; then the server tends not to be upgraded often. (Releases vary
widely - almost like wine vintages: the bad ones get flushed down the
drain and forgotten about; the good ones last, and even get better over
the years.) Apart from the hassle and interruption to normal usage
caused by the upgrade process, there's always the possibility that the
new release may introduce new problems and vulnerabilities. In the case
of going from a non-systemd to a systemd release, there's (at least) the
extra problem of the sysadmin having to learn a complete new set of
management tools and procedures, and problem-solving skills, and
potentially being less effective and productive during that learning
period. Which obviously, given the potential issues from the new
release, is just the time when the sysadmin needs to be on top form.

In my experience, the only valid motivations for upgrading to a new
release have been one or more of:

- a new exploitable vulnerability being discovered in the existing
release

- new hardware being installed, which isn't supported by the existing
release

- new releases of application software which require functionality not
present within the existing release.

Giving in to pressure from management to go to the latest shiny new
release just for the sake of it is - IMHV - just about the worst case in
existence of fixing what ain't broke. (I still remember when I and a
number of fellow sysadmins were persuaded to migrate our workstations
from (pre-enterprise) Red Hat 6.2 to the brilliant new RH9. After doing
so, we discovered that extensive "upgrading and enhancement" had been
done on the fonts and font management tools - and as a result, we
couldn't read what was on our screens any more. I believe it was the
first time that "anti-aliasing" had been rolled out: with it on, the
characters were blurs; with it off, they were horribly jagged and
multi-coloured. It took several days of fiddling with settings before we
could comfortably use GUI-based text tools again, and we never got back
to the readability and "look" of the RH6.2 desktop fonts.)

I've been hunting for data on internet-facing servers, to try to find
out what proportion are now running systemd-based releases. There are
plenty of statistics about which distros are running on them, but it
seems impossible to go any deeper and find out details about the release
levels. If anyone knows a source for this information, I'd be very
interested.

Apart from my visceral hatred of systemd due to the size and penetration
of it, the violation of basic unix principles, and the fact that I can't
see why I should have to use it if I don't want to use it, etc. etc. -
the main reason why I'm scared of using a systemd-based distro/release
on a server is because on several occasions, I've experienced hangs
during shutdown and startup when running such a distro/release. The most
recent occurrence of this was when I took on the management of a server
which was malfunctioning occasionally. I found that at some point -
probably accidentally - it had been upgraded to Ubuntu 15, the first
fully-systemd release. Also, it was running mysql, and the mysql root
directory had been moved to a non-default location. One problem was that
mysql would not shut down properly - often, the mysqld process had to be
killed manually.

One day, the power to the server got interrupted, and it wouldn't come
up again. I managed to boot it from a rescue CD, and spent several hours
trying to discover what had gone wrong - without success. The systemd
"tools" drove me insane - it felt as though they were actively
preventing me from accessing the information I needed. Eventually, I
tried a hunch: maybe the default credentials required by mysqladmin
might not have been copied across when the mysql database directory was
moved (one of the nastiest gotchas in running mysql under debian and
derivatives). This proved to be the case, and when I'd fixed it, the
system came up happily, and a reboot also worked. (As soon as possible,
I rolled it back to Ubuntu 14.) The problems were: (a) I'd not been able
to find any explicit information about the issue. Obviously, my
unfamiliarity with the systemd tools probably didn't help - but it
seemed that journalctl was only showing a sanitized and censored version
of the available information; (b) systemd would apparently not proceed
without confirmation that mysql was up and running. I've had similar
mysql-related problems happen on non-systemd systems - but they always
came up in spite of the mysql startup problem, and the source of the
problem was obvious from the (text-based) logs. In this case, at least I
had physical access to the server, and could press the power button and
stick a rescue CD into the drive. The thought of the same kind of
problem happening on a box in an unattended server farm, or on a VM in a
cloud-based hosting environment, with no or limited virtual console
access - and of course it would happen in the middle of the night ...
Well, it just doesn't bear thinking about!

I've said it before and I'll say it again. All this could have been
avoided - if systemd had been made optional from day 1. People who liked
it could use it; people who didn't like it could use something else.
Email traffic to the systemd developers would tend to consist of genuine
problem reports and requests for enhancement, rather than hate mail. So
there would have been no need for the systemd team to barricade
themselves behind a wall of silence and arrogance. And systemd itself
would gradually improve. Massive win-win in every conceivable way. The
scary thing is that LP and co. must see that as clearly as anyone
else.

On Fri, 2021-11-19 at 11:29 +0000, Peter Duffy wrote:
> I've recently been asked to recommend an upgrade route for a number of
> linux servers, and I proposed going to devuan. In response, I've had a
> concern raised which took me by surprise. It was suggested that in the
> future, it may not be possible to find staff who have the skills to
> administer and manage servers running non-systemd or pre-systemd
> distros/releases.
>
> I've tried to give reassurance - but I'm still wondering if this could
> be a valid concern. I'd always taken the view that it's primarily the
> linux sysadmin community which is trying to stop the onslaught of the
> systemd juggernaut - but obviously, the greater the proportion of
> servers running systemd-based distros/releases, the less staff get
> exposed to non-systemd management techniques and tools.
>
> I'd be grateful for thoughts and comments.
>
> _______________________________________________
> Dng mailing list
> Dng@???
> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng