On Wed, Jul 05, 2017 at 12:55:04AM -0700, Rick Moen wrote:
> weland>
> But for systemd, the last straw was trying to mount a USB drive one day.
> I would do "mount", and nothing was mounted. I looked through the logs,
> and noticed it was being unmounted on mount. What the fuck, right? Turns
> out, a prior unmount of a USB stick killed some service that systemd runs.
> So systemd thinks the service is dead, which means it should not be
> mounted, which means it keeps umounting the damn thing.

Failing to mount an USB stick might be dismissed, but systemd does the very
same thing with RAID if you dare to mount it degraded (at least btrfs, not
sure about md).

If it's a data filesystem, a workaround is to try to "cd" onto the mount
point after the mount -- it's a race you are likely to win, if you don't,
try again. Yes, this is what systemd users have to put with.

But if you're trying to _boot_ from a degraded mount, there is no recourse;
apparently Fedora uses systemd inside initrd, Debian does not (yet?).
Fortunately, I've never been hit by this case, only by the data filesystem
one thus I did not have to investigate. And it was like: me: here's the
bogosity -- but why in the blazes does this box run systemd? The server's
regular admin: what's systemd? I did not touch that, it installed itself
this way. Me: may I rip it the heck away? -- which once again instantly
saved the day.

> sandGorgon>
> This is actually a good thing. You might have found it
> bothersome, but what it was doing was intelligently trying
> to reason whether a device would cause problems in your machine.

At this point, I think it defies Hanlon's razor, and can't be adequately
explained by a mere bug -- it takes so much effort it smells of malice.
To get this misfunctionality, systemd apparently implemented a worker that
periodically polls all mounted filesystems to check if their "prerequisites"
are met. If systemd's view and reality disagree -- all the worse for
reality! Well, a degraded RAID by definition doesn't have all its
underlying devices in good state: but "I think this would not be possible
to mount" doesn't imply "it's mounted anyway -- I should sabotage it".

And with RAID flexibility offered by btrfs there is no way to definitely
answer "is this ready to mount" other than try an actual mount. Here's such
a scenario:
The RAID has sda, sdb and sdc. Suddenly the cable of sda gets slightly
unseated -- or perhaps the controller has a hiccup, making it go offline.
The admin gets notified, attaches a new disk: sdd, adds to the array. Next
boot, sdd (attached via USB or otherwise slow) takes a while to appear.
Systemd asks the first disk (sda) about how many devices there are: the
answer is "3". M'kay, there are three disks with the right fsid, so all is
ready, right? Here comes the mount() call -- but upon reading the chunk
tree, btrfs uses a copy with highest generation number (ie, not from sda)
and correctly notices there's not enough devices yet; the mount() fails.

> weland>
> What was intelligent about that? I don't need intelligent tools,
> I need tools that do what I say! That's why they're tools, damn it!

A tool that breaks such basic vital functionality as RAID has no place
anywhere around a server.


Meow!
--
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can.
⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener.
⠈⠳⣄⠀⠀⠀⠀ A master species delegates.