On 24/11/2018 02:45, Steve Litt wrote:
> On Wed, 21 Nov 2018 12:17:21 +0000
> Roger Leigh <rleigh@???> wrote:

>> Some general points to consider:
>>
>> 1) A separate /usr serves no practical purpose on a Debian/Devuan
>> system
>>

>>      Historically, /usr was separately mountable, shareable over NFS.
>> With a package manager like dpkg, / and /usr are an integrated,
>> managed whole.  Sharing over NFS is not practical since the managed
>> files span both parts, and you can't split the package database
>> between separate systems.

>
> What I hear in the preceding paragraph is that dpkg considers /
> and /usr a package deal (no pun intended), and so can't abide an NFS
> mounted /usr. Telling people to merge / and /usr for this reason is
> fixing the symptom and letting the root cause remain. That's usually
> not a good idea, but perhaps in this case fixing dpkg would be too
> difficult...

This part isn't a problem with the dpkg tool itself. It's down to the
content of packages not having a clean separation between / and /usr.
Programs, libraries and data on both sides of the split are mutually
interdependent upon the other. KatolaZ illustrated this nicely in his
last post with library dependencies across the divide. (This was
forbidden before the MountUsrInInitramfs work.)

There's no clean separation between the "base" system and "everything else".

You can absolutely mount /usr over NFS. But it's not very practical to
share that filesystem between multiple systems. And it isn't a very
useful configuration choice compared with other possibilities. If you
want to use NFS, you can mount the rootfs over NFS (including /usr).
It's a simpler, more practical arrangement, and it's exactly what tools
like debian-live do (for example).

Contrast this with the BSDs where there's a defined "base system" and
then a separate and largely independent collection of packages under
/usr/local. But even on the BSDs, the primary split is between / and
/usr/local, not / and /usr. /usr/local/etc and /usr/local/var exist,
while /usr/etc and /usr/var do not exist on Debian (or FreeBSD); they go
on the rootfs, which is one of the causes of the tight coupling. And
it's not necessarily a bad thing. It's simply a part of the basic
design of Debian that we've accepted for over two decades.

Take a copy of e.g.
http://mirrorservice.org/sites/ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.0-RC1/base.txz

The "base" is the content of / and /usr from a single build of the base
source tree. While there's a separate static /rescue and it's
technically possible to mount /usr separately (there's similar
convoluted logic in the initscripts to what Debian used to have), the
system as provided is a collection encompassing both.

Because it's not (yet) managed by a true package manager, you could
actually set this up in the traditional way if you wished, and share a
static /usr between several systems. But it might still interact badly
with freebsd-update (not tested it myself), and it's planned to come
under the remit of the pkg package manager in time, so like Debian it
may run into logistical problems due to the package management.

>> Modern disk sizes make partitioning a
>> separate /usr unnecessary and undesirable. (Both are of
>> course /possible/, but there is precious little to gain by doing so.)
>
> Well, *you* might not want a mounted /usr, and *I* certainly don't want
> a mounted /usr, but we don't speak for every user in every context, and
> we can't anticipate every context. So "serves no purpose" as a blanket
> statement is false if we find one person using or wanting to use a
> separate /usr on a De??an system.

Absolutely. However, "wanting" to use a separate /usr doesn't imply
that the reasons for that desire are sound or reasonable. This is why I
very much would like to know the underlying rationale for each use.
Some may be genuinely valid. Others may not be. But we need to be able
to objectively evaluate each one to determine this.

If you look back in the debian-devel and other list archives 5-7 years
back or more, this was discussed on several occasions, and it was
increasingly a struggle to identify genuinely valid use cases. Some
were bad. Others were valid, but... pointless. Over the years, the
need for a separate /usr has weakened. Most of the time, a single root
partition is just fine, and this is the default for the installer, and
the way the vast majority of people run their systems. For these
people, a separate /usr doesn't solve any of their problems.

Several of the uses are borne out of habit rather than necessity. The
sharing of /usr over NFS is an instructive one. In discussions a few
years back, this was brought up as a valid use case. One or two users
and developers brought this possibility up. Some people claimed to be
working with this setup. But when questioned about how they actually
made it work, it came down to doing rather custom and fragile stuff
outside the package manager. It turned out that we hadn't /ever/
actually supported this arrangement officially, and it wasn't possible
to do this without stepping completely outside the bounds of good
practice and supportability. It's one of these things which are assumed
to be OK due to historical practice, but when you dig more deeply into
the problem, you realise that it's actually non-viable, and not only
that, it's always been non-viable but people never actually tested it in
anger to find this out. I spent a good while pondering over the
logistics of making it work, articulated my thoughts, and there was a
general response of "hang on, you're right, this doesn't actually work,
does it?". It was never broken intentionally, it was just that we
discovered after many years that if you tried to do it, you'd run into
all sorts of caveats that made it impractical and unsafe to do. This is
one example of the group consensus of what's supportable and reasonable
to do gradually evolving over the course of a few years as positions are
discussed and people dig into the problem in more detail themselves.

At no point has anyone been forbidden from sharing /usr over NFS. But
the group consensus shifted from "Hmm, should work if you really wanted
to do that, but I've never needed it or tried it myself" to "Yeah,
that's not supported and is a really bad idea in practice because it's
fundamentally incompatible with modern package management". That
changed due to investigation, discussion, and people becoming more
informed about the problems.

> I can imagine that somewhere there's a guy who gains speed by, boot
> after boot, copying executables to /usr on a mounted RAM drive, and who
> doesn't want to use an initramfs, who would be quite perturbed by the
> merge.

While people are of course free to do just this, my position is from the
point of view of what is reasonable and supportable for the distribution
maintainers. Can that system be managed and updated by dpkg, or is it a
custom ephemeral point-in-time snapshot of a managed system. Or is it
just a unmaintainable mess?

I also dislike this type of scenario because you've coupled together two
separate things:

- making the system faster
- using a separate /usr

The two are largely unrelated. I'd prefer not to jump from "how do I
make my system faster" straight to "use a separate /usr". I would
rather approach the problem more objectively by going back to the
fundamental requirements, and working forward from there, investigating
all the possibilities.

You could add more RAM to cache reads.
You could replace the old drive with an SSD.
You could run the entire system from a RAM drive.
You could run ZFS and add an NVMe L2ARC to front all the slow storage
with a fast SSD. This could cover all the system, not just limited to /usr.

There are multiple possibilities here to make a system faster, probably
lots of good ones I haven't even thought of. But very few of them
require a separate /usr.

The other part of the scenario you mentioned here is "doesn't want to
use an initramfs". Why? It's the standard way of booting the system
for a reason. It provides a standardised, powerful, flexible and
extensible means to bring up the rootfs before handover to init. If
people don't want to use it, I'd very much like to hear some genuine
rationale for doing so.

As I mentioned in my other post, there is a very real complexity and
maintenance cost for supporting multiple methods of booting. "Just
because I felt like it" isn't a reasonable justification. There must be
some actual technical benefit to be gained from using a non-standard
approach to justify supporting it. I can understand wanting things to
be done as directly and simply as possible. I used to do all this stuff
myself, after all. But after having to maintain all this stuff, I'm
very, very much aware of the true cost behind it. If you want full
control over the initramfs, it's trivial to modify. It's just a bunch
of shell scripts. You can have as much control over them as you do over
the initscripts, should you want to dig into them.

>>      The point about /usr being a good place for "static" content is a
>> reasonable one.  But for better or worse, / is "the system".  It's
>> still part of the managed whole, and hiving off a static /usr rather

>
> That's not what I said upthread. What I said upthread is to continue to
> have static programs in /sbin, sufficient to mount everything and to
> troubleshoot if /usr fails to mount or something else goes wrong. As
> far as /usr, its executables can and should use loadable libraries.

I meant static as in "unchanging" here, rather than for linking.

>
> [snip]
>
>> 2) Moving the content to /usr doesn't preclude moving it to / later
>
> Huh?

I thought I explained this part. Most of the work in the transition is
in removing duplicate paths from every .deb (e.g. /bin/vi vs
/usr/bin/vi) so that the unification can occur without conflicts. Once
it's done, and dpkg supports the altered file paths correctly, it's not
difficult to envisage the symlinks being customisable so that e.g /usr
can by a symlink to /, or /usr/bin being as symlink to /usr/bin.

So if the initial implementation does move a lot of content to /usr,
that doesn't preclude moving it back again at a later date, or for a
fresh installation to do this.

>>      RedHat/systemd have decided to move everything to /usr, and

>
> If you agree with me that the Linux landscape is no longer a
> technocracy, then the preceding half sentence is a reason not to make
> the move.

I don't agree that this is a correct conclusion to draw. I think every
proposal should be considered objectively upon its own merits. I do
think the Linux landscape is less of a meritocratic technocracy than it
once was, and I'm not very happy about that. But I don't think that we
should prejudge a proposal solely based upon who is proposing it. The
systemd developers have had some good ideas, after all. The problem is
that you have to accept them wholesale along with all the hideously bad
ones!

Whenever I've had some idea I wanted to implement, I had to convince
others that they had merit and solved real problems. And I could only
work on one at once. Bad ideas, or ideas which were cool but didn't
solve enough important problems to justify, got naturally filtered out.

The problem we have right now is a loss of accountability,
responsibility and control. A small number of developers have an
incredible amount of power over the rest of the free software
communities, and we lack the means to reign in their wilder impulses and
bad decisions. This loss of control was my primary objection to
systemd, even above the technical considerations, and I don't think I
was wrong in this assessment.

However, I still implemented a number of the systemd ideas in
sysvinit/initscripts, because they provided some useful benefit, and
when reviewed in isolation did have merit.

>> I'd prefer it to be the other way around;
>> it's cleaner and it's what we would choose given a clean slate.
>
> The preceding sentence is true, I'd imagine. But like the original
> systemd debate, it's not an either-or situation. A third alternative is
> to not copy anything anywhere, but instead leave the
> early-boot-necessary stuff in /sbin, where it can be accessed the
> microsecond / is mounted, without the need for /usr to be mounted.

As I mentioned in an earlier post, this separation caused so many
problems that it was removed.

Starting in maybe the mid-2000s we began to run into problems. Stuff
like the NSS or PAM LDAP modules not working without /usr being mounted.
They needed dependencies like libSSL to function, and that was on
/usr. It started with a handful of large universities or corporations
where LDAP was a hard requirement. But as systems got more and more
complex, the amount of stuff required to mount /usr got larger and
larger. Like needing working wifi for networking. Or tools requiring
interpreters.

The whole concept of "early boot" became difficult to support because it
required more and more of the content of /usr to be accessible in order
to mount it and continue with the boot process. This is a really nasty
chicken-and-egg situation. And it's not solvable without:

- moving all required bits onto the rootfs (which was increasingly a
huge chunk of the /usr dragged in via dependencies)
- requiring /usr to be on the rootfs and dropping support for a separate
/usr entirely
- mounting /usr in the initramfs to make it available along with the rootfs

The first option was untenable. It would make lots of systems with a
small rootfs run out of disc space and be unable to upgrade or function.
It's also a losing battle; more and more packages would be dragged
onto the rootfs, so we would gradually be moving more and more content
from /usr onto the rootfs over time. This is very disruptive.

The second option was possible. But had the same space issues as the
first option on upgrade, as well as breaking all installed systems with
a separate /usr.

The third option was the most minimalistic and least invasive approach.
No system would be affected on upgrade. But it did break any remaining
notion of there being a true separation between / and /usr. After the
release of wheezy, / and /usr are *completely interdependent* and
tightly coupled, without any formal restriction.

This is at the heart of my argument that there is little practical
purpose served by putting /usr on a separate filesystem. For all
intents and purposes, the combined / and /usr are "the system", and are
an inseparable, integrated whole. They always were in Debian, *after
boot*. But this made it a reality for *during boot* as well.

>>      The whole discussion is based on the premise that they are
>> separate. In practice, the vast majority of us have them on the same
>> filesystem, given that there is no practical purpose to the
>> separation as in (1) above.

>
> "vast majority of us". Not a distro on this earth has avoided
> disenfranchising a few for perceived ease for the many. But unlike
> Ubuntu and Mint, pre-2014 Debian and post-2014 Devuan have
> traditionally opted for maximum user configurability.

Absolutely. It's hard to satisfy everyone even when you try hard. But
the ability to configure a system is not done just for the sake of it as
an end in and of itself. Every configurable option is there because it
served a real practical need for someone. And as I said in my other
post, every one of those options had a cost to design, implement, test
and maintain.

As needs and requirements change over time, it's necessary to
re-evaluate whether certain options continue to be necessary. Is the
cost of supporting it worth paying for the benefit it provides?

In the case of a separate /usr, if you look at why it's supported since
wheezy, the primary justification was so that we didn't break upgrades
from pre-wheezy Debian systems where the admin had opted to use a
separate /usr. It wasn't supported since it was the best choice, or
because it had particularly great advantages. It was for compatibility.

So if you're doing a fresh installation today, you will find that a
separate /usr is still explicitly supported. But that doesn't imply
that it's a useful choice, or the best choice. It's mainly there to
avoid breakage.

So support continues because we put it in place and left it there. To
date, it's not broken, and it might well work for many years into the
future. But I would strongly caution anyone who does a new installation
with a separate /usr to truly justify to themselves that it's genuinely
necessary for their needs, rather than a merely a novelty or a
historical habit.

(Like many, I used to routinely use a separate /usr on a separate
partition, then LVM LV, until I really looked at the practice and
questioned the real underlying problems which it was solving. I've not
needed one in over a decade at this point. I'm not particularly for or
against having one. It's simply ceased to be a relevant concern for any
of the diverse systems I've worked on, from desktops and workstations,
to cluster nodes, VMs, servers and container images. None of them
needed it.)

For people who find they do still need a separate /usr, that's fine.
It's still supported. But please, do try to think deeply about the
underlying requirements which drove the need for this use case, and see
if they still hold true.


Regards,
Roger