On 22/11/2018 22:24, Alessandro Selli wrote:
> On 22/11/18 at 19:21, Roger Leigh wrote:
>> On 21/11/2018 16:11, Alessandro Selli wrote:
>>> On 21/11/18 at 13:17, Roger Leigh wrote:
>>>> Hi folks,
>>>>
>>>> I've been following the discussion with interest.
>>>
>>>
>>>    No, you definitely have not followed it.  In fact you are
>>> disregarding
>>> all the points that were expressed against the merge.
>>
>> Let me begin by stating that I found your reply (and others) to be
>> rude, unnecessarily aggressive, and lacking in well-reasoned objective
>> argument.
>
>
>   Oh poor show flake, did I hurt your tender feelings when I state facts?

I did find your reply unacceptably rude. This had nothing to do with
the "facts" (which were largely absent) or the points you were trying to
make, and everything to do with the manner in which you said it.

Impolite and intemperate language does not add anything of value to the
points you are making. It rather detracts from them and leads the
reader to devalue and dismiss what you are saying as a result. This
doesn't just apply to yourself, but several of the other posters over
the last few days. Productive technical discussion is impeded by
treating others with contempt and accusations of bad faith.

On a personal note: I spent well over a decade working on different
parts of Debian and loved being a part of it, and had many friends
there. I left the Debian project after enduring over two years of
abusive and disrespectful communications, primarily with regard to the
systemd "debate" becoming overly personal, rather than sticking to
technical facts. It got to the point that I dreaded opening my mail
client to see what new abuse had been sent. Over the course of several
months I became thoroughly stressed out, demoralised and demotivated by
the continual barrage of negativity and disrespect. It caused real
depression which seriously affected my wellbeing in the real world.
Coupled with severe RSI problems (which might not have been unrelated),
I decided leave the project. Not because I wanted to, but for my
physical and mental wellbeing. When you're saddled with a huge
responsibility for keeping everyone's systems working, and have an
unwritten obligation to do so, and you have to spend a huge fraction of
your unpaid off-work time on it, and that time has become unpleasant,
stressful, physically damaging due to the RSI and no longer provides
even the last bit of enjoyment you once derived from it, it's time to
call it quits. It was the RSI which forced the issue; I could no longer
physically meet my duties and commitments as a developer while also
doing my day job. But I'd been desperately unhappy for many, many
months as well.

The words you write from the anonymity and comfort of your home or
office do have an effect upon those who receive them. I'm not a
snowflake. I'm a 39 year old professional software developer with over
18 years of experience in several different fields, and a science PhD as
well. I'm very happy to engage in robust technical debate, but that
isn't what you're doing here. If people aren't prepared to attempt a
minimum amount of politeness and respect for their fellow human beings,
I simply move to work on other projects which have more mature people
working on them. Life is too short to tolerate or suffer from
unnecessary abuse.

In short: Don't remove the joy people get out of participating in free
software projects by being abusive. We can all be better than that.

>   Again I express something so simple I'm really beginning to lose my mind:
>
>
>   I do not intend to deprive anyone with the freedom to merge /usr to /,
> damn it!
>
>   I want to preserve *MY* freedom of choice, I want to be able to split
> from / anything that is not required on *MY* systems or that will never
> be required on any system (Java, Apache, Squid, Xorg, LibreOffice etc).
>
>   I am not fighting against people who want to introduce different
> filesystem layouts because of their special needs, I WANT THEM TO STOP
> FORCING THEIR DESIGN CHANGES TO ME FOR NO OTHER REASON THAT TO THEIR
> SYSTEMS THEIR LAYOUT MAKES MORE SENSE!

The individual components of the operating system are not developed in a
vacuum. They are developed in collaboration with and consideration of
the rest of the system they reside within and interoperate with.

For the /usr mounting in initramfs history I posted, this body of work
took over a year to finish. Not because of technical complexity, but
because it required precise coordination between several different parts
of the system, including: initramfs-tools, initscripts, grub and others
(including systemd). And it needed staging in a fixed order to avoid
breakage. All this required coordination and collaboration between
multiple groups of people. Each group needed to understand the big
picture issues, as well as how their part was affected in relation to
the parts around it. This required extensive (and polite) negotiation
and discussion. They all needed convincing that it was both necessary
and the best course of action to take. And that discussion also fed
back into iterative refinement of both the design and the eventual
implementation.

In short: a big project can't easily change direction due to the actions
of a single person; it requires extensive work by multiple people to do
so. This imposes a barrier which filters out ideas which are bad, or
are not sufficiently good to warrant disruption. cost/benefit again.
It can be frustrating and make things take a long time, but this is part
of what provides stability by limiting the speed and scope of drastic
change. At least until systemd came along; they didn't have quite the
same priorities, but even they have been constrained to some degree by
the factors at play at the scale of a whole distribution.

Debian, the operating system, isn't an anarchistic free-for-all where
anything goes. It has a coherent design. That design allows for
certain expected use cases. The initscripts and initramfs also have a
coherent design with expected use cases. They were designed and
engineered to work in a certain way for certain scenarios. Any changes
to the design are a result of bug reports, discussion and careful
consideration.

There is no absolute freedom of choice. You do have freedom, but that
freedom is constrained by the design and implementation choices of the
system as a whole. What you can do today is made possible because the
system was *designed to allow it*, and you either fit your choices into
that overall design, or you change the design (but at great expense and
difficulty).

Think of the ability to make a choice as factors along two separate axes:

1) Support status: unsupported -- partially supported -- fully supported
-- recommended

2) Functionality: broken -- partially functional -- fully functional

(these are fairly arbitrary labels along the two axes, but should serve
the point)

In the standard workflow for booting the system:

a) initramfs with single filesystem

    recommended and fully functional (default)

b) initramfs with separate /usr

    fully supported and fully functional

c) direct boot with single filesystem

    fully supported and fully functional

d) direct boot with separate /usr

    unsupported and unknown functionality (may be broken or functional; 
we don't know because we don't test it)

These are just four; there are several other combinations including all
the NFS possibilities.

The point here is that your "freedom of choice" depends upon support
pre-existing in the system to enable that choice. That support required
conscious design effort, implementation and testing, all of which
required time and effort from many others collaborating to make it
possible as a feature within an integrated system. It had a very real
cost to implement, and an ongoing cost to maintain. And while you have
the freedom to use unsupported configurations (and I've always
encouraged this for people who wanted to explore the limits of the
possible), that lack of support means you are entirely on your own if it
fails to work correctly or breaks on upgrades. Which is why, in
reality, you are advised to use a supported setup for production use.

Take just one possibility: the choice of using an initramfs or not. The
requirement to support booting directly, without an initramfs, means
duplicating a good bit of functionality between the initramfs and early
boot scripts, since it could be done in either place depending upon your
setup. Stuff like mounting and setting up all the pseudo-filesystems.
This means there is an ongoing maintenance and testing burden to ensure
both scenarios work identically and correctly (and idempotently). Any
change needs duplicating and testing at least twice. Someone (or
several someones) pay the cost for you to have that choice. And some
distributions dropped that choice because the cost was too high for the
small benefit it provided.

A separate /usr is no different. It has a very real cost to support,
with dedicated logic in several packages to enable it, and that cost
requires justification for its existence using objective criteria. It
needs practical concrete use cases to demonstrate a requirement for the
feature. And those needs have to be sufficiently unique and compelling
that they could not be met by alternative, simpler approaches.

The only reason that you can mount a separate /usr today is because of
the work I did 5-6 years ago to mount it in the initramfs. If I hadn't
done that work to keep things working, support would have been dropped 5
years back to allow the use cases which a separate /usr mount broke.
Because the implementation complexity and constraints of the two-phase
late/early boot which /usr mounting forced upon us were causing more far
problems than the benefits they once provided.

The same applies to considering merging (or not merging) / and /usr. It
needs to be driven by a detailed cost/benefit analysis for both
approaches. But once one approach is chosen, it's going to be part of
the fundamental design of the entire Debian system. This makes it
difficult for an individual to chose their own approach--the system
design itself isn't something you can pick and chose. It's something
that will be complied with by every package and enforced by Debian
Policy. It might be some degree of control over the filesystem layout
will be possible, or it might not. We'll have to wait and see.

>>> 1) mounting /usr with different mount options (like barrier, ro,
>>> nodev etc);
>>
>> Could you describe the specific goals of the separation?
>
>   Damn it, again?  Re-read the thread yourself, it's going to take you
> less time than it must have taken writing 58 lines, 1458 words and 8565
> characters of text to justify the necessity of a / -> /usr merge.
>
>> think about the specific problems which you are really trying to
>> solve, rather than tying the solution to this specific mountpoint.
>
>   What? Did you read the Subject line at least?  All this discussion is
> centered on "this specific mountpoint"!

Yes, I did read it all. But you didn't appear to answer the question I
was asking, or understand the rationale for it, which I explained.

What you've provided here are solutions to problems. Using different
mount options is a solution to a problem. But the actual problem you
want to solve hasn't been clearly stated. I'd like you to take a few
steps back away from this solution, and think about the underlying use
cases you have which could be used to detail some requirements.

>> Most of / should be ro+nodev just like for /usr.
>
>   Yeah.  Only, / can't be, but /usr can be.

That's not strictly true; you can have a ro+nodev /.

>> So one deeper question is which bits of / shouldn't be ro/nodev?
>> /etc?  /var?
>
> Everything except /dev, of course.

/dev is a separate filesystem, so this can apply equally to /.

However, both these examples are too specific. As I said above, I'm not
too interested in the specific individual options you might want to use
on / or /usr. I'm more interested, as I said above, in you being able
to articulate a more general set of requirements.

>> Maybe it should be between / and /etc and /var?  Others?
> Are you kidding?  How could you split /etc and /var from /?  Are you
> really saying that keeping /usr split from / makes no sense because you
> can't split /var and /etc from /?

Splitting /var is extremely common. Moreso than /usr in my experience.
/etc is very uncommon. It was in fact just an example, in the interest
of discussion. I'd like to encourage some real thought in where the
ideal points of splitting the filesystem should be, rather than hanging
onto dogma. The content of /etc is different from the content of /bin,
/sbin and /lib (configuration vs binaries). Maybe it's something to
consider. Maybe it's not. Maybe there are other, better options. I'd
like to hear some interesting arguments which are perhaps based upon
categorisation of the filesystem paths by intended usage patterns and
roles, and logic. I'd like a rationalisation of your solution from
first principles.

As an aside: I did actually implement complete support for mounting /etc
in initramfs-tools, initscripts and grub. It was pretty neat. But I
never released it. Not because it didn't work, it worked perfectly, but
because I didn't think that the use case for it (different mount options
and separate filesystem) were sufficiently compelling to introduce the
maintenance burden for such a niche feature. The cost/benefit wasn't
there. But if there is demonstrable demand for such a feature in the
future, it's eminently possible to implement again.

>> I should point out that I wrote a set of patches for mounting /etc in
>> the initramfs as well as /usr, specifically so that you could do this.
>
> First you stated that having a separate /usr is to be avoided because
> it's too troublesome and unmanageable, now you say you split even /etc
> from /?

No. You're drawing unwarranted conclusions from something which was
merely an example.

/usr *as a separately mountable filesystem* is no longer supported for
*direct booting*. It's still supported for *initramfs booting*, but
might be considered deprecated if the usrmerge is going to go ahead.

> Will you keep /usr splittable from /, as it's been the case for decades
> even before Linux existed?

I won't personally do anything. I'm no longer directly involved. I
was, however, one of the people who began the process, and I hope that
after reading that little bit of potted history, you have some
appreciation of the reasons why it was originally thought necessary.
There may be additional reasons today.

>>> 2) having /usr mounted over the network keeping / local;
>> While the initramfs does support both local / and remote /usr (by *my
>> own design and intent*), this is purely to avoid breakage on upgrades.
>> It's not a recommended setup.
> What does not "recommended" entail?

I outlined that above in the two support axes. I'd classify

/ local                fully supported and fully functional
/ remote               fully supported and fully functional
/ local + local /usr   partially supported and fully functional
/ remote + remote /usr partially supported and fully functional
/ local + remote /usr  partially supported and fully functional
/ remote + local /usr  partially supported and fully functional

The latter four combinations are supported by the initramfs. I
explicitly added support for them. But they aren't a recommended
configuration. And they aren't actively tested that I'm aware of (I did
test these actively back in 2012-13, but no longer). So support may end
without notice, or it may break without notice. It's not a possibility
the systemd people care about to support or test the best of my
knowledge, so there's a the possibility of bitrot.

This also demonstrates how a combinatorial explosion of configuration
choices causes a huge maintenance and testing burden. Those six don't
even begin to demonstrate the full complexity here. That's one reason
why limiting choice can be the only way to make this burden manageable.
Otherwise you have choices which are broken in some way without anyone
realising, and that's not very professional.

The support for mounting /usr was added to the initramfs not because it
was a required or desirable feature, but because we didn't want people
upgrading to wheezy to end up with an unbootable system due to their
non-standard configuration choices. It's there primarily for
compatibility with custom setups. Not as a recommended configuration
choice.

> Is moving /home to /var/Drake/shared recommended?  I don't think so.
> Should this be made impossible then?
>
> And is so why?

I fail to understand the question you are asking here.

"Recommended" doesn't just mean "it's a good idea". It means it's a
choice explicitly supported and implemented by the system, and moreover,
it's the optimal choice if more than one is possible.

>> All the cluster nodes
> Here we have it again: the changes that are being pushed have clusters
> as their reference system.

That's not a logical conclusion to draw. The NFS support works for any
system. I set up a debian-live-based compute cluster a few years back.
I've also set up workstations and virtual machines with NFS. None of
these are indicative of a bias towards any particular usage scenario. I
would hope that you might have noticed that I have attempted to be
scrupulously fair and unbiased in my design and implementation choices
over the years, should you care to check. Debian was a "universal"
operating system after all, and many of us took that to heart.

>> The content of that /usr filesystem is under the control of *one* dpkg
>> package database on a single system.  If *any* of the other systems
>> install or remove any package touching /usr (which is all of them!),
>> they will be corrupting the installation.
> This only has sense for cluster installs, not desktop ones.  If what
> you're writing means: "Debian from now on will only cater to the needs
> of BDCs, local desktop customizations will be ignored" well, I'm fine
> with it.  I'll just not go back to Debian, as I do not run a BDC home or
> on my portable devices.

This is an incorrect conclusion. It has nothing to do with cluster
installs. You simply can't share /usr alone. It's a logically and
practically flawed concept. I explained precisely why in detail.

> Anyway, syncing updates of shared filesystems can be done with shared
> mounts.  I thought to do some experiments with it some time ago, but
> then didn't.
>
>   If I am correct, now both / and /usr should be visible under
> /mnt/root.  This way a local update should be propagated to the NFS
> filesystem.

While I'm sure that you can use this to do some neat stuff, the key
considerations (for all setups) are:

- that the state of the entire collection of managed files is kept in
sync with the package database state
- that the entire collection of managed files is visible and accessible

Both of these require sharing the entire filesystem, making any
separation of /usr pointless. Simply share the root, and be done with
it. This is the practical reality of using a package manager. Pop a
writable overlay on top if you want to modify it on a client.

>> Even mounting it read-only is giving you an incomplete view of the
>> installed packages' contents.
>
> Why is that?

Simply that the managed collection of files encompasses much more than
/usr. You're missing out on the contents of /etc, /bin, /lib and /var
amongst others, all of which are tightly coupled to the state in /usr.
They are inseparable. You're missing out on all the init scripts and
configuration files, the variable state and any library and tool
dependencies located on the rootfs. This is why, in practice, sharing
/usr alone is an exercise with little point, and is just asking for trouble.

It's not a sensible way to run a system, not if you don't want it to
randomly break or misbehave.

>> This is not a sensible or robust strategy
> I beg to differ, as it's been done for decades.  It might not make sense
> *on* *a* *cluster*, but allow me not to care about the corporate
> datacenter needs, they are not mine own.

No. It has nothing to do with clusters or workstations. It has
everything to do with this being out of scope for a distribution with an
integrated package manager.

It worked "for decades" when you could load the system from tape. It
doesn't work properly with a package manager. It never has. Yes, you
can set up such an arrangement if wish. But you won't have a robust
environment. It's logistically impossible, totally unsupported and
extremely unsafe.

>>> 4) having the smallest possible / filesystem to ease recovery of a
>>> botched system.
>
>> I'd personally go for a dedicated rescue disk/USB stick/live CD for
>> this specific purpose.
[...]
>> If you can emergency boot a current system with a separate /usr, that
>> is a lucky happenstance.
> I did it dozens of times.  In fact I do it expressly  in my system
> recovery classes: "Now let's see what happens if in /etc/fstab the /usr
> line is commented out and I reboot!".

That's great. But, it's *unsupported* since 5 years back. It might
work, or it might not. It's *unknown and indeterminate*. This setup is
no longer supported, tested or guaranteed to work in any way.

It will potentially break the moment you add a package requiring a file
from /usr in early boot; that's the bit which is no longer supported or
tested. Or it might break due to the dependency ordering; since /usr is
now guaranteed to be available before init(8) starts, there are no
restrictions upon use of /usr at any point during the boot. You use
this unsupported, fragile setup entirely at your own risk.

>>   I hope that the explanations I've given above give you just a little
>> bit of insight into some of the problems for which I have thought
>> quite deeply upon over the course of several years, and spent
>> significant time investigating, implementing and testing.

> All you wrote boils down to: "We're designing Debian for the Data
> Center.  Like it or leave it".
>
> Well, I leave it.

I'm rather confused that you drew this conclusion, because it is
completely unrelated to anything I mentioned. I have personally gone to
great lengths to ensure that we catered for as many different use cases
as practically possible.


Regards,
Roger