:: [devuan-dev] Luminously Unparallele…
Top Page
Delete this message
Reply to this message
Author: Mason Loring Bliss
To: devuan developers internal list
Subject: [devuan-dev] Luminously Unparalleled Repository Coalescer design doc
lurc - the Luminously Unparalleled Repository Coalescer
Initial design document

Single tool with a collection of single-shot functions, each invoked
separately and as separate processes, but with batched versions that invoke
copies of the tool with the correct arguments, in series. Goal: Be able to
(re-)run each individual piece independently.

Individual operations will assert kernel advisory locks (via fcntl) to
guarantee coherency during processing. (Id est, no new pulls during a
merge, no new merges during a pull, with a configurable timeout.)

Will have a --force flag or similar to force re-merging of data sets in the
absense of new data, which will be flagged and recorded during download
attempts. Will also use --force to insist on redownloading evidently-
unchanged datasets. (Freshness trusts HTTP headers.)

Configuration syntax will be simple flat text. Blacklisted packages will be

TBD: Syntax/semantics for specifying precedence amongst repositories.
(Example, department > organization > devuan > debian-security > debian,
with each step potentially asserting blacklists.) Current favourite: linked
list specified in config? Guard against loops or any dist being masked more
than once, or directly masking more than one subordinate dist.

1. Pull down repo data from all specified repositories. (Invocation of tool
can specify a single dist to pull or a batch mode that calls the tool to
collect all repositories. For single-repository mode only, a --force option
will allow re-pull even for files that look unchanged.)

    a. config will specify dist locations

    b. config will have a suite mapping

    c. snag each relevant Packages, Release, Contents file

        TBD: What's the minimal set of files I need to regenerate, beyond
        the Packages files?

2. Write out merged data where

    a. higher-precedence packages mask lower-precedence packages and
    blacklists. (Examples, local apt built without libsystemd0, local
    Plymouth built without systemd deps, local dist blacklists libsystemd0
    and pulseaudio.)

    b. Packages are blacklisted per-dist, with each level offering a
    blacklist of packages from that level or in subordinate dists.

    c. Per-dist blacklisting is applied with each successive application of
    a dist, from lowest-precedence to highest. As such, if a higher-rank
    repository supplied a package blacklisted below, it will appear in the
    final results, unless a still-higher-priority dist again blacklists it.

3. Sign.

4. Publish data. We want to be *really* atomic, and not have network
latency impact this, so:

    a. rsync the produced merge to holding directory on destination

    b. Once on pkgmaster, rsync into place - more likely to be atomic

        TBD: but consider better guarantees? Either way, this is outside
        of the scope of lurc and merely a suggestion.
Open questions:

1. What logging detail do we want? Question listed in weekly meeting doc.
Config data:

set of repositories with dist keys (fields: repo <key> <url>)
map of overlays (fields: map <dist> <subordinate>)

Blacklist data per-dist in /etc/lurc/blacklist.d.
Method to merge data:

1. In-memory map of most-subordinate remaining set.

2. Apply blacklist.

3. Overlay next-most-subordinate set atop initial data, apply blacklist.

4. Write out remaining dataset to file. Preserve deb822(5). (Consider
formal use of deb822 for configs?)

Packaged dependencies so far: libhttp-tinyish-perl

Modification status: Return code is 200 or 2xx if new, 304 if unmodified

Config in /etc/lurc
Work in /var/spool/lurc
role user: turkey (why? because sudo turkey lurc)

todo: provide bash-completion

todo: perldoc as base for docco

Mason Loring Bliss (( If I have not seen as far as others, it is because
mason@??? )) giants were standing on my shoulders. - Hal Abelson