lurc - the Luminously Unparalleled Repository Coalescer
Initial design document
---------------------------------------------------------------------------
Overview:

Single tool with a collection of single-shot functions, each invoked
separately and as separate processes, but with batched versions that invoke
copies of the tool with the correct arguments, in series. Goal: Be able to
(re-)run each individual piece independently.

Individual operations will assert kernel advisory locks (via fcntl) to
guarantee coherency during processing. (Id est, no new pulls during a
merge, no new merges during a pull, with a configurable timeout.)

Will have a --force flag or similar to force re-merging of data sets in the
absense of new data, which will be flagged and recorded during download
attempts. Will also use --force to insist on redownloading evidently-
unchanged datasets. (Freshness trusts HTTP headers.)

Configuration syntax will be simple flat text. Blacklisted packages will be
per-dist.

TBD: Syntax/semantics for specifying precedence amongst repositories.
(Example, department > organization > devuan > debian-security > debian,
with each step potentially asserting blacklists.) Current favourite: linked
list specified in config? Guard against loops or any dist being masked more
than once, or directly masking more than one subordinate dist.
---------------------------------------------------------------------------
Procedure:

1. Pull down repo data from all specified repositories. (Invocation of tool
can specify a single dist to pull or a batch mode that calls the tool to
collect all repositories. For single-repository mode only, a --force option
will allow re-pull even for files that look unchanged.)

    a. config will specify dist locations

    b. config will have a suite mapping

    c. snag each relevant Packages, Release, Contents file

        TBD: What's the minimal set of files I need to regenerate, beyond
        the Packages files?

2. Write out merged data where

    a. higher-precedence packages mask lower-precedence packages and
    blacklists. (Examples, local apt built without libsystemd0, local
    Plymouth built without systemd deps, local dist blacklists libsystemd0
    and pulseaudio.)

    b. Packages are blacklisted per-dist, with each level offering a
    blacklist of packages from that level or in subordinate dists.

    c. Per-dist blacklisting is applied with each successive application of
    a dist, from lowest-precedence to highest. As such, if a higher-rank
    repository supplied a package blacklisted below, it will appear in the
    final results, unless a still-higher-priority dist again blacklists it.

3. Sign.

4. Publish data. We want to be *really* atomic, and not have network
latency impact this, so:

    a. rsync the produced merge to holding directory on destination
    (pkgmaster)

    b. Once on pkgmaster, rsync into place - more likely to be atomic

        TBD: but consider better guarantees? Either way, this is outside
        of the scope of lurc and merely a suggestion.
---------------------------------------------------------------------------
Open questions:

1. What logging detail do we want? Question listed in weekly meeting doc.
---------------------------------------------------------------------------
Config data:

set of repositories with dist keys (fields: repo <key> <url>)
map of overlays (fields: map <dist> <subordinate>)

Blacklist data per-dist in /etc/lurc/blacklist.d.
---------------------------------------------------------------------------
Method to merge data:

1. In-memory map of most-subordinate remaining set.

2. Apply blacklist.

3. Overlay next-most-subordinate set atop initial data, apply blacklist.
Loop.

4. Write out remaining dataset to file. Preserve deb822(5). (Consider
formal use of deb822 for configs?)
---------------------------------------------------------------------------
Details/notes:

Packaged dependencies so far: libhttp-tinyish-perl

Modification status: Return code is 200 or 2xx if new, 304 if unmodified

Config in /etc/lurc
Work in /var/spool/lurc
role user: turkey (why? because sudo turkey lurc)

todo: provide bash-completion

todo: perldoc as base for docco
---------------------------------------------------------------------------

--
Mason Loring Bliss (( If I have not seen as far as others, it is because
mason@??? )) giants were standing on my shoulders. - Hal Abelson