lurc - the Luminously Unparalleled Repository Coalescer
Initial design document
---------------------------------------------------------------------------
Overview:
Single tool with a collection of single-shot functions, each invoked
separately and as separate processes, but with batched versions that invoke
copies of the tool with the correct arguments, in series. Goal: Be able to
(re-)run each individual piece independently.
Individual operations will assert kernel advisory locks (via fcntl) to
guarantee coherency during processing. (Id est, no new pulls during a
merge, no new merges during a pull, with a configurable timeout.)
Will have a --force flag or similar to force re-merging of data sets in the
absense of new data, which will be flagged and recorded during download
attempts. Will also use --force to insist on redownloading evidently-
unchanged datasets. (Freshness trusts HTTP headers.)
Configuration syntax will be simple flat text. Blacklisted packages will be
per-dist.
TBD: Syntax/semantics for specifying precedence amongst repositories.
(Example, department > organization > devuan > debian-security > debian,
with each step potentially asserting blacklists.) Current favourite: linked
list specified in config? Guard against loops or any dist being masked more
than once, or directly masking more than one subordinate dist.
---------------------------------------------------------------------------
Procedure:
1. Pull down repo data from all specified repositories. (Invocation of tool
can specify a single dist to pull or a batch mode that calls the tool to
collect all repositories. For single-repository mode only, a --force option
will allow re-pull even for files that look unchanged.)
a. config will specify dist locations
b. config will have a suite mapping
c. snag each relevant Packages, Release, Contents file
TBD: What's the minimal set of files I need to regenerate, beyond
the Packages files?
2. Write out merged data where
a. higher-precedence packages mask lower-precedence packages and
blacklists. (Examples, local apt built without libsystemd0, local
Plymouth built without systemd deps, local dist blacklists libsystemd0
and pulseaudio.)
b. Packages are blacklisted per-dist, with each level offering a
blacklist of packages from that level or in subordinate dists.
c. Per-dist blacklisting is applied with each successive application of
a dist, from lowest-precedence to highest. As such, if a higher-rank
repository supplied a package blacklisted below, it will appear in the
final results, unless a still-higher-priority dist again blacklists it.
3. Sign.
4. Publish data. We want to be *really* atomic, and not have network
latency impact this, so:
a. rsync the produced merge to holding directory on destination
(pkgmaster)
b. Once on pkgmaster, rsync into place - more likely to be atomic
TBD: but consider better guarantees? Either way, this is outside
of the scope of lurc and merely a suggestion.
---------------------------------------------------------------------------
Open questions:
1. What logging detail do we want? Question listed in weekly meeting doc.
---------------------------------------------------------------------------
Config data:
set of repositories with dist keys (fields: repo <key> <url>)
map of overlays (fields: map <dist> <subordinate>)
Blacklist data per-dist in /etc/lurc/blacklist.d.
---------------------------------------------------------------------------
Method to merge data:
1. In-memory map of most-subordinate remaining set.
2. Apply blacklist.
3. Overlay next-most-subordinate set atop initial data, apply blacklist.
Loop.
4. Write out remaining dataset to file. Preserve deb822(5). (Consider
formal use of deb822 for configs?)
---------------------------------------------------------------------------
Details/notes:
Packaged dependencies so far: libhttp-tinyish-perl
Modification status: Return code is 200 or 2xx if new, 304 if unmodified
Config in /etc/lurc
Work in /var/spool/lurc
role user: turkey (why? because sudo turkey lurc)
todo: provide bash-completion
todo: perldoc as base for docco
---------------------------------------------------------------------------
--
Mason Loring Bliss (( If I have not seen as far as others, it is because
mason@??? )) giants were standing on my shoulders. - Hal Abelson