Hi All,

as promised, please find below a RFC about the Devuan Build
Infrastructure and the support for user-initiated commits, privilege
separation, and other distributions.

It's a RFC, not a final doc by any means. So please *do* comment on
its content, if you like.

HND

KatolaZ

=======================================================================

# The Devuan build infrastructure (D1build?) - RFC

This document is a Request-For-Comments aiming at describing how the
Devuan build infrastructure (D1build?) works and how it could support
git-triggered builds for Devuan and other distributions.

## Nomenclature

* PKG: the name of a source package, e.g. `surf2`
* PKG-REPO: the name of the git repo of a source package, e.g.
`surf2.git`
* PKG-REPO-URL: the complete URL of the git repo of a source package, e.g.
`https://git.devuan.org/devuan-packages/surf2.git`
* DISTRO: the name of a distribution, all lowercase, e.g.
`devuan`, `heads`, `maemo`.
* SUITE: the section of a distro release, e.g.:
`jessie`, `jessie-proposed`, `ascii-updates`
* PKG-BRANCH: a branch of PKG-REPO identifying the files to be built
for a specific DISTRO and SUITE, e.g.: `suites/jessie`,
`maemo/ascii-proposed`, `heads/ceres`.
* SCORSH-CMD: a JSON-encoded command to be interpreted by SCORSH
* SCORSH-ACTION: one of the actions triggered on the server by a
given SCORSH-CMD
* RELEASEBOT2: a slightly refactored version of devuan-releasebot


## Overview

D1build consists of a git-driven pipeline that supports the creation
and management of packages and repos for different distributions. The
pipeline is based on a few components:

- a naming scheme

- a command authentication and management system (SCORSH)

- a Jenkins CI server


## Naming Scheme

The names of repositories, branches, packages, and build jobs are
meaningful. Each package (PKG) of a distribution that uses D1build is
associated to a git repository (PKG-REPO) which can be cloned at
PKG-REPO-URL. The basename of the PKG-REPO of a package (without the
'.git' extension) **should** match the name of the source package. For
instance, for the `surf2` package we have `PKG=surf2`,
`PKG-REPO=surf2.git` and
`PKG-REPO-URL=https://git.devuan.org/devuan-packages/surf2.git`.

Each distribution is assigned a DISTRO label which defines a
distribution-specific namespace. Also, each distribution will have
their own SUITE labels. Each PKG-BRANCH of PKG-REPO in the format
`DISTRO/SUITE` identifies the distribution and section for which
that version of the package can be built. For instance, the repository

    surf2.git

is associated to the source package for *surf2*. It contains several
branches:

    $ git branch -a
    master
    upstream
    devel
    maemo/ascii
    maemo/ascii-updates
    heads/ascii
    suites/jessie
    suites/ascii-proposed
    suites/ceres

The PKG-BRANCH `maemo/ascii` is meant to be built for `ascii` SUITE
of the `meamo` DISTRO. Similarly, the PKG-BRANCH
`suites/ascii-proposed` is meant to be built for the `ascii-proposed`
SUITE of the `devuan` DISTRO (yes, Devuan retains the right to
allocate DISTRO labels, and to reserve some of them for Devuan, and
the DISTRO label `suites` is an example). In the example above, the
branches `master`, `upstream` and `devel` are not associated to any
DISTRO.


## Command authentication and management (SCORSH)

Package builds can be triggered by means of gpg-signed git commits
containing a SCORSH-CMD in their comment. A SCORSH-CMD is a
JSON-encoded string containing a command name and a list of
arguments. The command name identifies the action that the command
will trigger on the server. A typical example is the command `build`,
that triggers the build of the current branch of the package. Each
SCORSH-CMD is bound to the repo branch where it is issued. Hence, a
`build` SCORSH-CMD issued in the PKG-BRANCH `suites/jessie-proposed`
of the PKG-REPO `surf2.git` will trigger the build of the PKG `surf2`
for the SUITE `jessie-proposed` of the DISTRO `devuan` (remember
that the label `suites` is reserved for Devuan).

SCORSH-CMDs can be included in a commit using the SCORSH git plugin.

On the server-side, each SCORSH-CMD is associated to a list of
keyblocks (ascii-armoured exports of gpg keyrings), which are allowed
to issue that specific command on that specific PKG-REPO:PKG-BRANCH
(PKG-REPO:PKG-BRANCH can also be specified using regexps).

Each SCORSH-CMD is processed on the server side by SCORSH, when the
user pushes to the remote repo. Each SCORSH-CMD is treated differently
depending on the PKG-REPO:PKG-BRANCH from which it was issued.

The set of recognised SCORSH-CMDs for each PKG-REPO:PKG-BRANCH is
completely configurable using a simple JSON configuration file.

If SCORSH recognises a SCORSH-CMD in a git commit, it will check that
the command is avaliable in that PKG-REPO:PKG-BRANCH. If yes, then
SCORSH checks whether the commit was gpg-signed by one of the gpg-keys
which are allowed to run that SCORSH-CMD on that specific
PKG-REPO:PKG-BRANCH. If yes, then SCORSH proceeds with executing all
the pre-configured SCORSH-ACTIONs for that SCORSH-CMD. At the moment,
a SCORCH-ACTION is just the execution of a script. However, the script
associated to a SCORSH-ACTION **must** match the sha256 hash specified
in the SCORSH configuration file. If the hash does not match, the
action is discarded.


### Typical SCORSH-CMDs

We provide in the following a few examples of SCORSH-CMDs which would
be useful to have. These are just examples, since SCORSH is a general
and configurable command authentication system, and can in principle
support any kind of SCORSH-CMD and SCORSH-ACTION (i.e., not only those
related to the Devuan build system).

The examples below match the commands currently available in
releasebot.

#### build

The SCORSH-CMD `build` will just trigger the build of the package on
Jenkins. The name of the Jenkins job to be run is obtained from the
PKG-REPO:PKG-BRANCH in which the SCORSH-CMD was issued. For instance,
a `build` SCORSH-CMD on the PKG-BRANCH
`suites/jessie-proposed-updates` of the PKG-REPO `surf2.git` will
attempt to trigger the build of `surf2` for the SUITE
`jessie-proposed-updates` of `devuan`. To trigger the build,
RELEASEBOT2 uses the Python Jenkins plugin, asking for the execution
of the Jenkins job `DISTRO:PKG-sources`. Please see the section
[Jenkins][] below. Devuan reserves the right to use Jenkins builds of
the format `PKG-sources` (i.e., without any `DISTRO` qualifier).

#### `buildadd`

The SCORSH-CMD `buildadd` will create all the Jenkins jobs needed to
build a specific package, if they don't exist already. For instance, a
`buildadd` SCORSH-CMD on the PKG-BRANCH `maemo/ascii` of the PKG-REPO
`surf2.git` will attempt to create all the Jenkins jobs to build
`surf2` for `maemo`. To create the jobs, RELEASEBOT2 will use the set
of XML templates associated to the DISTRO `maemo`. The new Jenkins
jobs will be named `maemo:surf2-sources`, `maemo:surf2-binaries`,
`maemo:surf2-repos`.

It is important to note that, in general, only a small number of
administrators should be allowed to use `buildadd`. This will be
achieved by using a specific **distinct** keyblock for the `buildadd`
command.

#### `buildmodify`

This SCORSH-CMD allows to modify the parameters of the Jenkins jobs of
a PKG. For instance, it is possible to change the set of architectures
for which the package is built.

It is important to note that, in general, only a small number of
administrators should be allowed to use `buildmodify`. This will be
achieved by using a specific **distinct** keyblock for the
`buildmodify` command.


#### `builddelete`

This SCORSH-CMD removes all the Jenkins jobs associated to a certain
PKG.

It is important to note that, in general, only a small number of
administrators should be allowed to use `builddelete`. This will be
achieved by using a specific **distinct** keyblock for the
`builddelete` command.


## Jenkins

The Jenkins installation at [ci.devuan.org][ci.devuan.org] maintains a
set of three jobs for each PKG to be built for a specific DISTRO,
namely:

* [DISTRO:]PKG-sources
* [DISTRO:]PKG-binaries
* [DISTRO:]PKG-repos

These jobs are chained, so that the completion of PKG-sources triggers
the execution of PKG-binaries, and the completion of PKG-binaries
triggers the execution of PKG-repos. The DISTRO prefix is **optional**
for Devuan packages, but it is **mandatory** for packages belongin to
other distributions.



### Package jobs

* [DISTRO:]PKG-sources: this job clones a specific PKG-BRANCH of
PKG-REPO. The PKG-BRANCH to be cloned is specified as a parameter by
RELEASEBOT2, and is obtained from the PKG-BRANCH from which the
`build` SCORSH-CMD was issued. (This replaces the usage of Gitlab
issue labels to identify the corect branch to be cloned). When it
finishes, this job triggers [DISTRO:]PKG-binaries

* [DISTRO:]PKG-binaries: this job builds the package PKG using
cow-builder/pbuilder, and creates all the .deb packages and the
associated .dsc files. The parameters of the build are taken from
the XML configuration snippet. When it finishes, this job triggers
[DISTRO:]PKG-repos

* [DISTRO:]PKG-repos: this job is responsible for copying all the .deb
files of PKG to the repo machine, and for triggering any action
related to repo management, i.e. invoking `dak` or `reprepro` to
update the repos.

### Job templates

Each DISTRO is associated to a set of XML Jenkins job templates, which
contain the configuration of the three Jenkins jobs associated to each
PKG. These XML templates can be customised to account for
DISTRO-specific needs, including the specific commands to be run in
order to update the corresponding repo.

=======================================================================

-- 
[ ~.,_  Enzo Nicosia aka KatolaZ - GLUGCT -- Freaknet Medialab  ]  
[     "+.  katolaz [at] freaknet.org --- katolaz [at] yahoo.it  ]
[       @)   http://kalos.mine.nu ---  Devuan GNU + Linux User  ]
[     @@)  http://maths.qmul.ac.uk/~vnicosia --  GPG: 0B5F062F  ] 
[ (@@@)  Twitter: @KatolaZ - skype: katolaz -- github: KatolaZ  ]