Hi, I've chucked this RFC up in gitlab and commented on it there.

https://git.devuan.org/devuan-infrastructure/infrastructure_doc/commit/fe95a4715e26c612f90da510727f07371c327fd8

My thoughts are we should rework it to describe the current setup, give
an overview of the proposed solutions and define the actual issues in
detail and how they might be solved in the various implementation
approaches. At least this way we'd get a clear understanding of the
problem space and be able to weigh the solutions on a problem by problem
basis.

Cheers,
    Daniel

On 26/07/17 05:46, KatolaZ wrote:
> Hi All,
>
> as promised, please find below a RFC about the Devuan Build
> Infrastructure and the support for user-initiated commits, privilege
> separation, and other distributions.
>
> It's a RFC, not a final doc by any means. So please *do* comment on
> its content, if you like.
>
> HND
>
> KatolaZ
>
> =======================================================================
>
> # The Devuan build infrastructure (D1build?) - RFC
>
> This document is a Request-For-Comments aiming at describing how the
> Devuan build infrastructure (D1build?) works and how it could support
> git-triggered builds for Devuan and other distributions.
>
> ## Nomenclature
>
> * PKG: the name of a source package, e.g. `surf2`
> * PKG-REPO: the name of the git repo of a source package, e.g.
> `surf2.git`
> * PKG-REPO-URL: the complete URL of the git repo of a source package, e.g.
> `https://git.devuan.org/devuan-packages/surf2.git`
> * DISTRO: the name of a distribution, all lowercase, e.g.
> `devuan`, `heads`, `maemo`.
> * SUITE: the section of a distro release, e.g.:
> `jessie`, `jessie-proposed`, `ascii-updates`
> * PKG-BRANCH: a branch of PKG-REPO identifying the files to be built
> for a specific DISTRO and SUITE, e.g.: `suites/jessie`,
> `maemo/ascii-proposed`, `heads/ceres`.
> * SCORSH-CMD: a JSON-encoded command to be interpreted by SCORSH
> * SCORSH-ACTION: one of the actions triggered on the server by a
> given SCORSH-CMD
> * RELEASEBOT2: a slightly refactored version of devuan-releasebot
>
>
> ## Overview
>
> D1build consists of a git-driven pipeline that supports the creation
> and management of packages and repos for different distributions. The
> pipeline is based on a few components:
>
> - a naming scheme
>
> - a command authentication and management system (SCORSH)
>
> - a Jenkins CI server
>
>
> ## Naming Scheme
>
> The names of repositories, branches, packages, and build jobs are
> meaningful. Each package (PKG) of a distribution that uses D1build is
> associated to a git repository (PKG-REPO) which can be cloned at
> PKG-REPO-URL. The basename of the PKG-REPO of a package (without the
> '.git' extension) **should** match the name of the source package. For
> instance, for the `surf2` package we have `PKG=surf2`,
> `PKG-REPO=surf2.git` and
> `PKG-REPO-URL=https://git.devuan.org/devuan-packages/surf2.git`.
>
> Each distribution is assigned a DISTRO label which defines a
> distribution-specific namespace. Also, each distribution will have
> their own SUITE labels. Each PKG-BRANCH of PKG-REPO in the format
> `DISTRO/SUITE` identifies the distribution and section for which
> that version of the package can be built. For instance, the repository
>

>     surf2.git

>
> is associated to the source package for *surf2*. It contains several
> branches:
>

>     $ git branch -a
>     master
>     upstream
>     devel
>     maemo/ascii
>     maemo/ascii-updates
>     heads/ascii
>     suites/jessie
>     suites/ascii-proposed
>     suites/ceres

>
> The PKG-BRANCH `maemo/ascii` is meant to be built for `ascii` SUITE
> of the `meamo` DISTRO. Similarly, the PKG-BRANCH
> `suites/ascii-proposed` is meant to be built for the `ascii-proposed`
> SUITE of the `devuan` DISTRO (yes, Devuan retains the right to
> allocate DISTRO labels, and to reserve some of them for Devuan, and
> the DISTRO label `suites` is an example). In the example above, the
> branches `master`, `upstream` and `devel` are not associated to any
> DISTRO.
>
>
> ## Command authentication and management (SCORSH)
>
> Package builds can be triggered by means of gpg-signed git commits
> containing a SCORSH-CMD in their comment. A SCORSH-CMD is a
> JSON-encoded string containing a command name and a list of
> arguments. The command name identifies the action that the command
> will trigger on the server. A typical example is the command `build`,
> that triggers the build of the current branch of the package. Each
> SCORSH-CMD is bound to the repo branch where it is issued. Hence, a
> `build` SCORSH-CMD issued in the PKG-BRANCH `suites/jessie-proposed`
> of the PKG-REPO `surf2.git` will trigger the build of the PKG `surf2`
> for the SUITE `jessie-proposed` of the DISTRO `devuan` (remember
> that the label `suites` is reserved for Devuan).
>
> SCORSH-CMDs can be included in a commit using the SCORSH git plugin.
>
> On the server-side, each SCORSH-CMD is associated to a list of
> keyblocks (ascii-armoured exports of gpg keyrings), which are allowed
> to issue that specific command on that specific PKG-REPO:PKG-BRANCH
> (PKG-REPO:PKG-BRANCH can also be specified using regexps).
>
> Each SCORSH-CMD is processed on the server side by SCORSH, when the
> user pushes to the remote repo. Each SCORSH-CMD is treated differently
> depending on the PKG-REPO:PKG-BRANCH from which it was issued.
>
> The set of recognised SCORSH-CMDs for each PKG-REPO:PKG-BRANCH is
> completely configurable using a simple JSON configuration file.
>
> If SCORSH recognises a SCORSH-CMD in a git commit, it will check that
> the command is avaliable in that PKG-REPO:PKG-BRANCH. If yes, then
> SCORSH checks whether the commit was gpg-signed by one of the gpg-keys
> which are allowed to run that SCORSH-CMD on that specific
> PKG-REPO:PKG-BRANCH. If yes, then SCORSH proceeds with executing all
> the pre-configured SCORSH-ACTIONs for that SCORSH-CMD. At the moment,
> a SCORCH-ACTION is just the execution of a script. However, the script
> associated to a SCORSH-ACTION **must** match the sha256 hash specified
> in the SCORSH configuration file. If the hash does not match, the
> action is discarded.
>
>
> ### Typical SCORSH-CMDs
>
> We provide in the following a few examples of SCORSH-CMDs which would
> be useful to have. These are just examples, since SCORSH is a general
> and configurable command authentication system, and can in principle
> support any kind of SCORSH-CMD and SCORSH-ACTION (i.e., not only those
> related to the Devuan build system).
>
> The examples below match the commands currently available in
> releasebot.
>
> #### build
>
> The SCORSH-CMD `build` will just trigger the build of the package on
> Jenkins. The name of the Jenkins job to be run is obtained from the
> PKG-REPO:PKG-BRANCH in which the SCORSH-CMD was issued. For instance,
> a `build` SCORSH-CMD on the PKG-BRANCH
> `suites/jessie-proposed-updates` of the PKG-REPO `surf2.git` will
> attempt to trigger the build of `surf2` for the SUITE
> `jessie-proposed-updates` of `devuan`. To trigger the build,
> RELEASEBOT2 uses the Python Jenkins plugin, asking for the execution
> of the Jenkins job `DISTRO:PKG-sources`. Please see the section
> [Jenkins][] below. Devuan reserves the right to use Jenkins builds of
> the format `PKG-sources` (i.e., without any `DISTRO` qualifier).
>
> #### `buildadd`
>
> The SCORSH-CMD `buildadd` will create all the Jenkins jobs needed to
> build a specific package, if they don't exist already. For instance, a
> `buildadd` SCORSH-CMD on the PKG-BRANCH `maemo/ascii` of the PKG-REPO
> `surf2.git` will attempt to create all the Jenkins jobs to build
> `surf2` for `maemo`. To create the jobs, RELEASEBOT2 will use the set
> of XML templates associated to the DISTRO `maemo`. The new Jenkins
> jobs will be named `maemo:surf2-sources`, `maemo:surf2-binaries`,
> `maemo:surf2-repos`.
>
> It is important to note that, in general, only a small number of
> administrators should be allowed to use `buildadd`. This will be
> achieved by using a specific **distinct** keyblock for the `buildadd`
> command.
>
> #### `buildmodify`
>
> This SCORSH-CMD allows to modify the parameters of the Jenkins jobs of
> a PKG. For instance, it is possible to change the set of architectures
> for which the package is built.
>
> It is important to note that, in general, only a small number of
> administrators should be allowed to use `buildmodify`. This will be
> achieved by using a specific **distinct** keyblock for the
> `buildmodify` command.
>
>
> #### `builddelete`
>
> This SCORSH-CMD removes all the Jenkins jobs associated to a certain
> PKG.
>
> It is important to note that, in general, only a small number of
> administrators should be allowed to use `builddelete`. This will be
> achieved by using a specific **distinct** keyblock for the
> `builddelete` command.
>
>
> ## Jenkins
>
> The Jenkins installation at [ci.devuan.org][ci.devuan.org] maintains a
> set of three jobs for each PKG to be built for a specific DISTRO,
> namely:
>
> * [DISTRO:]PKG-sources
> * [DISTRO:]PKG-binaries
> * [DISTRO:]PKG-repos
>
> These jobs are chained, so that the completion of PKG-sources triggers
> the execution of PKG-binaries, and the completion of PKG-binaries
> triggers the execution of PKG-repos. The DISTRO prefix is **optional**
> for Devuan packages, but it is **mandatory** for packages belongin to
> other distributions.
>
>
>
> ### Package jobs
>
> * [DISTRO:]PKG-sources: this job clones a specific PKG-BRANCH of
> PKG-REPO. The PKG-BRANCH to be cloned is specified as a parameter by
> RELEASEBOT2, and is obtained from the PKG-BRANCH from which the
> `build` SCORSH-CMD was issued. (This replaces the usage of Gitlab
> issue labels to identify the corect branch to be cloned). When it
> finishes, this job triggers [DISTRO:]PKG-binaries
>
> * [DISTRO:]PKG-binaries: this job builds the package PKG using
> cow-builder/pbuilder, and creates all the .deb packages and the
> associated .dsc files. The parameters of the build are taken from
> the XML configuration snippet. When it finishes, this job triggers
> [DISTRO:]PKG-repos
>
> * [DISTRO:]PKG-repos: this job is responsible for copying all the .deb
> files of PKG to the repo machine, and for triggering any action
> related to repo management, i.e. invoking `dak` or `reprepro` to
> update the repos.
>
> ### Job templates
>
> Each DISTRO is associated to a set of XML Jenkins job templates, which
> contain the configuration of the three Jenkins jobs associated to each
> PKG. These XML templates can be customised to account for
> DISTRO-specific needs, including the specific commands to be run in
> order to update the corresponding repo.
>
> =======================================================================
>
>
>
> _______________________________________________
> devuan-dev internal mailing list
> devuan-dev@???
> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/devuan-dev
>


--
Daniel Reurich
Centurion Computer Technology (2005) Ltd.
021 797 722