In last meeting, I noticed apt uses SRV records in DNS
(https://tools.ietf.org/html/rfc2782) in order to decide which
repository mirror to contact and promised to take a look at it.

Sorry for the length, but it is both interesting and has a lot of
relevant details.

So, here are my findings:


The main documentation for this behaviour appears to be here:
https://salsa.debian.org/apt-team/apt/blob/master/doc/srv-records-support.md

From RFC2782:

The SRV RR allows administrators to use several servers for a single
domain, to move services from host to host with little fuss, and to
designate some hosts as primary servers for a service and others as
backups.

The TL;DR of SRV records is: for $SERVICE (say, http or https) offered
under $DOMAIN (say, deb.de{vu,bi}an.org), a client queries DNS for SRV
entries for _$SERVICE._(tcp|udp).$DOMAIN
This returns a list of *domain names* that do not have to be related to
$DOMAIN. The only limitation of those domain names is that they cannot
be CNAMES.

Since that RFC is for any generic $SERVICE, it does not specify how a
client is to proceed with the connection, just that those are the hosts.
Specifically for HTTP(S), it is quite relevant what happens with the
"Host" header:
Option 1: Host contains domain name in the SRV response. This would
allow off-loading of TLS certificate issuance to mirror hosters.
Option 2: Host stays the same as in original request ($DOMAIN). This
kind of makes mores sense though as you want to connect to $DOMAIN.

Turns out, there is an RFC draft for that:
https://tools.ietf.org/html/draft-andrews-http-srv-02

The use of a SRV record does not affect the contents of the "Host:"
field of the HTTP transaction. Its only effect is to potentially
change the address and port the client connects to. All other parts
of the HTTP transaction are not affected by the presence of a SRV
record.

So, Option2. This is exactly what apt is doing, that means TLS remains
an issue.

The only thing that apt is *not* doing is respecting weights as defined
in the RFC (it does respect priorities though).


So, does this mean that SRV entries are pointless and we shouldn't look
into it?
Not really, this actually has the potential to simplify mirror
management a bit:

Right now we are managing IP addresses that are listed as A entries for
deb.devuan.org.
This means, those entries are used for *everything* that tries to
contact deb.devuan.org.

What we could be doing:
* Managing a bunch of *hostnames* that handle HTTP requests to
deb.devuan.org properly, by having those in _http._tcp.deb.devuan.org
* Managing a handful of devuan-controlled servers that deal with HTTPS
listed either as SRV (under _https._tcp.deb.devuan.org) or as A/AAAA
records.


Indeed, I think deb.debian.org is doing this:

# dig SRV _http._tcp.deb.debian.org

;; ANSWER SECTION:
_http._tcp.deb.debian.org. 264  IN      SRV     10 1 80
prod.debian.map.fastly.net.

# dig SRV _https._tcp.deb.debian.org
* no answer *

# host deb.debian.org
deb.debian.org is an alias for static.debian.org.
static.debian.org has address 128.31.0.62
static.debian.org has address 149.20.4.15
static.debian.org has address 5.153.231.4
static.debian.org has address 130.89.148.14
static.debian.org has IPv6 address 2001:4f8:1:c::15
static.debian.org has IPv6 address 2001:67c:2564:a119::148:14
static.debian.org has IPv6 address 2001:41c8:1000:21::21:4

This translates to: they are using fastly for HTTP and using
"debian-infra" (likely just debian-managed and actually hosted at MIT,
and others) for the actual deb.debian.org host entries, which deal with
HTTPS.
However, they are using 302 redirects after the TLS connection has been
established:

# curl -I https://deb.debian.org/debian-security/
HTTP/1.1 302 Found
Location: https://cdn-aws.deb.debian.org/debian-security/

This approach has the drawbacks we have discussed already (apt makes a
TLS connection to deb.de{vu,bi}an.org each time then follows the redirect).


Even if we don't want to officially support HTTPS RR at the moment, an
advantage of only managing SRV records, which use hostnames instead of
IPs, would be that IPv6 support for the mirror RR system is pretty
immediate and does not require us or our mirrors to touch anything
(besides DNS, but only on our side and it's still at least half the
entries because we don't need both A and AAAA records but only one
hostname in the SRV).
Also, if any mirror decides or needs to use multiple hosts under the
same name and RR at DNS level themselves or add IPv6 or change their IP,
this is fully transparent to us.


Below you will find the relation of IPv6 support, IPv4 address and PTR
(which can and should be used in SRV records) for all mirrors in the RR.
I also manually checked that the IPv6 entries are properly set up
(respond correctly to requests for deb.devuan.org); this should be
automatic, once I set up my upcoming mirror checks.

IPv6 IPv4             PTR

# BR
OK   200.236.31.1     sagres.c3sl.ufpr.br

# CH
OK 185.203.112.44 mirror.datacenterlight.ch

# DE
OK   46.4.50.2        mirror.checkdomain.de
OK   141.24.220.40    dist-mirror.fem.tu-ilmenau.de

# DK
OK 130.225.254.116 mirrors.dotsrc.org

# NL
OK   31.220.0.151     mirror.koddos.net
OK   37.220.36.58     sledjhamr.org

# UA
OK   95.216.15.86     megumin.packet-gain.de

# PTR OK, but no IPv6
# FR
     5.196.38.18      ip18.ip-5-196-38.eu
     37.187.111.86    ns327660.ip-37-187-111.eu
     185.183.113.129  mirror.vpgrp.io

# Wrong PTR, IPv6 unknown
# FR
??   91.121.196.103   Wrong: PTR exists, does not resolve back.
# IT
??   185.26.197.8     Wrong: PTR exists, but it is an NXDOMAIN.
# BG
??   195.85.215.180   Wrong: PTR exists, but it is an NXDOMAIN.

Notice that there are 6 mirrors that appear to not support IPv6, that's
fine.
For the 3 that have valid PTRs that resolve back to the IPv4, we can ues
those PTRs, for the other 3, we can always create A entries on
devuan-controlled domains, so that we can include them in the SRVs
(although, ideally they'd fix their thing :-D).

FWIW, I setup for testing a couple servers as reverse proxies for
deb.devuan.org and added the necessary SRV entries for HTTP to a domain
I control and it worked flawlessly.


Another thing I've noticed, is that we probably added a *.deb.devuan.org
CNAME that resolves to pkgmaster.devuan.org.

I know we plan on further clustering mirrors under localised subdomains
but: wouldn't it be better to have that CNAME be to deb.devuan.org for
the time being?
Right now anyone using a localised CC.deb.devuan.org entry is hitting
pkgmaster (can't check traffic / state since I don't have access there).

This reminds me: nginx probably needs a kick on pkgmaster and
auto.mirror, TLS certs expire in a few days; that should be fixed so we
both don't get alerts and don't risk the certs actually expiring.
--
Evilham