On 2025-10-23 09:55:03, sdomi wrote:
> On 2025-10-23 08:25, onefang wrote:
> > On 2025-10-23 05:59:51, Shengqi Chen wrote:
> > > Hi,
> > >
> > > > -----Original Message-----
> > > > From: devuan-mirrors <devuan-mirrors-bounces@???> On Behalf Of
> > > > onefang
> > > > Sent: Thursday, October 23, 2025 1:11 PM
> > > >
> > > > G'day, Devuan's package mirror herder here.
> > > >
> > > > I've been working on a new update script for Devuan package mirrors.
> > > >
> > > > https://sledjhamr.org/cgit/devuan_package_mirror_sync/
> > > >
> > > > Which tries to update a Devuan package mirror with minimal issues. It
> > > > should help prevent "this mirror is currently updating and might give
> > > > errors", or at least shrink that window down to fractions of a second.
> > >
> > > It looks rather like Debian's two-stage rsync used in ftpsync.
> >
> > I've looked at ftpsync, it's much more complicated than what I think we
> > need, but yes a two stage rsync.
>
> Seconding that. Debian's convoluted mirrorscript (and their insistence that
> everyone needs to use it) is one of the top reasons why I don't run a Debian
> mirror alongside Devuan. As long as something boils down to one (... two?)
> rsync calls, I don't buy that we need to migrate to a "real" programming
> language. Let's not over-engineer :)
>
> > > And we (with my TUNA mirror's admin hat on) have also implemented
> > > our own version.
> >
> > Go isn't one of my 100 programming languages, and I still haven't
> > decided
> > what language 101 will be.
> >
> > > I suppose devuan does not have much deviation with Debian's folder
> > > structure,
> > > so we could still re-use those mechanisms safely?
> >
> > We could, but I'm trying to simplify and speed up things.
> >
> > > Thanks,
> > > Shengqi Chen
>
> As likely every other mirror operator, I have my own bespoke script for the
> purpose - which did already include --delay-updates, and --delete-after. Or
> is the new script different in some other way?
>
> As an aside: I don't like how we're calling rsync twice, and I don't fully
> understand the intent (especially with --delete-delay only being in one of
> those calls). Care to elaborate about the source of the problem? Because to
> me, this sounds like core mirror doesn't get updated atomically, and this is
> a workaround? I'm confused.

The main problem is symlinks. The various metadata files are sometimes
symlinks, but rsync refuses to create the file the link points to before
it creates the link. So the first one excludes the metadata files, and
the delete is in the second one to delete things when everything is
settled.

I tried to reduce it to a single rsync, but it just wasn't gonna happen.
If anyone can do that, let us know.