On Thu, Nov 19, 2020 at 11:16:11AM +0100, Ivan J. wrote:

> Also "with a configurable timeout" sounds wrong. Instead, I would
> implement proper error handling and cleanup upon error.

That's one of the things noted about Amprolla - it wasn't noticing
connections hanging. The only way to address this is to assert a timeout.

That said, the basic plan for lurc is to have many small parts. As such a
timeout or error will be a failure of a subpart, and dependent parts will
understand that a depenendency has failed. The error will be logged, and
monitoring should notice errors and alert folks who can have a look. They
can either re-run the individual piece that broke or await the next overall
run.

That said, the notion of clean-up is good, and I'll test download failures
mid-flight. I can download into a staging error and only move files into
the working area if they pass muster.


> I recommend parsing Release files rather than trusting HTTP headers,

We want to trust HTTP headers for automated download, or we'll be
downloading large files constantly, which is far too heavy. The --force
flag will be a way around this should the stars converge such that a new
upstream file somehow doesn't have a changed modification header, but this
should be quite rare.


> Keep in mind not only Source/Package names should be blacklisted, but
> also other packages in which their names appear in the dependencies.

Thank you. Excellent point. I need to explicate that in the SDD.


> Unfortunately transferring files like this will never truly be atomic.

See LeePen's by-hash notes, below. (New to me, but it sounds just right.)


> > Modification status: Return code is 200 or 2xx if new, 304 if unmodified
>
> Could you explain this?

Sure:

    https://tools.ietf.org/html/rfc7231 section 6.4, Redirection 3xx

If we get a 304, we've already got that file. If we see a 2xx, we've got a
version we didn't have previously. Other results, we had an error.


On Thu, Nov 19, 2020 at 09:13:55PM +1000, onefang wrote:

> Some of what I have done in apt-panopticon might be relevant, as it
> consumes the files you are generating, if I understand what lurc is
> doing. Amprolla replacement?

Yes. And I'll keep that in mind, and coordinate some testing with you to
make sure that what lurc emits is pleasing to apt-panopticon.


> > I recommend parsing Release files rather than trusting HTTP headers,
>
> I agree with that, it's what apt-panopticon does to check if a mirror is
> up to date.

I'd be willing to use that as verification for files we've downloaded, but
I'm hesitant to make more of a traffic burden unless it consistently
becomes problematic. I don't think it'll in general be a problem.


> "sudo turkey lurc" might be some sort of joke that Aussies like me that
> only have bush turkeys, not real turkeys, might not understand.

Old European folktale, captured in print by the Brothers Grimm:

    https://en.wikipedia.org/wiki/Turkey_lurkey

On Thu, Nov 19, 2020 at 11:20:11AM +0000, Mark Hindley wrote:

> Perhaps look at https://pkgmaster.devuan.org/merged/. As far as I can see,
> everything there is necessary.

I've been using that as my model thus far and revisiting as the code starts
to take an interest in various parts.


On Thu, Nov 19, 2020 at 11:26:07AM +0000, Mark Hindley wrote:

> I should also have added that support for acquire by-hash[1] would be a real
> benefit and should get rid of the intermittent Hash Mismatch errors that we
> occassionally see.

I absolutely love that idea. Thank you for noting it. I wasn't aware.

-- 
Mason Loring Bliss   ((  "In the drowsy dark cave of the mind dreams
mason@???     ))  build  their nest  with fragments  dropped
http://blisses.org/  ((   from day's caravan." - Rabindranath Tagore