On 2015-06-01 05:00, Eric Voskuil wrote:
> On 05/29/2015 01:47 PM, mlmikael wrote:
>> On 2015-05-25 06:52, Eric Voskuil wrote:
>>> So while Neill is working on Bitcoin protocol review I'm now working
>>> on
>>> instrumenting the network stack so that we can get a view into
>>> concurrency issues.
>>
>> Eric,
>>
>> Just a low-prio wishlist thing, it would be nice if at least the most
>> fundamental of the LB state instrumentation/introspection routines you
>> implement, would be exported and have well-defined interfaces, so that
>> a
>> program using LB could get some nice stat of thread use & activity and
>> heap activity, even if just a listing like "Thread 1, blockchain, req
>> 123", "Thread 2, network IO, select()", "Orphaned blocks 123 occupying
>> 234MB", "Total blocks cached on the heap 345 occupying 456MB".
>
> Sure, makes sense.

Ok!

Just in case something would break, for post-mortem analysis, an
application could keep some kind of log of LB's state too, every X
seconds. Like, LB.debugState(),

to be used to understand why a SIGSEGV, dying from excess memory
consumption, eating all CPU, central race condition, getting
unresponsive, stack overflow (those still happen at 4MB don't they),
happened.

> I've found that the orphan pool and transaction pool during sync are
> generally empty, although the orphan pool can spike. The larger memory
> variations are from internal asynchronous message queuing.
>
> What we see is the network operations continue even on the stalled
> strand (accumulating memory), and operations continue that are not
> ordered by the stalled strand, like heartbeat. And eventually the
> stalled strand will clear, freeing up its accumulated blocks and
> transactions. I don't think anything is leaking, but because the node
> continues to ask for and obtain blocks and transactions, it can
> generate
> quite a backlog if the strand is held up.

What's the best way to counteract this, so that the spike is capped to a
pre-set X MB +-10%?


Also in the case of Absoluely Worst Case disorder on the Bitcoin
network, say

  * a huge reorganization because of
     * Network segmentation,
     * Superceding work,
     * Consensus bug in Bitcoin miner,
  * DoS,

what's the worstcase RAM use situation then?