On 05/19/2015 10:51 AM, William Swanson wrote:
> On Tue, May 19, 2015 at 1:04 AM, mlmikael <mlmikael@???> wrote:
>> Am all with you that everything about LibBitcoin and BitcoinD is different
>> altogether.
>
> To be a little more detailed, I think the bitcoind implementation just
> tracks the UTXO set when it does its block validation. The "unspent
> funds" at any given point in time is obviously a much smaller working
> set than "all funds ever", so they can get away with a lot less memory
> usage. It's possible that libbitcoin could start tracking the utxo set
> as well, allowing us to optimize our memory usage.
>
> Maybe we could have two modules, a blockchain sync module that just
> downloads and validates blocks, and a database module that indexes and
> serves those blocks to clients. The download module would only need to
> track UTXO's, giving it the same memory overhead as bitcoind. There
> are currently no plans to do it this way, so I'm just throwing this
> out as a thought experiment.

Once we move to header first download I think the issue will look very
different. In terms of major optimizations we should keep that in mind.

Excessive memory consumption during sync appears to be related to the
flooding problem.

https://github.com/libbitcoin/libbitcoin-server/issues/67

This would not occur if we were to process headers first. We start to
see memory increases coincident with large numbers of orphan blocks,
which as we know is a problem without a current solution:

https://lists.dyne.org/lurker/message/20150106.144028.a1f533a9.en.html

So it's looking like prioritizing header first might make sense. This
would be supportive of the work on v3, in terms of blockchain to support
the privacy protocol.

e