To give some comparison I attempted to benchmark Core initial block
download in the same environment. After 2.0 hours it had completed 10%
of the chain (by it's own measure), where libbitcoin-node is 100%
complete. It's measure appears to be based on a fraction of total
transactions processed, and from my measures does appear to give a fair
estimate of it's projected performance.

It was initially about 50% faster, but as block size increases it bogs
down dramatically due to at least two factors. Both implementations pass
block 200,000 at about 40 minutes, but at that point our per block speed
has hardly reduced at all and Core has slowed to a crawl.

Basically it appears that Core is fairly linear in tx rate but
libbitcoin is close to liner in block rate. Our .5MB blocks take about
twice the time as our .5KB blocks. This implies that peer performance
and selection (time to download a certain number of bytes) dominates our
performance. At some point I'll run a test against our own peers and
find out!

First, it becomes fully CPU bound on its single worker thread, never
consuming much more than 25% of my quad core - despite being connected
to 8 peers. This is a 3:1 advantage on a quad core, where we use 75% of
the CPU. In an 8 core test our advantage could be as high as 7:1 (we are
not I/O bound). Note that both implementations use reduced thread priority.

Second, there is on the order of 100 times more I/O than libbitcoin.
This is due to the use of LevelDB, as libbitcoin has almost no disk I/O
owing to its memory-mapped file implementation (yet our startup and
shutdown is fast).

Third, it consumes more than twice the RAM as libbitcoin, and it ramps
up its RAM consumption much more rapidly.

Also, our indexing is far more efficient than LevelDB in terms of query
speed. So the result of the sync is much better. We do however consume
more disk space. I assume Core will finish around the advertised 60GB
where we take 90GB. I'll have to wait another 17 hours or so to find out :O.

e

On 03/15/2016 10:52 PM, Eric Voskuil wrote:
> Just a quick update on the state of work on libbitcoin 3.0.
>
> We've been working for a while on initial block download (IBD) in
> libbitcoin-node/server. This began with work on libbitcoin-network,
> which has been extracted from libbitcoin into a distinct repo. Work has
> more recently progressed to libbitcoin-database, which has been split
> out from libbitcoin-blockchain.
>
> We've achieved our first ever fully parallel IBD. Headers are downloaded
> first, then blocks in parallel across multiple peers (9 in this run).
> The file-mapped memory database now allows concurrent block write,
> necessary for getting the benefit of concurrent download.
>
> My test run used a 5+ year old quad core Windows machine with an HDD and
> an SSD data drive and 8GB RAM and fast Internet connectivity. The test
> run completed in 2.0 hours without interruption.
>
> I consider this a baseline from which to apply further optimizations.
> Currently there is limited peer selection and balancing, so the slowest
> peer dominates the completion time. Rough calculations imply that we
> should be able to get this down to 1.0 hours on this same machine.
>
> Memory consumption during sync is consistent and reasonably predictable,
> depending mainly on peer performance. Memory climbs fairly linearly from
> about 30MB to about 250MB at current peak rate. This compares to about
> numbers in the several GB range for version2 IBD. Blocks are no longer
> buffered, so there is no opportunity for memory to grow uncontrollably.
>
> As with memory, CPU utilization climbs fairly linearly from about 25% to
> about 75%. This implies that 8 cores and a modern processor/bus may
> significantly improve performance. This combined with peer management
> optimizations should give libbitcoin the < 1 hr startup that we've been
> targeting.
>
> Network utilization is very reasonable and based primarily on peer
> performance. You wouldn't likely notice it operating in the background.
>
> Finally, disk space for the indexed mainnet blockchain, without address
> or stealth history, is just under 90GB. This includes anew feature to
> reduce the file size when the process is closed, since there is
> preallocation of unused space. So currently one should expect a 50%
> expansion during execution, so about 150GB to be safe. Address and
> stealth history will slow operation and expand store size, which I
> haven't measured recently.
>
> I'll be pushing this latest work to master in the next couple of days.
> Note that the node/server do not fully implement post-IBD sessions, so
> the server effectively halts at the completion of IBD. Use the version2
> branch for release functionality.
>
> e
>