Hello,

I've recently started looking into this and found some interesting
results.

1) It's slow for various reasons, but I think it's largely attributed
to transaction processing and larger RAM demands. In the ~337000+
block range, bs quite quickly consumes a majority of RAM on my 16GB
machine causing swap to be hit and things sort of degrade from there.
I suspect the many calls to sync are fairly brutal as well in that
case, but I have not looked into that performance yet. My network
inspection also showed a healthy amount of data incoming when
requested, so nothing odd there.

2) I found a degenerate case that is user facing and likely mixed up
or confused with the above more 'normal' scenario. What I'm seeing
(and can easily reproduce) is a sync issue that causes block
processing to essentially freeze and drop connections from peers ad
infinitum.

The setup: bs built with libbitcoin-blockchain compiled
--with-consensus (default). After some time of running, blocks
mysteriously stop being processed and the node largely appears to be
inactive on that front, even though it's clear the sync hasn't
completed yet. Days may go by without a new block accepted.

So after tracking it down, here's what I've found happening:

bs starts up from the last save point and requests blocks from peers
and retrieves some. On the correct 'latest' block, it's failing to
validate a transaction input script for reason
verify_result::verify_result_pubkeytype instead of the expected
verify_result::verify_result_eval_true (search "Input script consensus
validation failed" in validate.cpp's validate_block::connect_input and
dig deeper from there). The exact block I have in testing that shows
this is block 337930 (though there are more later than that as well).
Since this check fails, the block fails to be organized into the main
chain and replace_chain fails (invalid_reason). At that point, it
seems no reconciliation can be done and all the remaining received
blocks remain orphans as well (fork_index ==
simple_chain::null_height). The same blocks are requested and
received again, and the process repeats. After a number of times, it
seems the peers disconnect bs (likely being detected as a bad actor),
but I haven't tracked down the exact reason for the disconnects yet
(bs may initiate disconnect, not sure). But even upon establishing
new connections, the same things happens over and over again and no
progress can be made on block acceptance.

NOTE: I can get things to continue chugging along as far as accepting
      blocks goes by reverting back to the local script processing
      (see validate_consensus in libitcoin-blockchain/src/validate.cpp
      and use the WITH_CONSENSUS alternate code).

      If I allow things to continue using this method, blocks are
      accepted again and things chug along subject to the 'normal'
      issue #1 above.

So I believe there's a bug somewhere. Either the
pre-libbitcoin-consensus code is not correctly running the scripts and
letting in blocks it shouldn't (does not appear to to be the case,
based on testing/debugging and block explorer hunting), or
libbitcoin-consensus does not properly handle this case and returns an
unacceptable value (currently treated as an error) on a block
containing a 'funny' input script when it should be accepted. The
code makes it appear to be related to BIP62 (SCRIPT_ERR_PUBKEYTYPE),
but ultimately appears to fail from
libbitcoin-consensus/src/script/interpreter.cpp's
CheckPubKeyEncoding). I don't know that BIP or this part well enough
yet, so I'm looking for input from others at this point.

-Neill.

On Wed, Apr 29, 2015 at 03:53:34PM -0700, William Swanson wrote:
> Yeah, it's pretty slow. Part of the problem, I think, is that the
> Satoshi peers aren't feeding blocks very quickly, since libbitcoin is
> somehow triggering their DDoS protection. Therefore, we only get
> blocks at a reduced rate. It would be interesting to wireshark this
> and figure out what's actually going on.
>
> One workaround is to setup a Satoshi bitcoind server, and edit it's
> bitcoin.conf file to explicitly add the libbitcoin server as a peer.
> This doesn't speed things up too much, but it seems to improve
> stability a bit.
>
> Even when the blocks come in at a decent rate, there also seems to be
> some slowness in the storage backend. It would be interesting to
> profile this and figure out what's actually going on. ;)
>
> In the mean time, our approach was to just wait a few weeks for our
> first server to sync up with the blockchain. Once we had the first
> server running, we rsync'ed the database across to the other servers.
> This is crude but effective.
>
> - William Swanson, airbitz.co
>
> On Wed, Apr 29, 2015 at 1:41 PM, mauricio munoz
> <gmauricio.munoz@???> wrote:
> > Hey!
> >
> > I'm currently setting up libbitcoin-server, but I found that it's very slow
> > to sync the blockchain on mainnet, I'm around the 230K block now, but it
> > looks like it's going to take many days to get up to date.
> >
> > Do you have any recommendations on how to make it sync faster ? I have it
> > running on AWS, tried setting a machine with good memory, around 15G, but
> > still very slow, and it's not consuming it all anyway, same for CPU.
> > Maybe there are tweaks that can be done to the configuration to make it
> > faster.
> >
> > We do have running, fully synced now, a testnet server, we're currently
> > setting the domain for it, and we'll share it later, because as far as we
> > know, none of the obelisk servers for testnet, listed on the wiki are
> > working or have the blockchain up to date.
> >
> > --
> > Germán Mauricio Muñoz
> >
> > _______________________________________________
> > Libbitcoin mailing list
> > Libbitcoin@???
> > https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/libbitcoin
> >
> _______________________________________________
> Libbitcoin mailing list
> Libbitcoin@???
> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/libbitcoin