Hi Neill,
Much thanks for taking the time to look into this.
Regarding #2, it looks like the default libbitcoin-consensus based build
is having a problem with your version of OpenSSL. Some more info here:
https://github.com/libbitcoin/libbitcoin-server/issues/61
The libbitcoin-blockchain install doesn't control for a maximum version
on OpenSSL. This is something we should at least document in the server
stack.
e
On 05/16/2015 10:15 PM, Neill Miller wrote:
> Hello,
>
> I've recently started looking into this and found some interesting
> results.
>
> 1) It's slow for various reasons, but I think it's largely attributed
> to transaction processing and larger RAM demands. In the ~337000+
> block range, bs quite quickly consumes a majority of RAM on my 16GB
> machine causing swap to be hit and things sort of degrade from there.
> I suspect the many calls to sync are fairly brutal as well in that
> case, but I have not looked into that performance yet. My network
> inspection also showed a healthy amount of data incoming when
> requested, so nothing odd there.
>
> 2) I found a degenerate case that is user facing and likely mixed up
> or confused with the above more 'normal' scenario. What I'm seeing
> (and can easily reproduce) is a sync issue that causes block
> processing to essentially freeze and drop connections from peers ad
> infinitum.
>
> The setup: bs built with libbitcoin-blockchain compiled
> --with-consensus (default). After some time of running, blocks
> mysteriously stop being processed and the node largely appears to be
> inactive on that front, even though it's clear the sync hasn't
> completed yet. Days may go by without a new block accepted.
>
> So after tracking it down, here's what I've found happening:
>
> bs starts up from the last save point and requests blocks from peers
> and retrieves some. On the correct 'latest' block, it's failing to
> validate a transaction input script for reason
> verify_result::verify_result_pubkeytype instead of the expected
> verify_result::verify_result_eval_true (search "Input script consensus
> validation failed" in validate.cpp's validate_block::connect_input and
> dig deeper from there). The exact block I have in testing that shows
> this is block 337930 (though there are more later than that as well).
> Since this check fails, the block fails to be organized into the main
> chain and replace_chain fails (invalid_reason). At that point, it
> seems no reconciliation can be done and all the remaining received
> blocks remain orphans as well (fork_index ==
> simple_chain::null_height). The same blocks are requested and
> received again, and the process repeats. After a number of times, it
> seems the peers disconnect bs (likely being detected as a bad actor),
> but I haven't tracked down the exact reason for the disconnects yet
> (bs may initiate disconnect, not sure). But even upon establishing
> new connections, the same things happens over and over again and no
> progress can be made on block acceptance.
>
> NOTE: I can get things to continue chugging along as far as accepting
> blocks goes by reverting back to the local script processing
> (see validate_consensus in libitcoin-blockchain/src/validate.cpp
> and use the WITH_CONSENSUS alternate code).
>
> If I allow things to continue using this method, blocks are
> accepted again and things chug along subject to the 'normal'
> issue #1 above.
>
> So I believe there's a bug somewhere. Either the
> pre-libbitcoin-consensus code is not correctly running the scripts and
> letting in blocks it shouldn't (does not appear to to be the case,
> based on testing/debugging and block explorer hunting), or
> libbitcoin-consensus does not properly handle this case and returns an
> unacceptable value (currently treated as an error) on a block
> containing a 'funny' input script when it should be accepted. The
> code makes it appear to be related to BIP62 (SCRIPT_ERR_PUBKEYTYPE),
> but ultimately appears to fail from
> libbitcoin-consensus/src/script/interpreter.cpp's
> CheckPubKeyEncoding). I don't know that BIP or this part well enough
> yet, so I'm looking for input from others at this point.
>
> -Neill.
>
> On Wed, Apr 29, 2015 at 03:53:34PM -0700, William Swanson wrote:
>> Yeah, it's pretty slow. Part of the problem, I think, is that the
>> Satoshi peers aren't feeding blocks very quickly, since libbitcoin is
>> somehow triggering their DDoS protection. Therefore, we only get
>> blocks at a reduced rate. It would be interesting to wireshark this
>> and figure out what's actually going on.
>>
>> One workaround is to setup a Satoshi bitcoind server, and edit it's
>> bitcoin.conf file to explicitly add the libbitcoin server as a peer.
>> This doesn't speed things up too much, but it seems to improve
>> stability a bit.
>>
>> Even when the blocks come in at a decent rate, there also seems to be
>> some slowness in the storage backend. It would be interesting to
>> profile this and figure out what's actually going on. ;)
>>
>> In the mean time, our approach was to just wait a few weeks for our
>> first server to sync up with the blockchain. Once we had the first
>> server running, we rsync'ed the database across to the other servers.
>> This is crude but effective.
>>
>> - William Swanson, airbitz.co
>>
>> On Wed, Apr 29, 2015 at 1:41 PM, mauricio munoz
>> <gmauricio.munoz@???> wrote:
>>> Hey!
>>>
>>> I'm currently setting up libbitcoin-server, but I found that it's very slow
>>> to sync the blockchain on mainnet, I'm around the 230K block now, but it
>>> looks like it's going to take many days to get up to date.
>>>
>>> Do you have any recommendations on how to make it sync faster ? I have it
>>> running on AWS, tried setting a machine with good memory, around 15G, but
>>> still very slow, and it's not consuming it all anyway, same for CPU.
>>> Maybe there are tweaks that can be done to the configuration to make it
>>> faster.
>>>
>>> We do have running, fully synced now, a testnet server, we're currently
>>> setting the domain for it, and we'll share it later, because as far as we
>>> know, none of the obelisk servers for testnet, listed on the wiki are
>>> working or have the blockchain up to date.
>>>
>>> --
>>> Germán Mauricio Muñoz
>>>
>>> _______________________________________________
>>> Libbitcoin mailing list
>>> Libbitcoin@???
>>> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/libbitcoin
>>>
>> _______________________________________________
>> Libbitcoin mailing list
>> Libbitcoin@???
>> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/libbitcoin
> _______________________________________________
> Libbitcoin mailing list
> Libbitcoin@???
> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/libbitcoin
>
::
Re: [Libbitcoin] Tips for syncing l…
