In my last major update (9/15, below) I said that we had taken a detour
into validation, in order to get the performance up to a level
consistent with other aspects of the code. This code has been complete
for block validation for a couple of weeks, although we still have some
work to do in reconnecting the pools.

Validation is now and very easy to follow. For example, see the
"validation" sections of these files:

https://github.com/libbitcoin/libbitcoin/blob/master/src/chain/header.cpp

https://github.com/libbitcoin/libbitcoin/blob/master/src/chain/block.cpp

https://github.com/libbitcoin/libbitcoin/blob/master/src/chain/transaction.cpp

Blockchain enhances this simple single-threaded type system validation
with a small bit of code to parallelize the invocation of input
validation. This is very fast, and well factored for test.

By populating the necessary metadata and the chain state, validation can
be executed completely independent of the store. This relies on
validation metadata on header, block, transaction and input, as well as
a new chain_state class in libbitcoin:

https://github.com/evoskuil/libbitcoin/blob/master/src/chain/chain_state.cpp

This is a big boost for testability and
simplicity/readability/reliability of the implementation.

In the past week I've taken the time to rewrite our script
implementation I've covered the details in the extensive comments of
this pull request:

https://github.com/libbitcoin/libbitcoin/pull/577

This is the first time we've reviewed the script implementation for
consensus-criticality. You will notice that three consensus bugs have
been resolved. These would have been avoided by the use of
libbitcoin-consensus (libconsensus).

We've got a little more work to do in this area, specifically emitting
precise error codes for script validation failures. Then it's back to
finish up networking. In the mean time we've disabled initial block
download (and catch-up sync cannot surpass a multiblock reorg due to
pool disconnection).

We've also exposed database hash table bucket counts:

https://github.com/libbitcoin/libbitcoin-node/blob/master/data/bn.cfg#L80-L87

and database file growth rate:

https://github.com/libbitcoin/libbitcoin-node/blob/master/data/bn.cfg#L78-L79

to config. This not only significantly speeds up database unit testing
but also allows users to tune hash tables for expected use. The current
values are (sort-of) tuned for mainnet. They are certainly not optimal
for testnet. But it would be great if someone would optimize these for
the current chains. We'll probably also configure initial file sizes so
that users can make optimize for expected use. Note that bucket size
changes will invalidate an existing hash table.

Another important change has been the addition of corruption detection.
I wrote about this recently. It is now impossible to corrupt the store
via hard shutdown and not know it. There is a trade-off in terms of
precision of that knowledge and performance. The default is in favor of
performance, but for wallet-like scenarios this can be changed with no
noticeable impact. This reduces the chance of a hard shutdown flagging a
database corruption to almost zero, but materially slows block acceptance.

https://github.com/libbitcoin/libbitcoin-node/blob/master/data/bn.cfg#L98-L99

At the same time pmienk has integrated boost_log. We now have logs that
can be rotated, and we have configurable internal rotation (in server
and node):

https://github.com/libbitcoin/libbitcoin-node/blob/master/data/bn.cfg#L3-L15

He's now working on statsd integration via a new log sink, which we
expect to have complete for v3 as well.

We have also enabled SOCKS5 client support in libbitcoin-client, and in
bx configuration:

[server]
socks_proxy = 127.0.0.1:1080

and we have received word that we will have dedicated hidden services
for both mainnet and testnet, as well as public Server and P2P endpoints
on libbitcoin.net!

e

On 09/15/2016 04:12 AM, Eric Voskuil wrote:
> Just wanted to give a quick update on our v3 progress. I didn't make my
> target date before heading off to Mongolia for a couple of weeks. After
> getting back I started back up on validating the new network protocols
> work. I resolved a couple of issues in the new implementation and it was
> going great. At that point I decided to fully validate the mainnet and
> testnet chains, as I like to do before a release.
>
> As many of you know, the blockchain library is slated for a major rework
> in v4, so we've been deferring optimizations in that code. For example,
> we don't even parallelize script validation. But v3 has been looking so
> good in other areas this started too look even worse by comparison. So
> we decided to take a week or two to redesign the block and transaction
> validation code, allowing us to get some significant optimizations
> implemented. This is coming along nicely and I hope to have some
> improvements to report soon.
>
> Once we complete this work we will need to finish protocol validation.
> The incoming block protocol is done, working perfectly. We've written
> the other major protocols, including outgoing block and in/out
> transaction, just need to verify them as well. The minor protocols have
> been implemented and working well for some time. I don't spend a lot of
> time documenting, because there has just been so far to go in just the
> code. But believe me, it's getting really good.
>
> For example the network redesign has allowed me to isolate protocols by
> version and implement robust version negotiation based on config. So you
> can specify protocol version min/max in config and the node will
> perfectly implement that range. So it can simulate older and eventually
> newer node behavior. It's also really easy to see protocol behavior
> through source inspection, and will get even better before it ships.
> Right now I'm thinking about 4 more weeks until we can complete the
> blockchain work and get back to validating the remaining network
> protocols. Sorry about the scope creep, I just couldn't help myself :).
>
> e
>