libbitcoin updates i wrote for a friend.


- -------- Original Message --------
Subject: libbitcoin update: new blocks optimisation inside store()
Date: Tue, 18 Jun 2013 20:31:55 +0200
From: Amir Taaki <genjix@???>
To: caedes@???

#######################
New blocks optimisation
#######################

When new blocks are written to the blockchain then the reorganize
subscribed handlers are called. store()ing a block is a single
operation, but the reason we have the ability to register callbacks is
because when you call store(), blocks might not get into the chain and
so just get added to the orphan pool.

src/blockchain/leveldb/leveldb_blockchain.cpp
See leveldb_blockchain::do_store(..)

It's only later once the missing block comes in, that they suddenly
make it into the chain. Also blocks get removed and so on in a single
operation.

Another argument is that the code involved with monitoring and acting
on changes to the blockchain is not necessarily linked with the code
that calls store() (the poller just waits for new blocks and blindly
called chain.store(blk)).

The argument for reorganize subscribers is strong.

- ------

The subscriber mechanism (include/bitcoin/utility/subscriber.hpp) is
used often in libbitcoin. Because we have this common utility,
subscriptions in libbitcoin are entirely consistent:

- - all handlers are cleared from the queue after they are called.
- - handlers take an error_code that is mainly used for signalling the
end of the service (and a premature unsubscription).

- -------

The reorganize subscribe handler in leveldb_blockchain is called after
a new block is stored.

src/blockchain/leveldb/leveldb_organizer.cpp

Every call to do_store() is ran through an async_strand
(threadpool.hpp) object. This async_strand has the method push() and
queue(). Both methods ensure that any handler (void handler();) passed
to them will not execute at the same time. This way you can post
handlers to your threadpool and ensure that shared data is not
modified by many threads simultaneously (race conditions).

Because the blockchain does not worry (or should not) the order of the
blocks fed to it (the end state is always the same regardless of the
order), then we use push() which doesn't guarantee any ordering of the
write operations. queue() would enforce ordering (queue(a); queue(b);
queue(c); -> a() b() c() always - not so with push(handler);).

- --------------

Before libbitcoin would call inside the strand, the subscription
handlers with the results. This would hold up any subsequent write
operations to the blockchain that are queued if the API user didn't
exit from his registered handlers fast enough.

Solution: we use a separate async_strand (calling queue() to guarantee
ordering) to call user subscribed callback handlers with blockchain
updates. Using queue() guarantees a consistent ordered view of
blockchain changes that makes contextual sense).

We pass our delegating handler into the organizer.

    auto reorg_handler = [this](
        const std::error_code& ec, size_t fork_point,
        const blockchain::block_list& arrivals,
        const blockchain::block_list& replaced)
    {
        // Post this operation without using the strand. Therefore
        // calling the reorganize callbacks won't prevent store() from
        // continuing.
        reorg_strand_.queue(
            [=]()
            {
                reorganize_subscriber_->relay(
                    ec, fork_point, arrivals, replaced);
            });
    };
    organize_ = std::make_shared<leveldb_organizer>(
        common_, orphans_, chainkeeper, reorg_handler);