We've been making great progress on v4, but changes are relatively big so not a lot getting merged presently, as we want to keep master working. Latest code is in my repo in the headers_first branches.

I'll be traveling a lot this summer, so progress will slow a bit. Also, I'll be in Cape Town 6/26-27 if anyone happens to be in the area and wants to meet up.

We've resolved a longstanding conundrum. We achieve constant time validation for individual blocks, because the store is optimized for the main chain. However validation performance in weak chains has been linear at best (actually quadratic in parts). We've competed the design and most of the implementation to make validation of blocks building on a weak chain constant time, up to the point of reorg, where there is small but linear cost for merging a new chain of any length. This is the perfect balance of optimization for the most common scenario with a tiny tradeoff for reorgs. The tradeoff gives us constant time validation and query (as opposed to log2) in the vast majority of cases, and minimizes storage cost.

We have also eliminated the block memory pool )"orphan pool". V3/4 doesn't retain any actual orphans, but we do store valid weak chain blocks (required to allow reorg), and in v3 those blocks are pooled in RAM. Now they are stored to disk, just as tx pool. There is a memory index for both, but just metadata. This significantly flattens memory utilization and improves overall performance. In v3 we can handle any size tx pool with no problems, and now we can manage a reorg of any size in the same manner.

It also appears that we will have eliminated hard-faulting when the disk becomes full. Instead the node will just stall until disk space becomes available.

Testing has shown that an older computer can easily keep up with current tx volume with write flushing enabled, and we can now automatically manage its configuration. So it should become much easier and more likely to survive hard shutdown.

We have also completed the most interesting aspects of header first parallel block download. The node now maintains a header chain index independent of the block chain index. The header chain is built in the same manner that v3 builds the block chain. However since headers are small, the full chain (tree) builds in about 1 minute from any number of peers. The node always maintains the header chain and when a new long chain overpowers the long block chain and is current in time, it starts downloading the related blocks in parallel, and validating them in order. Once the long header chain validates to a level of work above the block chain, the two chains are reconciled with a reorg. This gives us maximal parallelism at any time, with no need for checkpointing.

Lots more to come!

e