Excellent news -- looking forward to v3 all around!
-Neill.
On Fri, Dec 09, 2016 at 05:29:21PM -0800, Eric Voskuil wrote:
> Reposting as text, formatted HTML got moderated :/.
>
> ---
>
> Weve spent a good amount of time on node, following a rewrite of validation
> and script processing, and on the heels of the previous network redesign.
> Im very happy with the results. We have a very stable node and excellent
> block validation performance. We have been lucky to have the full time
> support of Bitprim in operational testing of the node on both high and low
> end platforms. You may recall that we previously achieved full chain sync of
> about 15 minutes (all blocks checkpointed). We have since validated the full
> mainnet chain countless times. Despite the fact that our networking is still
> not optimal in terms of catch-up sync (blocks-first), we are able to sync
> 0-295,000 and validate 295,001-top (Core parameters) in about 7 hrs. I
> expect that number to come down significantly in v4 owing to more advanced
> networking.
>
> Below is a sample of the new console reporting format for block acceptance,
> on a high end server (SSD, 12 Cores, 128GB RAM). You can see that we are
> processing at a rate of about 3-4 blocks per second at the top of the
> current chain. But given network limitations this is not the full story. As
> you can see our median time process a block is about 62.5 µs per input (1 µs
> = one millionth of a second), or full block validation times (i.e. excluding
> the empty blocks) averaging under .3 seconds. We have not tested on more
> than 12 cores. The number of cores used (and validation thread priority) is
> configurable, so we can easily see improvement in speed from 1-12 cores
> presently. I expect those numbers would continue to improve with additional
> cores up to some point of diminishing returns.
>
> A libbitcoin node store is 3 files for a total of ~102GB. The files are
> portable across machines. In other words 128GB RAM is more than sufficient
> to hold the entire chain, reducing paging costs. However even an 8GB RAM
> machine with 4 cores maintains healthy performance. We have not tested at
> intermediate RAM levels (e.g. 64 or 32 GB) but these sizes are expected to
> be very performant.
>
> I should point out that libbitcoin does not use a UTXO database nor does it
> cache UTXOs, so there is no problem with scale as the blockchain grows.
> Per-input performance is linear across the full range of the chain
> regardless of hardware (assuming a drive of sufficient space for the chain).
> Outputs are read, and updated as spent, directly via transaction records.
> The cost of all store reads, including outputs, activation state, block and
> transaction duplicate checks, difficulty, and timestamps, is incurred all at
> once and captured in the pµs value below (averaging 15 µs per input). The
> cost of storing the new block, including spent output information, is
> reflected in the dµs value below (averaging 5 µs per input). Reading and
> writing independently from validation and other processing is not only huge
> boost for testability but also dramatically simplifies code and exposes
> important optimizations. For example, all per-input operations with the
> exception of block check are parallelized either by tx or input.
>
> Our script validation uses a fully-generalized script processor:
>
> https://github.com/libbitcoin/libbitcoin/blob/master/include/bitcoin/bitcoin
> /impl/machine/interpreter.ipp
>
> This makes Bitcoin script easy to understand/ and test, and safer to
> maintain. Despite this high level of generalization compared to other
> implementations, we are achieving a script validation cost of 41 µs per
> input. This is 13.5% faster than we achieve using libconsensus (which can be
> swapped in via a configuration option). Testing has confirmed that this is
> entirely the consequence of the need to serialize each transaction and
> previous output for that librarys interface, and because each must then be
> deserialized internal to that library. As such libconsensus is not viable
> for high performance implementations, such as mining, but the cost is small
> enough to not adversely impact other scenarios.
>
> e
>
>
> Block Totals (milliseconds)
> ----------------------------------
> txs: count of transactions
> ins: count of inputs
> wms: wait on previous block
> vms: validate (*) time
>
> Per Input (microseconds)
> ----------------------------------
> vµs: validate (*)
> rµs: deserialization (read from wire)
> cµs: check block (context-free validation)
> pµs: populate (all database reads)
> aµs: accept block (contextual validation)
> sµs: connect block (script verify)
> dµs: deposit (all database writes)
>
> (*) validate is the time from start of block-check to the end of
> block-connect. This is the time that it takes to fully validate a block
> object. This is important in that it is the time, after receipt and
> deserialization (time also shown) of a (whole) block from a peer, that a
> miner must wait until a notification of validity is available.
>
> 09:28:37.985756 INFO [blockchain] Block [441133] 2412 txs 4214 ins 188 wms
> 307 vms 73 vµs 1 rµs 5 cµs 19 pµs 2 aµs 48 sµs 5 dµs
> 09:28:38.295939 INFO [blockchain] Block [441134] 2691 txs 4863 ins 291 wms
> 286 vms 59 vµs 2 rµs 5 cµs 21 pµs 1 aµs 31 sµs 5 dµs
> 09:28:38.751393 INFO [blockchain] Block [441135] 2136 txs 4484 ins 270 wms
> 434 vms 97 vµs 1 rµs 4 cµs 26 pµs 2 aµs 65 sµs 5 dµs
> 09:28:38.966397 INFO [blockchain] Block [441136] 1918 txs 3528 ins 427 wms
> 198 vms 56 vµs 1 rµs 4 cµs 15 pµs 1 aµs 36 sµs 5 dµs
> 09:28:39.170047 INFO [blockchain] Block [441137] 1915 txs 3258 ins 185 wms
> 186 vms 57 vµs 2 rµs 5 cµs 13 pµs 2 aµs 37 sµs 5 dµs
> 09:28:39.474600 INFO [blockchain] Block [441138] 2367 txs 4197 ins 172 wms
> 284 vms 68 vµs 1 rµs 5 cµs 17 pµs 2 aµs 45 sµs 5 dµs
> 09:28:39.798699 INFO [blockchain] Block [441139] 2386 txs 4434 ins 270 wms
> 302 vms 68 vµs 1 rµs 4 cµs 18 pµs 1 aµs 45 sµs 5 dµs
> 09:28:40.161076 INFO [blockchain] Block [441140] 2150 txs 4648 ins 293 wms
> 342 vms 73 vµs 1 rµs 4 cµs 15 pµs 1 aµs 53 sµs 4 dµs
> 09:28:40.493881 INFO [blockchain] Block [441141] 1768 txs 4754 ins 325 wms
> 313 vms 66 vµs 2 rµs 4 cµs 19 pµs 1 aµs 42 sµs 4 dµs
> 09:28:40.688693 INFO [blockchain] Block [441142] 1860 txs 3726 ins 301 wms
> 178 vms 48 vµs 2 rµs 4 cµs 12 pµs 1 aµs 30 sµs 5 dµs
> 09:28:40.977133 INFO [blockchain] Block [441143] 2737 txs 4598 ins 161 wms
> 265 vms 58 vµs 1 rµs 5 cµs 18 pµs 1 aµs 34 sµs 5 dµs
> 09:28:41.295907 INFO [blockchain] Block [441144] 1530 txs 4592 ins 251 wms
> 301 vms 66 vµs 1 rµs 4 cµs 16 pµs 1 aµs 45 sµs 4 dµs
> 09:28:41.539977 INFO [blockchain] Block [441145] 1625 txs 3637 ins 282 wms
> 228 vms 63 vµs 1 rµs 4 cµs 10 pµs 1 aµs 47 sµs 4 dµs
> 09:28:41.823650 INFO [blockchain] Block [441146] 2820 txs 4539 ins 210 wms
> 261 vms 57 vµs 2 rµs 5 cµs 17 pµs 1 aµs 34 sµs 5 dµs
> 09:28:42.163806 INFO [blockchain] Block [441147] 1395 txs 4515 ins 245 wms
> 321 vms 71 vµs 1 rµs 3 cµs 20 pµs 2 aµs 46 sµs 4 dµs
> 09:28:42.476815 INFO [blockchain] Block [441148] 999 txs 5157 ins 312 wms
> 295 vms 57 vµs 1 rµs 3 cµs 15 pµs 1 aµs 39 sµs 3 dµs
> 09:28:42.813068 INFO [blockchain] Block [441149] 1026 txs 5413 ins 284 wms
> 313 vms 58 vµs 1 rµs 2 cµs 18 pµs 1 aµs 37 sµs 4 dµs
> 09:28:42.813827 INFO [blockchain] Block [441150] 9 txs 8 ins 323 wms
> 1 vms 67 vµs 4 rµs 17 cµs 10 pµs 9 aµs 30 sµs 15 dµs
> 09:28:43.101809 INFO [blockchain] Block [441151] 2669 txs 4608 ins 0 wms
> 253 vms 55 vµs 1 rµs 4 cµs 16 pµs 1 aµs 34 sµs 5 dµs
> 09:28:43.389835 INFO [blockchain] Block [441152] 2870 txs 4474 ins 238 wms
> 264 vms 59 vµs 2 rµs 5 cµs 18 pµs 1 aµs 35 sµs 5 dµs
> 09:28:43.696813 INFO [blockchain] Block [441153] 2451 txs 4484 ins 249 wms
> 285 vms 64 vµs 1 rµs 5 cµs 16 pµs 1 aµs 41 sµs 5 dµs
> 09:28:44.005768 INFO [blockchain] Block [441154] 2217 txs 4515 ins 274 wms
> 288 vms 64 vµs 1 rµs 4 cµs 19 pµs 2 aµs 39 sµs 5 dµs
> 09:28:44.488287 INFO [blockchain] Block [441155] 1407 txs 4516 ins 279 wms
> 462 vms 102 vµs 1 rµs 3 cµs 37 pµs 2 aµs 60 sµs 5 dµs
> 09:28:45.096894 INFO [blockchain] Block [441156] 1291 txs 3971 ins 456 wms
> 591 vms 149 vµs 1 rµs 4 cµs 34 pµs 1 aµs 109 sµs 4 dµs
> 09:28:45.373956 INFO [blockchain] Block [441157] 2556 txs 4412 ins 582 wms
> 255 vms 58 vµs 1 rµs 4 cµs 13 pµs 1 aµs 39 sµs 5 dµs
> 09:28:45.751849 INFO [blockchain] Block [441158] 2122 txs 4869 ins 245 wms
> 356 vms 73 vµs 1 rµs 4 cµs 18 pµs 1 aµs 50 sµs 4 dµs
> 09:28:45.752609 INFO [blockchain] Block [441159] 10 txs 9 ins 359 wms
> 0 vms 54 vµs 2 rµs 12 cµs 6 pµs 7 aµs 30 sµs 17 dµs
> 09:28:46.167439 INFO [blockchain] Block [441160] 2260 txs 4798 ins 0 wms
> 380 vms 79 vµs 1 rµs 3 cµs 24 pµs 1 aµs 51 sµs 5 dµs
> 09:28:46.534458 INFO [blockchain] Block [441161] 2037 txs 4720 ins 373 wms
> 347 vms 74 vµs 1 rµs 4 cµs 16 pµs 1 aµs 53 sµs 4 dµs
> 09:28:46.786271 INFO [blockchain] Block [441162] 1478 txs 4044 ins 328 wms
> 236 vms 58 vµs 2 rµs 4 cµs 14 pµs 2 aµs 39 sµs 4 dµs
> 09:28:47.062390 INFO [blockchain] Block [441163] 2620 txs 4455 ins 217 wms
> 253 vms 57 vµs 2 rµs 5 cµs 13 pµs 1 aµs 37 sµs 5 dµs
> 09:28:47.063237 INFO [blockchain] Block [441164] 10 txs 9 ins 255 wms
> 1 vms 69 vµs 3 rµs 15 cµs 7 pµs 7 aµs 40 sµs 14 dµs
> 09:28:47.381445 INFO [blockchain] Block [441165] 2490 txs 4611 ins 0 wms
> 285 vms 62 vµs 1 rµs 4 cµs 16 pµs 1 aµs 41 sµs 5 dµs
> 09:28:47.697020 INFO [blockchain] Block [441166] 2266 txs 4225 ins 271 wms
> 297 vms 70 vµs 2 rµs 4 cµs 15 pµs 1 aµs 49 sµs 4 dµs
> 09:28:47.904200 INFO [blockchain] Block [441167] 1689 txs 3652 ins 283 wms
> 191 vms 52 vµs 2 rµs 4 cµs 9 pµs 1 aµs 38 sµs 4 dµs
> 09:28:48.215998 INFO [blockchain] Block [441168] 2301 txs 4441 ins 174 wms
> 293 vms 66 vµs 2 rµs 4 cµs 12 pµs 1 aµs 48 sµs 4 dµs
> 09:28:48.490337 INFO [blockchain] Block [441169] 2823 txs 4950 ins 273 wms
> 250 vms 50 vµs 2 rµs 5 cµs 15 pµs 1 aµs 30 sµs 5 dµs
> 09:28:48.771430 INFO [blockchain] Block [441170] 2728 txs 4938 ins 232 wms
> 258 vms 52 vµs 2 rµs 4 cµs 16 pµs 1 aµs 30 sµs 5 dµs
> 09:28:49.252750 INFO [blockchain] Block [441171] 1967 txs 4328 ins 242 wms
> 464 vms 107 vµs 2 rµs 4 cµs 14 pµs 2 aµs 87 sµs 4 dµs
> 09:28:49.564182 INFO [blockchain] Block [441172] 2401 txs 4575 ins 451 wms
> 291 vms 64 vµs 1 rµs 4 cµs 13 pµs 2 aµs 45 sµs 5 dµs
> 09:28:50.094058 INFO [blockchain] Block [441173] 1482 txs 4119 ins 281 wms
> 514 vms 125 vµs 1 rµs 4 cµs 10 pµs 2 aµs 108 sµs 4 dµs
> 09:28:50.446385 INFO [blockchain] Block [441174] 2216 txs 4791 ins 502 wms
> 331 vms 69 vµs 1 rµs 4 cµs 15 pµs 1 aµs 49 sµs 4 dµs
> 09:28:50.860666 INFO [blockchain] Block [441175] 1172 txs 5169 ins 322 wms
> 396 vms 77 vµs 1 rµs 3 cµs 12 pµs 1 aµs 61 sµs 4 dµs
> 09:28:51.197153 INFO [blockchain] Block [441176] 2022 txs 4642 ins 388 wms
> 317 vms 68 vµs 1 rµs 4 cµs 13 pµs 2 aµs 50 sµs 4 dµs
> 09:28:51.512118 INFO [blockchain] Block [441177] 1852 txs 4358 ins 307 wms
> 296 vms 68 vµs 1 rµs 4 cµs 14 pµs 2 aµs 48 sµs 4 dµs
> 09:28:51.859025 INFO [blockchain] Block [441178] 2319 txs 5030 ins 285 wms
> 326 vms 65 vµs 1 rµs 4 cµs 13 pµs 1 aµs 47 sµs 4 dµs
> 09:28:52.185014 INFO [blockchain] Block [441179] 2612 txs 4344 ins 314 wms
> 305 vms 70 vµs 1 rµs 5 cµs 14 pµs 2 aµs 49 sµs 5 dµs
> 09:28:52.475703 INFO [blockchain] Block [441180] 2262 txs 4030 ins 287 wms
> 272 vms 68 vµs 2 rµs 5 cµs 15 pµs 2 aµs 46 sµs 5 dµs
> 09:28:52.798759 INFO [blockchain] Block [441181] 2885 txs 5092 ins 253 wms
> 299 vms 59 vµs 2 rµs 4 cµs 17 pµs 1 aµs 36 sµs 5 dµs
> 09:28:53.078336 INFO [blockchain] Block [441182] 2436 txs 4915 ins 282 wms
> 257 vms 52 vµs 2 rµs 4 cµs 14 pµs 1 aµs 32 sµs 5 dµs
> 09:28:53.398266 INFO [blockchain] Block [441183] 2843 txs 4446 ins 238 wms
> 298 vms 67 vµs 2 rµs 5 cµs 20 pµs 2 aµs 41 sµs 5 dµs
> 09:28:53.707133 INFO [blockchain] Block [441184] 2324 txs 4476 ins 278 wms
> 288 vms 64 vµs 2 rµs 4 cµs 16 pµs 2 aµs 42 sµs 5 dµs
> 09:28:53.918838 INFO [blockchain] Block [441185] 1512 txs 3339 ins 276 wms
> 196 vms 59 vµs 1 rµs 4 cµs 9 pµs 1 aµs 44 sµs 5 dµs
> 09:28:54.237412 INFO [blockchain] Block [441186] 2067 txs 4655 ins 180 wms
> 299 vms 64 vµs 2 rµs 4 cµs 14 pµs 2 aµs 45 sµs 4 dµs
> 09:28:54.525741 INFO [blockchain] Block [441187] 1403 txs 5186 ins 282 wms
> 268 vms 52 vµs 1 rµs 3 cµs 12 pµs 1 aµs 35 sµs 4 dµs
> 09:28:54.768170 INFO [blockchain] Block [441188] 770 txs 4352 ins 259 wms
> 227 vms 52 vµs 1 rµs 2 cµs 10 pµs 1 aµs 39 sµs 3 dµs
> 09:28:55.050891 INFO [blockchain] Block [441189] 2428 txs 4573 ins 220 wms
> 262 vms 57 vµs 1 rµs 4 cµs 14 pµs 1 aµs 38 sµs 5 dµs
> 09:28:55.129727 INFO [blockchain] Block [441190] 477 txs 1278 ins 261 wms
> 73 vms 57 vµs 1 rµs 4 cµs 4 pµs 2 aµs 47 sµs 4 dµs
> 09:28:55.287538 INFO [blockchain] Block [441191] 1355 txs 2932 ins 66 wms
> 146 vms 50 vµs 1 rµs 4 cµs 8 pµs 1 aµs 37 sµs 4 dµs
> 09:28:55.622908 INFO [blockchain] Block [441192] 1139 txs 5365 ins 136 wms
> 305 vms 57 vµs 1 rµs 3 cµs 12 pµs 1 aµs 41 sµs 6 dµs
> 09:28:55.717459 INFO [blockchain] Block [441193] 904 txs 1852 ins 314 wms
> 86 vms 46 vµs 2 rµs 4 cµs 7 pµs 1 aµs 34 sµs 5 dµs
> 09:28:55.986453 INFO [blockchain] Block [441194] 250 txs 4923 ins 74 wms
> 255 vms 52 vµs 1 rµs 2 cµs 11 pµs 1 aµs 38 sµs 3 dµs
> 09:28:55.987299 INFO [blockchain] Block [441195] 7 txs 10 ins 261 wms
> 1 vms 62 vµs 3 rµs 11 cµs 8 pµs 7 aµs 37 sµs 12 dµs
> 09:28:56.262784 INFO [blockchain] Block [441196] 2502 txs 4894 ins 0 wms
> 243 vms 50 vµs 1 rµs 4 cµs 14 pµs 1 aµs 31 sµs 4 dµs
> 09:28:56.539972 INFO [blockchain] Block [441197] 1963 txs 5490 ins 228 wms
> 257 vms 47 vµs 1 rµs 3 cµs 13 pµs 1 aµs 29 sµs 4 dµs
> 09:28:56.846339 INFO [blockchain] Block [441198] 2574 txs 4683 ins 246 wms
> 281 vms 60 vµs 1 rµs 4 cµs 16 pµs 1 aµs 38 sµs 5 dµs
>
> <end>
>
>
> _______________________________________________
> Libbitcoin mailing list
> Libbitcoin@???
> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/libbitcoin
::
Re: [Libbitcoin] libbitcoin node up…
