Oh that's an interesting project - to measure performance on different
setups. Is there a place where it costs nothing to setup VPS'es but they
are billed by usage? Then we could spin up nodes for different
configurations and generate test data. I would like to see the influence
of RAM size and number of cores, on sync time and query performance. I
doubt changing the swap size will make a difference.

Eric Voskuil:
> Sure, swap is generally necessary for the OS/process, but increases to swap above some typical minimum won't improve sync speed. This is because swap is trading SSD for RAM. The database is also trading SSD for RAM (and doesn't use the OS swap file). Scale from increased RAM derives from reducing database paging. With sufficient RAM to support the entire database in RAM (plus some fixed amount for OS/process requirements) the process will be full optimized in RAM+SSD. This should be at around 100GB. Beyond this the implementation scales with each new core, up to the limit of the network bandwidth and the overhead of distributing work across the cores. Due to this distribution there is a diminishing return that will at some point limit the benefit of additional cores. This is the one area that needs to be measured to determine the optimal node configuration for minimum initial block download.
>
> With respect to validation time cores (and therefore distribution overhead) will dominate. With respect to server query performance, 100GB SSD will be optimal, and the number of cores will be the limiting factor with some diminishing return limit. However the overhead of distributing work in the client-server API is less, so a higher number of cores (than for initial block download) should be fully utilizable. Again, something to measure. In recent server work I've made the number of workers utilized by the query API configurable. This defaults to 1, which should be more than sufficient for more requirements. But there is a high performance query requirement and there are sufficient cores, this number can be raised to utilize all of them.
>
> So in the areas of initial block download and server query performance, the implementation is fully scalable (will use all available hardware). The next step is to achieve full scalability in terms of blockchain maintenance, including block validation.
>
> e
>
> -----Original Message-----
> From: Libbitcoin [mailto:libbitcoin-bounces@lists.dyne.org] On Behalf Of Police Terror
> Sent: Saturday, July 30, 2016 11:57 AM
> To: libbitcoin@???
> Subject: Re: [Libbitcoin] initial block download
>
> Well hard disk space is the most scalable tradeoff. After I made the page file, everything has been working good now. I'm still syncing at around #26-9k blocks. Just stopped and restarted as a test but it's importing in parallel but seems to be working good.
>
> Results look great, I'll be very happy to have a new server running and play around with it.
>
> Eric Voskuil:
>> Version3 header first parallel initial block download is now
>> restartable after a controlled shutdown. Also...
>>
>> I have been informed by a customer who has been evaluating various
>> node implementations that version3 downloads 400,000 mainnet blocks in
>> about
>> 45 minutes on a 32 GB RAM machine. That's about twice as fast as on my
>> old 8 GB machine.
>>
>> So the < 1.0 hour goal has been achieved!
>>
>> I expect increasing speed up to 100GB RAM (full chain and indexes in
>> RAM). I can't say how many core will be fully utilized, as there is
>> some overhead in parallelization that will produce diminishing
>> returns, but I assume increasing speed up to around 32 cores. At some
>> point I'll pick up a new machine with these specs and test it out. My
>> new goal is 30 minutes for the full chain :).
>>
>> I am presently in the process of reintegrating post-initial block
>> download chain and mempool management back into version3 for a release
>> target in August.
>>
>> e
>>
>>
>>
>> _______________________________________________
>> Libbitcoin mailing list
>> Libbitcoin@???
>> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/libbitcoin
>>
> _______________________________________________
> Libbitcoin mailing list
> Libbitcoin@???
> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/libbitcoin
>