The best approach would be for you to profile the server under expected
conditions.

But until we full understand the cause of the excessive memory
consumption during sync and resolve it there is no good way to answer
these questions.

e

On 05/19/2015 03:09 PM, mlmikael wrote:> Wait, the figure there shows
every 100 000 blocks to take 1GB of RAM -
> though Linux could declare the file memory mapping to take just about
> anything couldn't it?
>
> From how LB's internal structures look, do you think that the
> requirement is indeed 10KB per block -
>
> Though that makes no sense when compared to the Windows numbers you
> showed that seemed to imply a flat ~~100MB after startup?
>
>
> (If it's 10KB per block, is this what motivated William's suggestion
> today?)
>
>
> Thanks :)
>
>

On 05/19/2015 01:44 PM, Eric Voskuil wrote:
> I don't think you can rely on those numbers. See my Ubuntu updates here:
>
> https://github.com/libbitcoin/libbitcoin-server/issues/76
>
> e
>
>
> On 05/19/2015 01:04 AM, mlmikael wrote:
>> Hi Eric,
>>
>> Thanks a lot for your response -
>>
>> So startup peaks ~1GB independent of blockchain size, however some of
>> that could be offloaded to swap.
>>
>> The resident RAM consumption after that is made up of libBitcoin's
>> internal mempool and that one is ~~100MB,
>>
>> so a sanctions ~1.2GB resident RAM for the v2 node should be safe,
>> though really 100-200MB is the more correct figure after startup?
>>
>>
>> Am all with you that everything about LibBitcoin and BitcoinD is
>> different altogether.
>>
>> Some OS environments have max OS process resident RAM size limits and
>> virtual memory limits (ulimit) and take down the process or make malloc
>> return NULL, leading to c++ heap overflow exception or worse, so it's
>> good to have an idea here.
>>
>> Thanks!
>> Mlmikael
>>
>>
>> On 2015-05-19 11:36, Eric Voskuil wrote:
>>> Hi,
>>>
>>> There is not specific number for minimum required required RAM. If you
>>> are low enough on RAM then the process will page virtual memory.
>>> Depending on what that is being used for this can really slow things
>>> down or have a minimal effect. If you run out of swap space you should
>>> see an exception bring the process down (which is intended behavior).
>>> I'd like to give better info, but that's what we have right now.
>>>
>>> There is no relationship between the implementation of libbitcoin and
>>> bitcoind. Libbitcoin uses memory-mapped I/O, optimized for fast query
>>> response but not conservative use of memory. It's an intentional
>>> trade-off. The various data files are mapped into memory and can claim
>>> as much as they do on disk. However this will most certainly be virtual
>>> memory, so it has nothing to do with the amount of hardware RAM
>>> available.
>>>
>>> I've uploaded some resource snapshots one of my built test platforms:
>>>
>>> https://github.com/libbitcoin/libbitcoin-server/issues/76
>>>
>>> This is after doing some work today to manage thread priorities, which I
>>> am in the process of merging. That helps quite a bit with CPU hogging.
>>> As you can see CPU and network are typically pretty reasonable. This
>>> makes it possible to sync and still use the machine.
>>>
>>> Private bytes sort of represents the memory (excluding memory mapped I/O
>>> for the data files and binaries) that the process is claiming. As you
>>> can see this spikes to around 1GB when starting up. It can also climb
>>> over time as the mempool is filling up but drops down significantly once
>>> blocks are identified to consume those transactions.
>>>
>>> Also notice that this amount tends to plateau. This is probably the
>>> result of hitting maximums for the mempool, not some memory limit
>>> imposed in code. This machine has 8GB RAM, so this 1GB+ amount is not
>>> actually a significant issue. I've seen this number stay in the
>>> 50KB-100KB range for some time.
>>>
>>> I see no unexpected behavior in the logs, although the lack of
>>> header-first sync certainly consumes some time and memory (due to the
>>> receipt of blocks and transactions before they can be connected to the
>>> chain). I believe the blocking issues in the 330k range is resolved. I
>>> had misparameterized the libbitcoin-consensus library. The OpenSSL issue
>>> is worked around in the consensus lib, so we should not have to worry
>>> about that until the next OpenSSL update.
>>>
>>> We are working now to see if there is any actual issue remaining.
>>> Working off and an I've synced 270,000 blocks today and it's moving
>>> along as I would expect.
>>>
>>> e
>>>
>>> On 05/18/2015 10:34 AM, mlmikael wrote:
>>>> Hi!
>>>>
>>>> How much (resident) RAM should be sanctioned for a fully syncing v2
>>>> node?
>>>>
>>>> Is the RAM use here proportionate to the amount of parallellism in the
>>>> syncing work (if so a configurable upper limit possible)?
>>>>
>>>> BitcoinD required ~700MB last I checked.
>>>>
>>>> Thanks!
>>>> Mlmikael
>>>>
>>>> _______________________________________________
>>>> Libbitcoin mailing list
>>>> Libbitcoin@???
>>>> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/libbitcoin
>>
>