:: Re: [Libbitcoin] obworker uses up a…
Top Pagina
Delete this message
Reply to this message
Auteur: Amir Taaki
Datum:  
Aan: libbitcoin
Onderwerp: Re: [Libbitcoin] obworker uses up all memory and crashes [WAS: Re: New releases for libbitcoin/Obelisk/sx]
Could I have non-root SSH access to check about the spinning disks
thing. You have my PGP pubkey (I sign all my emails).

On 09/01/14 14:45, mlmikael wrote:
> Aha Amir, if basically on an SSD the IO issues should *never* happen
> with very very very good margins, then I agree with you, this is a good
> solution, in particular if this workaround you suggested (quoted below;
> which as a side-effefct provides a non-graceful shutdown) does prevent a
> full safeguard against the crash and burn scenario i.e. libbitcoin is
> robust. That's what it does right?
>
> Is it committed so l.b. is now safe?
>
> Thanks
>
> On 2014-01-09 00:29, Amir Taaki wrote:
>
>> As a programmer writing systems code targetting servers, I have to make
>> tradeoffs to better serve configurations/uses/scenarios over others.
>> I'm of the view that for old style HDDs are obsolete on servers,
>> especially for databases, network apps, ... anything doing disk IO.
>>
>> Seek times of SSDs are basically zero. To jump around a file is no
>> longer costly (think of HDDs like VCRs and SSDs like DVDs), so many
>> things change with how you optimise your software. An example, a large
>> part of LevelDBs design/activity is about locating data close together
>> to minimise head movement. The semantics of SSDs on the OS layer are
>> different too.
>>
>> This is why libbitcoin doesn't work so well on HDDs. If we can find a
>> way to make it perform better without significantly comprimising
>> elegance, and without comprimising on SSD performance then it should be
>> fixed. But I don't think it's a good idea to loose too much sleep over it.
>>
>> Especially since consumer SSDs are wildly popular and dropping in price
>> fast. With the advent of new affordable high-end DRAM SSDs entering the
>> market, I expect to see a further drop in the price of SSDs.
>>
>> http://www.techpowerup.com/188507/crossbar-unveils-resistive-ram-non-volatile-memory-technology.htmllibbitcoin is a design for tomorrow; solving yesterday’s problems today
>> is for slow-moving leviathans.
>>
>> On 08/01/14 15:51, mlmikael wrote:
>>> Aha. So what's the way to a "clean" (as in, robust/non-leaky)
>>> feedback mechanism for making earlier parts degrade gracefully and
>>> stop adding more work, when the buffer is full/close to full? Again
>>> thank you for the libbitcoin, it's basically essential to the Bitcoin
>>> world and may be the primary implementation if you look at it really.
>>> On 2014-01-08 11:42, Amir Taaki wrote:
>>>> Not really, and it doesn't seem like a good solution (more of a
>>>> workaround) now I'm thinking. The problem (if this is it), is that
>>>> the block writer is IO bound, and blocks are being queued such that
>>>> memory is exhausted before writes can complete. I'm not sure if it
>>>> is the problem, but certainly should be something along these lines
>>>> if it's only caused by HDD. Only 500 blocks are downloaded at a time
>>>> and attempted to be stored.
>>> _______________________________________________ Libbitcoin mailing
>>> list Libbitcoin@??? <mailto:Libbitcoin@lists.dyne.org>
>>> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/libbitcoin
>>
>> _______________________________________________
>> Libbitcoin mailing list
>> Libbitcoin@??? <mailto:Libbitcoin@lists.dyne.org>
>> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/libbitcoin
>
>
>
>
>
>
>
> On 2014-01-07 22:08, Amir Taaki wrote:
>
>> So if the buffer queue is too big, then don't queue writes?
>>
>> Would this be a good scheme:
>>
>> * Have a value in the service poller called:
>>
>>     size_t blocks_queued_count_ = 0;

>>
>> * Change the block storage code when a new block is received to:
>>
>>     if (blocks_queued_count_ > queue_capacity_)
>>     {
>>         log_debug(LOG_POLLER) << "Throttling writer.";
>>         return;
>>     }
>>     chain_.store(block, ...)
>>     ++blocks_queued_count_;

>>
>> * Change the callback in the store to decrement the counter:
>>
>>     --blocks_queued_count_;

>>
>> How does that sound?
>>
>>
>
>
>
> _______________________________________________
> Libbitcoin mailing list
> Libbitcoin@???
> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/libbitcoin
>