Before Dec last year I made the prototype memory mapped hashtable
specialised for Bitcoin inspired by Martin Hedenfalk's append-only
BTREE. Spend time to gather data, learn.

Then the stealth_database came very quickly thrown together to have the
functionality working. Since stealth isn't widely used yet, it can last
a while.

These last months I've been working on the new databases to replace
LevelDB. I started with the history_scan_database, which was my first
foray. Then after I spent some time to properly develop a plan for the
entire structure, and implemented it. Now I'm quite pleased but see
several areas for improvement still.

Based on my knowledge gained, and improving the code, I think the
stealth_database and history_scan_database will be replanned and
rewritten, maybe sharing a common base since their semantics are very
similar.

However I want to move into the next stage of iteration because I'm
worried I won't reach production. I want to get the things stable to
give breathing space so that then we can go back... also armed with
knowledge of what is or isn't working well. I don't expect things to go
without a bumpy ride and some problems.

On 13/08/14 03:13, Amir Taaki wrote:
> Oh Eric, I forgot to mention, the scan() method is *not* for stealth...
> now Obelisk will support privacy enhanced history for address- scan by
> prefix.
>
> When you restore your wallet, you should use the much faster
> fetch_history(). But for updating the wallet, use the scan() method-
> it's also faster because you avoid pulling the entire history. Also we
> can increase privacy and increase performance (less calls to scan()) by
> 'mining' addresses with the same prefix, therefore using single calls to
> scan() at a time. However I'm not going to push that on DW team- at
> least not right away. Designer also has his own list of requirements.
>
> On 13/08/14 02:46, Amir Taaki wrote:
>> Yep ok, I'll this list updated with news of my progress. Right now I'm
>> going to create the main database abstractions to encapsulate the lower
>> level database objects (transaction_database, history_database,
>> spend_database, block_database, ...). Then I'll integrate these into the
>> blockchain.
>>
>> It's better with stealth stuff to have the tx_hash because you'll
>> discard most of the results, and then you can choose to query a certain
>> subset of it. tx data is ~200 bytes so even with 10 blocks containing
>> say >10 txs, that's already 20 kb which from many hosts is a lot of data
>> (this is minimum case).
>>
>> Anyway I'm willing to implement whatever as long as it has decent
>> consideration given to scalability issues. We do have to make some
>> tradeoffs. Also the initial design is never going to be perfect.
>>
>> More is possible, but I want to focus on the absolute bare basics for
>> now to get them 100% solid and our the door on a production level. That
>> will involve significant Q&A work between now and the end of the year.
>> I'm also working with a designer for DW to craft the tools to create a
>> beautiful and great experience for the users rather than a tool that
>> "just works" but is horrible to use.
>>
>> On 12/08/14 22:25, Eric Voskuil wrote:
>>> I am pretty close to being able to merge the single_file SX branch. It's a total rewrite so it may not be worth your time to fix the SX dev branch. Once obelisk and libbitcoin dev branches are updated I can merge my SX branch into dev and reconcile open issues. That would of course lead to a period of breakage on the dev branch until the merge is complete. I can wait to do this until after the updated obelisk client/server merge from s_tec.
>>>
>>> Adding paging to fetch_history makes sense.
>>>
>>> Are you replacing fetch_stealth(prefix, handler, height) with scan(prefix, handler, from_height, to_height)? If so it also makes sense to me. Prefix searches (scans/subscriptions) should always return the full transaction, whether or not the tx appears to include a stealth payment. Currently fetch_stealth returns the tx hash, so a subsequent call is required to chase down the tx. This seems like a material privacy issue with fetch_stealth(). Also searching for transactions using a prefix is useful for both stealth and non-stealth payments, meaning that the return of a "stealth row" from a prefix search is problematic. The paging is also good.
>>>
>>> In general we should isolate the idea of a "prefix scan" with a "stealth payment". Any transaction may or may not include stealth payments, and there may or may not be stealth metadata associated with a stealth payment in a transaction (implying out of band / off blockchain communication of the ephemeral key). Identifying a stealth payment within a transaction should be strictly a client role subsequent to a prefix search (or other means of obtaining the transaction). If a server can readily identify and index all (or even a slight superset of) stealth payments then it's clear that stealth is not gaining meaningful cover from non-stealth transactions, which I would consider another material privacy issue.
>>>
>>> e
>>>
>>> -----Original Message-----
>>> From: Libbitcoin [mailto:libbitcoin-bounces@lists.dyne.org] On Behalf Of Amir Taaki
>>> Sent: Tuesday, August 12, 2014 8:37 AM
>>> To: libbitcoin@???
>>> Subject: [Libbitcoin] Proposed blockchain changes
>>>
>>> Beginning now I will begin integrating blockchain database changes
>>> inside libbitcoin_blockchain, afterwards porting obelisk and sx on the
>>> develop branches.
>>>
>>> New methods I propose:
>>>

>>>     fetch_block_data(...) -> block_header + [tx_positions]
>>>     fetch_transaction(position_type position, ...)
>>>     scan(prefix, handler, from_height, to_height)

>>>

>>>     fetch_history will have stop and start indexes.
>>>     (* or maybe 2 methods)

>>>
>>> Also there will be a distinction between required and optional databases.
>>> Optional databases:
>>> * history database
>>> * history scan database
>>> * stealth database
>>> You will have the option to activate these after certain block heights
>>> making initial sync much faster.
>>>
>>>
>>>
>>
>>
>>
>> _______________________________________________
>> Libbitcoin mailing list
>> Libbitcoin@???
>> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/libbitcoin
>>
>
>
>
> _______________________________________________
> Libbitcoin mailing list
> Libbitcoin@???
> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/libbitcoin
>