I believe that there is so much change involved we should fork Obelisk
and tackle the planned re-name (libbitcoin_server):

    https://wiki.unsystem.net/en/index.php/Libbitcoin

significant protocol change (first announced here):

    https://github.com/pmienk/libbitcoin_protocol

and blockchain rewrite:

    https://github.com/libbitcoin/libbitcoin_blockchain

in the new server repo.

It would likely be easier for all concerned to handle migrations as
side-by-side deployment of libbitcoin_server and obelisk until obelisk
is obsolete. Obelisk is nearly stateless (with the exception of
short-term subscriptions), so there are no data migration issues. This
eliminates the need for versioning and allows for perpetual
compatibility if it's desired. The new protocol is very flexible and
designed from the ground up for privacy-performance tradeoffs as well as
evolution, so in the future migrations will presumably be more evolutionary.

There's really no other option other than statefulness if you want
immediate change notification. I disagree however with the performance
assertion. The sending of prefixes is an order of magnitude less than
the data moving in the other direction. A registered prefix filter must
notify on any prefix hit, which means there will necessarily and
purposefully be a lot of false positives. Otherwise there is
insufficient privacy.

The subscriber must either receive the entire collection of transaction
information on the notification or it must request it in response to the
notification. In many cases the client will need to request the data in
response to the notification (standard mobile notification services for
example).

The presumption of the requirement for notification is that the
notification will always be honored. Otherwise you would just poll.
However this places bandwidth consumption at the mercy of the privacy
level, since the privacy level is exactly the level of false-positives.
More privacy means more notifications which means more requests for
large amounts of false-positive transaction data. You can't request just
the transaction data that's a true-positive, since that defeats the
privacy objective (i.e. with a trivial timing attack).

Polling on the other hand decouples the privacy level from the request
interval. So polling can be less frequent, despite a very high level of
false positives/privacy.

Notification and polling each have tradeoffs, I just wanted to make sure
that this one wasn't overlooked, as it's significant in a privacy context.

Having said that, the reason notification isn't described in the
protocol is that we didn't realize Dark Wallet still had the
requirement. Our respective scenarios do not require it, but I don't
think any of us who worked on the server have a problem with the feature
being implemented.

WRT timing, we are ready to fork Obelisk and start this work and can
remain flexible WRT the timing of the blockchain integration.

Eric


On 08/29/2014 05:04 PM, Amir Taaki wrote:
> to exploit new features such as prefix scanning addresses or performance
> improvements (the new database changes the API) will require an upgrade
> at some point.
>
> At the moment for the server, I will go with whatever you and Eric decide.
>
> BTW I changed my mind about the polling stuff. Pablo (caedes from DW)
> told me polling is a problem because a) you don't get immediate
> notification of changes and b) you continuously have to send all your
> prefixes or addresses which consumes bandwidth. So therefore some state
> is unavoidable in that we need a subscribe system.
> What are your thoughts on this? Would you agree with that or do you see
> a different way?
>
> I'm a fan of the IDL stuff because we can easily prototype an API
> together, and then once we have developed the standard we want to have
> based on our consensus, I/we can go implement it.
> I'm excited to finish this work, but am still a little away. I've began
> integration with main blockchain. The scan history database is working
> but not integrated and maybe needs more work, the stealth db needs a
> review too. But the blockchain code is a lot shorter and clearer now.
>
> On 08/30/2014 12:37 AM, William Swanson wrote:
>> Hello,
>> Amir an I have been having a private email conversion about the
>> Obelisk server, but this information should probably go on the mailing
>> list as well.
>>
>> My company, AirBitz is getting close to releasing our wallet to the
>> general public. Once this happens, will have a huge number of clients
>> expecting hard binary compatibility with the current obelisk protocol
>> (as of obelisk commit 7768117901). We can't count on these clients
>> upgrading, so we need to come up with a plan for future compatibility.
>>
>> One solution is to introduce explicit versioning into the protocol.
>> When clients first connect, they send a "version" command, which the
>> server replies to with a "major.minor" version number. Each time we
>> add a new command, we bump the minor version, and each time we remove
>> or change a command, we bump the major version number.
>>
>> The other solution is to freeze the obelisk protocol and never change
>> it again except in backwards-compatible ways. Most internet protocols
>> take this approach, which is why the internet works as well as it
>> does. We have lots of old, creaky protocols like HTTP and SMTP that
>> would be nice to "clean up", but the benefits of compatibility
>> outweigh the costs of upgrading. To enable this approach, we should at
>> least have the server return an "I don't understand that" message each
>> time the client sends an unsupported request.
>>
>> I should also mention that there are plans floating around to replace
>> "obelisk" with an entirely new "libbitcoin-server" project. This give
>> us an opportunity to create a clean break with the existing protocol,
>> so we don't need to get things perfect the first time. This is also an
>> argument in favor of just freezing the existing protocol, and only
>> making backwards-compatible additions.
>>
>> Whichever way we go, we need to make a decision soon (within the next
>> week or so). We are about to go into feature-freeze, and this issue
>> only just came up [1].
>>
>> -William
>>
>> [1] As it turns out, the current "protocol.broadcast_transaction"
>> command isn't really reliable. To make it reliable, we need to change
>> semantics a bit. The question is, do we stuff new behavior into the
>> existing "protocol.broadcast_transaction", or do we add a new command,
>> "node.push_transaction"? How we resolve the versioning issue will
>> determine how we approach this problem.
>> _______________________________________________
>> Libbitcoin mailing list
>> Libbitcoin@???
>> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/libbitcoin
>>
>
>
>
> _______________________________________________
> Libbitcoin mailing list
> Libbitcoin@???
> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/libbitcoin
>