On 02/02/2015 03:13 PM, Jorge Timón wrote:
> On Sat, Jan 31, 2015 at 12:05 AM, Eric Voskuil <eric@???> wrote:
>>> Well, a "standard" for consensus-related code as you propose may be a
>>> good idea as well, I don't know.
>>> But bitcoin has unique requirements when it comes to consensus-related
>>> code acting identically and that's what libconsensus is about.
>>
>> I'm not aware of any unique requirements. Many systems rely on
>> protocols/standards that if not properly implemented can lead to loss of
>> value. The purpose of protocols is to define these requirements so that
>> there can be multiple implementations.
>
> What defines the chain you listen to (you can think of this as a
> communication channel) is the genesis block, the consensus rules and
> only after that the proof of work.
> If your consensus checks are different from anyone else's you can end
> up listening to a different chain and then...well...people can cheat
> you.
> I believe this property of the system is quite unique.
> There can be multiple implementations of the protocol (which includes
> many non-consensus things like the p2p messages) and that's healthy.
> But all implementations must be functionally identical when it comes
> to consensus checks, period.

Not period, more like, comma (more below).

>>>>> I disagree that libconsensus is against decentralization. Just like a
>>>>> wide use of libsecp256k1 is not against decentralization.
>>>>
>>>> I'm much less concerned about one of multiple implementations of
>>>> well-defined standards rooted in math than definition of an ultimately
>>>> subjective standard based on whatever gets implemented by single
>>>> organization.
>>>
>>> Consensus::VerifyScript() is rooted in math as well, it's just an
>>> interpreter of a language with crypto features.
>>> Ideally once completed libconsensus should be as said touched rarely
>>> and with many organizations involved in those changes, not just one.
>>> Ideally libconsensus would be the only part of bitcoin core that
>>> everybody cares about.
>>>
>>>>> Consensus failures don't increase decentralization in any way that I'm aware of.
>>>>
>>>> One implementation across the entire ecosystem certainly increases
>>>> centralization.
>>>
>>> I believe there will be still be reimpletations for software where
>>> having a C lib as a dependency is problematic, for example, javascript
>>> where the target platform may be an html5 interpreter.
>>> It is always very risky though since they have to be "bug by bug
>>> identical" to prevent consensus failures.
>>
>> In which case the need for conformance testing remains and the reasoning
>> for a common implementation doesn't hold up.
>
> Again, tests can't cover all cases. There's simply too many possible cases.
> It's not that a common implementation for consensus code is *needed*
> but I believe it can be useful for many implementations and in the
> case of libbitcoin I still cannot see any good reason not to depend on
> it in the long term.

We seem to be talking past each other on a couple of points, including
this one. I agree that tests cannot practically enumerate every
possibility, and it may not be feasible to locate every corner case, but
this is really beside the point. One cannot execute tests without test
vectors, which are actually or effectively data. Comparing the results
of the two implementations therefore does not require integration, or
even benefit from it.

>>>> If multiple implementations strengthen Bitcoin it's in large part
>>>> because there are multiple implementations of consensus.
>>>
>>> I completely disagree. It is because there are multiple
>>> implementations of everything else (which is most of the code IMO).
>>> For example, bitcoinj having its own interpreter doesn't "strengthen
>>> Bitcoin" in any way that I'm aware of, I believe it only introduces
>>> high risks that come from subtle differences between C and java.
>>
>> It's very possible that higher level language implementations will catch
>> failures in what is considered the "reference" implementation.
>
> I still don't think that's the reason why having multiple
> implementations of the protocol is healthy.
> And I don't think you can extend that to consensus code.
> In fact, I think a clean libconsensus could be a great formal
> specification of the consensus rules.

This amounts to a difference in opinion, but based on my experience
different implementations, especially in different languages, will
surface correctness problems in each implementation, but more frequently
in the lower level implementations.

>> OpenSSL has been the reference implementation for the curve, but it
>> turns out that there are problems that are only being discovered due to
>> alternative implementations.
>
> I don't think secp256k1 is not a high priority for OpenSSL and
> certainly bitcoin has special properties when it comes to signatures.
> Checking signatures differently can produce consensus failures!
> Those are some of the reasons for bitcoind to move to libsecp256k1.
> I still don't understand why you think libbitcoin using libconsensus
> is a bad idea but using libsecp256k1 instead of reimplementing your
> own library.
> The main difference in my mind is that libsecp256k1 may actually get
> some non-bitcoin related uses, but that's about it.

My comment had nothing to do with the choice of one EC lib vs. another.
It was to point out that implementation-as-standard doesn't ensure
correctness, and that multiple implementations have a much better chance
of doing so.

>>>>> Besides, once finished libconsensus is expected to be very stable,
>>>>> only changing when consensus rules change (ie softforks or hardforks).
>>>>
>>>> Who decides when the rules are to change?
>>>
>>> Certainly not libconsensus maintainers.
>>> Miners decide in the case of softforks, everyone needs to agree in the
>>> case of hardforks.
>>
>> It is not the case that everyone needs to agree. Bitcoin is not that
>> fragile. If it was it would not stand a chance of survival. Miners can
>> forking the code and doing what they want is the example of disagreement.
>>
>> But with one implementation that choice is much harder. At some point
>> down the road, when the bulk of the world's transactions rely on the
>> blockchain, there is likely to be significant controversy of such choices.
>
>
> Miners can only collude to make softforks. They cannot force people to
> accept as valid things that were previously not valid.
> They can mine a chain with other rules, sure, but everybody will ignore them.
> Nobody can push hardforks unilaterally, period.
> IMO this is not a fragility but a strength.
> I think you're talking about miners having their own policies or maybe
> about some other parts of the protocol different from the consensus
> code (which excludes p2p messages).

This is another point on which we seem to be talking past each other.

>> A good example is what is happening presently with the Swift network and
>> the associated saber-rattling. Who gets to make these choices? The
>> people who can control the network operators. Such operators are
>> relatively few and exist in regulated meatspace. They do what they must
>> to continue to do business, including cutting others off the network.
>>
>> Bitcoin can end up being no different. The majority of hashing power may
>> easily fall prey to the same attacks. The resilience of Bitcoin against
>> such attacks is in the broad distribution of mining.
>>
>> https://blockchain.info/pools
>
> I agree with this, though I'm not sure what this has to do with
> consensus rules...

Maybe this will help explain, especially the latter parts regarding
alternative implementations.

https://bitcoinmagazine.com/3668/bitcoin-network-shaken-by-blockchain-fork/

>> For an alternative consensus to arise in this likely future scenario,
>> the "unknown" miners need to be able to form their own consensus. This
>> requires credible alternative implementations and maintainers with
>> experience to support them and help drive towards agreement.
>>
>> As we see presently, for an implementation to be credible it needs to be
>> in-use. That can take a long time to develop, but ultimately it brings
>> more people to the table to guide such choices, and may even help to
>> forestall significant conflict down the road.
>
> Remember, no majority of miners can force users to accept hardfork
> changes to the consensus rules.

That statement doesn't make sense. If no majority can do it, then it
can't be done. You seem to believe that a hardfork requires universal
acceptance. This is not the case.

> Again I think you're talking about "choices" beyond consensus rules.
> For example, I think libbitcoin has more p2p messages for prefix-based
> utxo queries or something similar, and it probably has different
> policies than bitcoind's standard policy.
> That doesn't have to change.

That's not what I'm talking about.

>>> Please try to come up with an example of something that could go wrong
>>> with different bitcoin implementations sharing libconsensus code,
>>
>> It has already happened that a bug has been pushed out broadly,
>> requiring a temporary shut-down of the entire network. Imagine a
>> scenario where there are many disparate code bases. The impact of a
>> failure caused by any one of them is minimized.
>
> That happened because the consensus reference implementation was
> coupled with bitcoind's bdb store.
> libconsensus needs to be completely storage agnostic so that nothing
> like this can happen again.

The reason it happened is not relevant to my point. The fact that it
happened means that it can happen. It not happening again depends on the
theory of human infallibility, which has been discredited.

>>> it is very hard for me to argue about "decentralization" and "strength"
>>> in this abstract and vague way.
>>> Please explain to me in which ways libconsensus implies risks that
>>> libsecp256k1 doesn't, for the shake of your examples.
>>
>> The curve library implies the same technical risk, but is independent of
>> the political risk.
>
> There's no political risk on libbitcoin using libconsensus.
> libconsensus cannot be forced to upgrade so no new checks can be
> pushed onto libbitcoin.
> If, for example, libbitcoin wanted to implement a softfork change that
> the official distribution of libconsensus doesn't implement,
> libbitcoin can implement the additional checks outside of libconsensus
> (or just fork libconsensus).
> Seriously, I don't think there's any "political risk" in libconsensus.

Your understanding of hardforks and lack if visibility into the source
of the leverage that the Foundation has with miners seems to be why you
are missing the larger issue.

>>>>> I would at least suggest to use libconsensus for testing
>>>>> your script interpreter implementation and your signatures checks.
>>>>
>>>> Consider for a moment what this implies. In order to run any such
>>>> testing we need test vectors to run through both implementations. There
>>>> is no other way to perform such testing, and notably there is no need
>>>> for shared code.
>>>
>>> Well, even running random bitstrings in both interpreters would be
>>> useful, as any difference in results would be a symptom of an
>>> extremely dangerous bug.
>>
>> Fuzzing is simply a means for generating test vectors.
>
> Yes, and it is still not enough for having 100% certainty that
> libbitcoin consensus code is functionally identical to other nodes'
> consensus code.
> All I was saying here is than even libbitcoin depending on
> libconsensus only for tests would be something good.
>
>> I'm not arguing for a libconsensus replacement, just pointing out that
>> comparative testing is not an integration scenario.
>
> Sure, and I was just saying that since there's no test vectors that
> can cover all cases, running random test against libconsensus would
> increase the testing surface.

If some subset of tests change on each run, and the code changes as well
(otherwise why retest?) then we have added a moving window of tests over
a moving window of code. Granted this is an informal analysis, but you
could just as easily fix your randomly-generated tests and changing code
will likely cover the same increased surface.

Furthermore, given that we're talking about randomly changing random
tests over a practically infinite space, is practically zero percent of
the surface area.

Also, in order to be useful test results need to be repeatable, which
means there is a new requirement to be able to recover the random test
vectors for any run.

>> My "in this case" was a reference to the conformance testing (vs.
>> integration) scenario. In the integration scenario OpenSSL remains a
>> current technical issue, but ultimately it's not the central issue.
>
> I see.
>
>> My first recommendation is to extract libconsensus into an independent
>> git repository, as I have done with libbitcoin-consensus. As a rule, if
>> one wants to publish an API all consumers of that API should be treated
>> equally, including the "native" implementations. Otherwise the
>> implementer fails to fully appreciate situations faced by users.
>
> I agree. Hopefully will get there but a previous step is isolating all
> the consensus code within bitcoind (decoupling it from other
> functionality).

Yes, and this underscores my point. Once you pull it apart, move
libconsensus to it's own repo, and expose only the VerifyScript call,
you will find that you have had to maintain a significant portion of the
code that implements VerifyScript in both bitcoin and in libconsensus.

>> I think that you will very quickly recognize one of the core technical
>> problems with the current approach when you do this. You will be faced
>> with a choice to (1) dramatically increase the surface area of the API,
>> or (2) reimplement a large portion of the API in the calling code.
>
> I'm not sure I understand this point but as said libconsensus
> interface is still to be defined (and other bitcoin implementations
> different from bitcoind will get involved in defining it).

See above. The point is that once you try what you are asking us to do,
you may not like it.

>>> I think starting with VerifyScript it's very good because its
>>> interface was easier than it will be for fully checking transactions
>>> or blocks, for example (but please help fix verifyscript's interface
>>> if it's flawed).
>>> I believe that libbitcoin eventually depending on libconsensus will be
>>> very good for libconsensus because it needs more software depending on
>>> it than the reference implementation to make sense.
>>
>> This is true for libconsensus, but it certainly has value even without
>> external uptake. I think that factoring it out fully and cleanly would
>> make it much more attractive to other implementations.
>
> I agree that isolating the consensus code within bitcoind itself. My
> point is that building libconsensus separately and exposing an API
> makes no sense if only bitcoind will use it.

And it makes little sense for other libraries to use it if you don't.
Developing for end-users is different than developing for developers.

> Currently libbitcoin only exposes the VerifyScript() function. Do you
> think its interface is bitcoind-specific in any way?

No, aside from the language conventions the call is general.

> How would you improve it so that it's more "libbitcoin-friendly".

I would detangle it from bitcoin, move it to an independent repo,
back-integrate it as a lib into bitcoin, and remove redundant code
between the two implementations. I think that would dictate a fairly
broad API. Making that API friendly would likely require a common type
library between libbitcoin and bitcoin. That would likely lead to
interesting discussions about design philosophy. I'm not sure how that
would go. Likely an intermediate type layer would be necessary. Then
we'd start talking about marshalling costs and performance.

>> I think that the independent repo approach I gave above would be a good
>> starting point for identifying an interface.
>
> I think it's not so urgent. I mean, we still need at the very least a
> function to verify transactions and another to verify blocks. Provably
> each of them divided in several functions to give more flexibility to
> the different implementations to fight DoS attacks. Or for example,
> maybe one implementation wants to do some policy checks for
> transactions before verifying the scripts of its inputs (which, with
> signatures and all, it's computationally more epensive).
>
>> I am very much against the idea of conditionally compiling libbitcoin
>> source for various dependencies. We had several such conditions in the
>> build previously and they have all been removed.
>>
>> The proper way to switch dependencies is to define a common interface
>> and switch at link time. This could be implemented using an internal
>> stub for the interface, as secp256k1 did originally for OpenSSL and GMP
>> WRT big number. But that was only necessary because both are independent
>> and mature libs with heavily utilized public interfaces. In consensus
>> case we have the opportunity to define the interface. The problem may be
>> however the choice between a large interface and quite redundant code.
>
> I meant, for example a compile option designed only for testing that
> devs may use but not usd in the official distributions, like using
> ./configure --disable-wallet --without-gui for building bitcoind, for
> example.
> Something like --with-libconsensus-tests. Maybe this doesn't make
> sense in the context of libbitcoin, I don't know.

You are talking about how the switch is turned, I'm talking about how
it's implemented.

I would not want to see it integrated using conditionally-compiled
source code. As an external library with a redundant implementation to
our own, we would want to have a common interface and the ability to
link one or the other lib. But I'm just restating what I said previously
so I'm not sure if it helps.

> If you mean for using libconsensus or libbitcoin's equivalent code,
> sure, you need an interface and it's still being defined (we only have
> one function).
> Anyway, it is of course up to you guys to decide how to integrate
> libconsensus if you decide to do so.
> I'm just trying to convince you to decide to use it at some point.

>>>>> On Sat, Jan 24, 2015 at 12:05 AM, Eric Voskuil <eric@???> wrote:
>>>>>> Hi Jorje,
>>>>>>
>>>>>> There are several potential problems with libbitcoin taking a dependency
>>>>>> on a big chunk of the Satoshi implementation. However the most
>>>>>> straightforward to address is the OpenSSL dependency:
>>>>>>
>>>>>> https://github.com/bitcoin/bitcoin/blob/master/libbitcoinconsensus.pc.in#L11
>>>>>>
>>>>>> After spending months getting OpenSSL out of libbitcoin I couldn't
>>>>>> imagine bringing it back.
>>>>>>
>>>>>> We do use bitcoin/libsep256k1, which will eventually help in consensus
>>>>>> on crypto. And that compact C lib has recently removed all dependencies
>>>>>> (GMP is now optional and OpenSSL is only used for regression testing).
>>>>>>
>>>>>> I think it makes a lot more sense to verify consensus in the same way
>>>>>> libsecp256k1 is verified, through a comprehensive test matrix. We used
>>>>>> to perform testing against OpenSSL directly, now we have a large matrix
>>>>>> of test vectors that we generated from OpenSSL:
>>>>>>
>>>>>> https://github.com/libbitcoin/libbitcoin/blob/master/test/script_number.hpp
>>>>>>
>>>>>> Community-published and verified test vectors are reusable regardless of
>>>>>> language or implementation. As an example, our implementation for script
>>>>>> is here:
>>>>>>
>>>>>> https://github.com/libbitcoin/libbitcoin/blob/master/test/script.hpp#L33
>>>>>>
>>>>>> In fact we tend to use vectors from public sources for just this reason,
>>>>>> for example:
>>>>>>
>>>>>> https://github.com/libbitcoin/libbitcoin/blob/master/test/ec_keys.cpp#L55
>>>>>>
>>>>>> https://github.com/libbitcoin/libbitcoin-explorer/blob/master/test/commands/hd-private.cpp
>>>>>>
>>>>>> The best way to ensure consensus is for everyone to use the same
>>>>>> implementation. However this is ultimately at odds with the higher
>>>>>> objective of decentralization.
>>>>>>
>>>>>> e
>>>>>>
>>>>>> On 01/23/2015 12:50 PM, Jorge Timón wrote:
>>>>>>> If libbitcoin doesn't produce non-standard signatures it cannot create
>>>>>>> a fork as William says.
>>>>>>> The risk is that libbitcoin nodes get fork because they think that an
>>>>>>> invalid signature is valid.
>>>>>>> By using libconsensus (which is a dynamic library that 0.10 produces)
>>>>>>> you can avoid any danger with regard to these changes completely.
>>>>>>> Currently libconsensus only exposes a C function to verify a script in
>>>>>>> a transaction:
>>>>>>>
>>>>>>> https://github.com/bitcoin/bitcoin/blob/master/src/script/bitcoinconsensus.h#L55
>>>>>>>
>>>>>>> C was chosen so that it's easier to call from other languages like python.
>>>>>>> The rest of the consensus checks will have to keep being done by libbitcoin.
>>>>>>> Hopefully by 0.11 libconsensus will be able to check a full
>>>>>>> transaction, although the interface for the relevant utxo subset is
>>>>>>> not clear (I'm working on those code movements with so if you have
>>>>>>> suggestions on what parameters Consensus::CheckTxInputs() should take
>>>>>>> I would love to hear them).
>>>>>>> But not being able to fork due to signature or script checks is
>>>>>>> already a huge deal IMO.
>>>>>>> It is a small dependency (and will be smaller when libsecp256k1 is
>>>>>>> more mature and bitcoin core drops SSL for signature checking) and it
>>>>>>> is specially easy to use by C or C++ alternative implementations like
>>>>>>> libbitcoin.
>>>>>>>
>>>>>>> I can't find the where you check scripts in libbitcoin, only where you
>>>>>>> check signatures:
>>>>>>> https://github.com/libbitcoin/libbitcoin/search?utf8=%E2%9C%93&q=check_signature
>>>>>>> But it seems it's only used on the tests so maybe your VerifyScript()
>>>>>>> is in some other project that consumes libbitcoin?
>>>>>>>
>>>>>>> Anyway, I know it's not the only solution, but I highly recommend
>>>>>>> libconsensus as a dependency for any bitcoin alternative
>>>>>>> implementation that can afford to use it (like it's clearly the case
>>>>>>> for any implementation that already uses C or C++).
>>>>>>>
>>>>>>> I will be glad to help with this if you want to do it but I don't know
>>>>>>> libbitcoin's code.
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jan 23, 2015 at 3:04 AM, William Swanson <swansontec@???> wrote:
>>>>>>>> Libbitcoin will never produce non-standard signatures, so there is no
>>>>>>>> chance of us *creating* a fork.
>>>>>>>>
>>>>>>>> Besides this, there aren't any mining implementations running on top
>>>>>>>> of libbitcoin, so our consensus rules can't possibly lead to bad
>>>>>>>> blocks. The flip side is that we are at risk of being locked out if we
>>>>>>>> somehow reject a block that the rest of the network thinks is valid.
>>>>>>>>
>>>>>>>> I don't know if libsecp256k1 accepts non-standard signatures or not.
>>>>>>>> If libsecp256k1 rejects non-standard signatures, then we are already
>>>>>>>> in danger. If somebody mines a block with one of these signatures,
>>>>>>>> libbitcoin will reject the block and be unable to continue, even
>>>>>>>> though the rest of the network is fine. Hopefully this isn't the case.
>>>>>>>>
>>>>>>>> If libsecp256k1 is willing to accept non-strict signatures, then we
>>>>>>>> are fine in any case. If the network accepts this rule and becomes
>>>>>>>> more strict than us, we certainly won't disagree with any mined
>>>>>>>> blocks. This would need to be tightened up if we were to ever try
>>>>>>>> mining on top of libbitcoin, though.
>>>>>>>>
>>>>>>>> -William
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jan 22, 2015 at 2:39 PM, Noel Maersk <veox@???> wrote:
>>>>>>>>> As you know, 0.10 is around the corner, and I was idly wondering about
>>>>>>>>> libbitcoin's compatibility with that. Chances are a noticeable portion
>>>>>>>>> of the network will be switching to 0.10.
>>>>>>>>>
>>>>>>>>> gmaxwell has also asked on #darkwallet@freenode:
>>>>>>>>>
>>>>>>>>>> < gmaxwell> genjix: Care to review
>>>>>>>>>> http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg06744.html
>>>>>>>>>> < gmaxwell> ?
>>>>>>>>>
>>>>>>>>> I didn't review it myself yet, hence asking.
>>>>>>>>>
>>>>>>>>> The proposal is "to make non-DER signatures illegal (they've been
>>>>>>>>> non-standard since v0.8.0)".
>>>>>>>>>
>>>>>>>>> We're not using OpenSSL in the new libbitcoin-server. However, what
>>>>>>>>> about Obelisk, older libbitcoin versions, and code that relies on that?
>>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Libbitcoin mailing list
>>>> Libbitcoin@???
>>>> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/libbitcoin
>>>>
>>