On Sat, Jan 31, 2015 at 12:05 AM, Eric Voskuil <eric@???> wrote:
>> Well, a "standard" for consensus-related code as you propose may be a
>> good idea as well, I don't know.
>> But bitcoin has unique requirements when it comes to consensus-related
>> code acting identically and that's what libconsensus is about.
>
> I'm not aware of any unique requirements. Many systems rely on
> protocols/standards that if not properly implemented can lead to loss of
> value. The purpose of protocols is to define these requirements so that
> there can be multiple implementations.

What defines the chain you listen to (you can think of this as a
communication channel) is the genesis block, the consensus rules and
only after that the proof of work.
If your consensus checks are different from anyone else's you can end
up listening to a different chain and then...well...people can cheat
you.
I believe this property of the system is quite unique.
There can be multiple implementations of the protocol (which includes
many non-consensus things like the p2p messages) and that's healthy.
But all implementations must be functionally identical when it comes
to consensus checks, period.

>>>> I disagree that libconsensus is against decentralization. Just like a
>>>> wide use of libsecp256k1 is not against decentralization.
>>>
>>> I'm much less concerned about one of multiple implementations of
>>> well-defined standards rooted in math than definition of an ultimately
>>> subjective standard based on whatever gets implemented by single
>>> organization.
>>
>> Consensus::VerifyScript() is rooted in math as well, it's just an
>> interpreter of a language with crypto features.
>> Ideally once completed libconsensus should be as said touched rarely
>> and with many organizations involved in those changes, not just one.
>> Ideally libconsensus would be the only part of bitcoin core that
>> everybody cares about.
>>
>>>> Consensus failures don't increase decentralization in any way that I'm aware of.
>>>
>>> One implementation across the entire ecosystem certainly increases
>>> centralization.
>>
>> I believe there will be still be reimpletations for software where
>> having a C lib as a dependency is problematic, for example, javascript
>> where the target platform may be an html5 interpreter.
>> It is always very risky though since they have to be "bug by bug
>> identical" to prevent consensus failures.
>
> In which case the need for conformance testing remains and the reasoning
> for a common implementation doesn't hold up.

Again, tests can't cover all cases. There's simply too many possible cases.
It's not that a common implementation for consensus code is *needed*
but I believe it can be useful for many implementations and in the
case of libbitcoin I still cannot see any good reason not to depend on
it in the long term.

>>> If multiple implementations strengthen Bitcoin it's in large part
>>> because there are multiple implementations of consensus.
>>
>> I completely disagree. It is because there are multiple
>> implementations of everything else (which is most of the code IMO).
>> For example, bitcoinj having its own interpreter doesn't "strengthen
>> Bitcoin" in any way that I'm aware of, I believe it only introduces
>> high risks that come from subtle differences between C and java.
>
> It's very possible that higher level language implementations will catch
> failures in what is considered the "reference" implementation.

I still don't think that's the reason why having multiple
implementations of the protocol is healthy.
And I don't think you can extend that to consensus code.
In fact, I think a clean libconsensus could be a great formal
specification of the consensus rules.

> OpenSSL has been the reference implementation for the curve, but it
> turns out that there are problems that are only being discovered due to
> alternative implementations.

I don't think secp256k1 is not a high priority for OpenSSL and
certainly bitcoin has special properties when it comes to signatures.
Checking signatures differently can produce consensus failures!
Those are some of the reasons for bitcoind to move to libsecp256k1.
I still don't understand why you think libbitcoin using libconsensus
is a bad idea but using libsecp256k1 instead of reimplementing your
own library.
The main difference in my mind is that libsecp256k1 may actually get
some non-bitcoin related uses, but that's about it.

>>>> Besides, once finished libconsensus is expected to be very stable,
>>>> only changing when consensus rules change (ie softforks or hardforks).
>>>
>>> Who decides when the rules are to change?
>>
>> Certainly not libconsensus maintainers.
>> Miners decide in the case of softforks, everyone needs to agree in the
>> case of hardforks.
>
> It is not the case that everyone needs to agree. Bitcoin is not that
> fragile. If it was it would not stand a chance of survival. Miners can
> forking the code and doing what they want is the example of disagreement.
>
> But with one implementation that choice is much harder. At some point
> down the road, when the bulk of the world's transactions rely on the
> blockchain, there is likely to be significant controversy of such choices.


Miners can only collude to make softforks. They cannot force people to
accept as valid things that were previously not valid.
They can mine a chain with other rules, sure, but everybody will ignore them.
Nobody can push hardforks unilaterally, period.
IMO this is not a fragility but a strength.
I think you're talking about miners having their own policies or maybe
about some other parts of the protocol different from the consensus
code (which excludes p2p messages).

> A good example is what is happening presently with the Swift network and
> the associated saber-rattling. Who gets to make these choices? The
> people who can control the network operators. Such operators are
> relatively few and exist in regulated meatspace. They do what they must
> to continue to do business, including cutting others off the network.
>
> Bitcoin can end up being no different. The majority of hashing power may
> easily fall prey to the same attacks. The resilience of Bitcoin against
> such attacks is in the broad distribution of mining.
>
> https://blockchain.info/pools

I agree with this, though I'm not sure what this has to do with
consensus rules...

> For an alternative consensus to arise in this likely future scenario,
> the "unknown" miners need to be able to form their own consensus. This
> requires credible alternative implementations and maintainers with
> experience to support them and help drive towards agreement.
>
> As we see presently, for an implementation to be credible it needs to be
> in-use. That can take a long time to develop, but ultimately it brings
> more people to the table to guide such choices, and may even help to
> forestall significant conflict down the road.

Remember, no majority of miners can force users to accept hardfork
changes to the consensus rules.
Again I think you're talking about "choices" beyond consensus rules.
For example, I think libbitcoin has more p2p messages for prefix-based
utxo queries or something similar, and it probably has different
policies than bitcoind's standard policy.
That doesn't have to change.

>> Please try to come up with an example of something that could go wrong
>> with different bitcoin implementations sharing libconsensus code,
>
> It has already happened that a bug has been pushed out broadly,
> requiring a temporary shut-down of the entire network. Imagine a
> scenario where there are many disparate code bases. The impact of a
> failure caused by any one of them is minimized.

That happened because the consensus reference implementation was
coupled with bitcoind's bdb store.
libconsensus needs to be completely storage agnostic so that nothing
like this can happen again.

>> it is very hard for me to argue about "decentralization" and "strength"
>> in this abstract and vague way.
>> Please explain to me in which ways libconsensus implies risks that
>> libsecp256k1 doesn't, for the shake of your examples.
>
> The curve library implies the same technical risk, but is independent of
> the political risk.

There's no political risk on libbitcoin using libconsensus.
libconsensus cannot be forced to upgrade so no new checks can be
pushed onto libbitcoin.
If, for example, libbitcoin wanted to implement a softfork change that
the official distribution of libconsensus doesn't implement,
libbitcoin can implement the additional checks outside of libconsensus
(or just fork libconsensus).
Seriously, I don't think there's any "political risk" in libconsensus.

>>>> I would at least suggest to use libconsensus for testing
>>>> your script interpreter implementation and your signatures checks.
>>>
>>> Consider for a moment what this implies. In order to run any such
>>> testing we need test vectors to run through both implementations. There
>>> is no other way to perform such testing, and notably there is no need
>>> for shared code.
>>
>> Well, even running random bitstrings in both interpreters would be
>> useful, as any difference in results would be a symptom of an
>> extremely dangerous bug.
>
> Fuzzing is simply a means for generating test vectors.

Yes, and it is still not enough for having 100% certainty that
libbitcoin consensus code is functionally identical to other nodes'
consensus code.
All I was saying here is than even libbitcoin depending on
libconsensus only for tests would be something good.

> I'm not arguing for a libconsensus replacement, just pointing out that
> comparative testing is not an integration scenario.

Sure, and I was just saying that since there's no test vectors that
can cover all cases, running random test against libconsensus would
increase the testing surface.

> My "in this case" was a reference to the conformance testing (vs.
> integration) scenario. In the integration scenario OpenSSL remains a
> current technical issue, but ultimately it's not the central issue.

I see.

> My first recommendation is to extract libconsensus into an independent
> git repository, as I have done with libbitcoin-consensus. As a rule, if
> one wants to publish an API all consumers of that API should be treated
> equally, including the "native" implementations. Otherwise the
> implementer fails to fully appreciate situations faced by users.

I agree. Hopefully will get there but a previous step is isolating all
the consensus code within bitcoind (decoupling it from other
functionality).

> I think that you will very quickly recognize one of the core technical
> problems with the current approach when you do this. You will be faced
> with a choice to (1) dramatically increase the surface area of the API,
> or (2) reimplement a large portion of the API in the calling code.

I'm not sure I understand this point but as said libconsensus
interface is still to be defined (and other bitcoin implementations
different from bitcoind will get involved in defining it).

>> I think starting with VerifyScript it's very good because its
>> interface was easier than it will be for fully checking transactions
>> or blocks, for example (but please help fix verifyscript's interface
>> if it's flawed).
>> I believe that libbitcoin eventually depending on libconsensus will be
>> very good for libconsensus because it needs more software depending on
>> it than the reference implementation to make sense.
>
> This is true for libconsensus, but it certainly has value even without
> external uptake. I think that factoring it out fully and cleanly would
> make it much more attractive to other implementations.

I agree that isolating the consensus code within bitcoind itself. My
point is that building libconsensus separately and exposing an API
makes no sense if only bitcoind will use it.
Currently libbitcoin only exposes the VerifyScript() function. Do you
think its interface is bitcoind-specific in any way?
How would you improve it so that it's more "libbitcoin-friendly".

> I think that the independent repo approach I gave above would be a good
> starting point for identifying an interface.

I think it's not so urgent. I mean, we still need at the very least a
function to verify transactions and another to verify blocks. Provably
each of them divided in several functions to give more flexibility to
the different implementations to fight DoS attacks. Or for example,
maybe one implementation wants to do some policy checks for
transactions before verifying the scripts of its inputs (which, with
signatures and all, it's computationally more epensive).

> I am very much against the idea of conditionally compiling libbitcoin
> source for various dependencies. We had several such conditions in the
> build previously and they have all been removed.
>
> The proper way to switch dependencies is to define a common interface
> and switch at link time. This could be implemented using an internal
> stub for the interface, as secp256k1 did originally for OpenSSL and GMP
> WRT big number. But that was only necessary because both are independent
> and mature libs with heavily utilized public interfaces. In consensus
> case we have the opportunity to define the interface. The problem may be
> however the choice between a large interface and quite redundant code.

I meant, for example a compile option designed only for testing that
devs may use but not usd in the official distributions, like using
./configure --disable-wallet --without-gui for building bitcoind, for
example.
Something like --with-libconsensus-tests. Maybe this doesn't make
sense in the context of libbitcoin, I don't know.
If you mean for using libconsensus or libbitcoin's equivalent code,
sure, you need an interface and it's still being defined (we only have
one function).
Anyway, it is of course up to you guys to decide how to integrate
libconsensus if you decide to do so.
I'm just trying to convince you to decide to use it at some point.

>>>> On Sat, Jan 24, 2015 at 12:05 AM, Eric Voskuil <eric@???> wrote:
>>>>> Hi Jorje,
>>>>>
>>>>> There are several potential problems with libbitcoin taking a dependency
>>>>> on a big chunk of the Satoshi implementation. However the most
>>>>> straightforward to address is the OpenSSL dependency:
>>>>>
>>>>> https://github.com/bitcoin/bitcoin/blob/master/libbitcoinconsensus.pc.in#L11
>>>>>
>>>>> After spending months getting OpenSSL out of libbitcoin I couldn't
>>>>> imagine bringing it back.
>>>>>
>>>>> We do use bitcoin/libsep256k1, which will eventually help in consensus
>>>>> on crypto. And that compact C lib has recently removed all dependencies
>>>>> (GMP is now optional and OpenSSL is only used for regression testing).
>>>>>
>>>>> I think it makes a lot more sense to verify consensus in the same way
>>>>> libsecp256k1 is verified, through a comprehensive test matrix. We used
>>>>> to perform testing against OpenSSL directly, now we have a large matrix
>>>>> of test vectors that we generated from OpenSSL:
>>>>>
>>>>> https://github.com/libbitcoin/libbitcoin/blob/master/test/script_number.hpp
>>>>>
>>>>> Community-published and verified test vectors are reusable regardless of
>>>>> language or implementation. As an example, our implementation for script
>>>>> is here:
>>>>>
>>>>> https://github.com/libbitcoin/libbitcoin/blob/master/test/script.hpp#L33
>>>>>
>>>>> In fact we tend to use vectors from public sources for just this reason,
>>>>> for example:
>>>>>
>>>>> https://github.com/libbitcoin/libbitcoin/blob/master/test/ec_keys.cpp#L55
>>>>>
>>>>> https://github.com/libbitcoin/libbitcoin-explorer/blob/master/test/commands/hd-private.cpp
>>>>>
>>>>> The best way to ensure consensus is for everyone to use the same
>>>>> implementation. However this is ultimately at odds with the higher
>>>>> objective of decentralization.
>>>>>
>>>>> e
>>>>>
>>>>> On 01/23/2015 12:50 PM, Jorge Timón wrote:
>>>>>> If libbitcoin doesn't produce non-standard signatures it cannot create
>>>>>> a fork as William says.
>>>>>> The risk is that libbitcoin nodes get fork because they think that an
>>>>>> invalid signature is valid.
>>>>>> By using libconsensus (which is a dynamic library that 0.10 produces)
>>>>>> you can avoid any danger with regard to these changes completely.
>>>>>> Currently libconsensus only exposes a C function to verify a script in
>>>>>> a transaction:
>>>>>>
>>>>>> https://github.com/bitcoin/bitcoin/blob/master/src/script/bitcoinconsensus.h#L55
>>>>>>
>>>>>> C was chosen so that it's easier to call from other languages like python.
>>>>>> The rest of the consensus checks will have to keep being done by libbitcoin.
>>>>>> Hopefully by 0.11 libconsensus will be able to check a full
>>>>>> transaction, although the interface for the relevant utxo subset is
>>>>>> not clear (I'm working on those code movements with so if you have
>>>>>> suggestions on what parameters Consensus::CheckTxInputs() should take
>>>>>> I would love to hear them).
>>>>>> But not being able to fork due to signature or script checks is
>>>>>> already a huge deal IMO.
>>>>>> It is a small dependency (and will be smaller when libsecp256k1 is
>>>>>> more mature and bitcoin core drops SSL for signature checking) and it
>>>>>> is specially easy to use by C or C++ alternative implementations like
>>>>>> libbitcoin.
>>>>>>
>>>>>> I can't find the where you check scripts in libbitcoin, only where you
>>>>>> check signatures:
>>>>>> https://github.com/libbitcoin/libbitcoin/search?utf8=%E2%9C%93&q=check_signature
>>>>>> But it seems it's only used on the tests so maybe your VerifyScript()
>>>>>> is in some other project that consumes libbitcoin?
>>>>>>
>>>>>> Anyway, I know it's not the only solution, but I highly recommend
>>>>>> libconsensus as a dependency for any bitcoin alternative
>>>>>> implementation that can afford to use it (like it's clearly the case
>>>>>> for any implementation that already uses C or C++).
>>>>>>
>>>>>> I will be glad to help with this if you want to do it but I don't know
>>>>>> libbitcoin's code.
>>>>>>
>>>>>>
>>>>>> On Fri, Jan 23, 2015 at 3:04 AM, William Swanson <swansontec@???> wrote:
>>>>>>> Libbitcoin will never produce non-standard signatures, so there is no
>>>>>>> chance of us *creating* a fork.
>>>>>>>
>>>>>>> Besides this, there aren't any mining implementations running on top
>>>>>>> of libbitcoin, so our consensus rules can't possibly lead to bad
>>>>>>> blocks. The flip side is that we are at risk of being locked out if we
>>>>>>> somehow reject a block that the rest of the network thinks is valid.
>>>>>>>
>>>>>>> I don't know if libsecp256k1 accepts non-standard signatures or not.
>>>>>>> If libsecp256k1 rejects non-standard signatures, then we are already
>>>>>>> in danger. If somebody mines a block with one of these signatures,
>>>>>>> libbitcoin will reject the block and be unable to continue, even
>>>>>>> though the rest of the network is fine. Hopefully this isn't the case.
>>>>>>>
>>>>>>> If libsecp256k1 is willing to accept non-strict signatures, then we
>>>>>>> are fine in any case. If the network accepts this rule and becomes
>>>>>>> more strict than us, we certainly won't disagree with any mined
>>>>>>> blocks. This would need to be tightened up if we were to ever try
>>>>>>> mining on top of libbitcoin, though.
>>>>>>>
>>>>>>> -William
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jan 22, 2015 at 2:39 PM, Noel Maersk <veox@???> wrote:
>>>>>>>> As you know, 0.10 is around the corner, and I was idly wondering about
>>>>>>>> libbitcoin's compatibility with that. Chances are a noticeable portion
>>>>>>>> of the network will be switching to 0.10.
>>>>>>>>
>>>>>>>> gmaxwell has also asked on #darkwallet@freenode:
>>>>>>>>
>>>>>>>>> < gmaxwell> genjix: Care to review
>>>>>>>>> http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg06744.html
>>>>>>>>> < gmaxwell> ?
>>>>>>>>
>>>>>>>> I didn't review it myself yet, hence asking.
>>>>>>>>
>>>>>>>> The proposal is "to make non-DER signatures illegal (they've been
>>>>>>>> non-standard since v0.8.0)".
>>>>>>>>
>>>>>>>> We're not using OpenSSL in the new libbitcoin-server. However, what
>>>>>>>> about Obelisk, older libbitcoin versions, and code that relies on that?
>>>>>
>>>
>>>
>>> _______________________________________________
>>> Libbitcoin mailing list
>>> Libbitcoin@???
>>> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/libbitcoin
>>>
>