Hi Jorge,

On 01/25/2015 05:18 AM, Jorge Timón wrote:
> It wouldn't be a big chunk of the bitcoin implementation but a minimal
> library extracted from the satoshi implementation.

I suppose that we could differ on what constitutes large.

It's a sizeable percentage of libbitcoin, essentially covering most of
the code outside of network, wallet, configuration and build.

...
> Having many language agnostic test vector is great but you can never
> test all possible cases as libsecp256k1 developers very well know.

This is of course an argument for implementation as standard. This isn't
IMO the best way to advance a standard. Generally there is formalization
based on the deliberation among interested parties, test cases and more
than one reference implementation. More about this below.

> I disagree that libconsensus is against decentralization. Just like a
> wide use of libsecp256k1 is not against decentralization.

I'm much less concerned about one of multiple implementations of
well-defined standards rooted in math than definition of an ultimately
subjective standard based on whatever gets implemented by single
organization.

> Consensus failures don't increase decentralization in any way that I'm aware of.

One implementation across the entire ecosystem certainly increases
centralization.

If multiple implementations strengthen Bitcoin it's in large part
because there are multiple implementations of consensus.

> Besides, once finished libconsensus is expected to be very stable,
> only changing when consensus rules change (ie softforks or hardforks).

Who decides when the rules are to change?

> If you still want to keep reimplementing all consensus validations
> yourselves

I consider it "implementing" :).

> I would at least suggest to use libconsensus for testing
> your script interpreter implementation and your signatures checks.

Consider for a moment what this implies. In order to run any such
testing we need test vectors to run through both implementations. There
is no other way to perform such testing, and notably there is no need
for shared code.

The choice of vectors is important, and needs to be informed by an
understanding of design, including common and corner cases. We could
write a test harness to shove our own through libconsensus, but this
speaks to my earlier point.

The community should ideally drive consensus around conformance test
cases and docs, which already happens in other areas. Each
implementation can then easily be tested for conformance with no need
for integration or for each implementation to test against another.

> Bringing back OpenSSL via libconsus (thus temporarily) only for some
> tests that are optional at build time seems something very reasonable.
> When libconsensus script interpreter and libbitcoin's don't produce
> exactly the same results for the same inputs...something is probably
> wrong (a bug in one of them is a certainty in that case).

Yes, but the question of OpenSSL is moot in this case because there is
no need for integration.

> Anyway, If you're not interested in libconsensus at this point, I
> don't want to be a distraction you from the reimplementing the
> softfork proposal (which apparently in libbitcoin's case will just
> mean some code from a standard script check to the consensus script
> check) which is was this thread was about.

No worries at all - it's not a distraction. I realize you put a good
amount of effort into libconsensus, and I think it's worthwhile. It will
make conformance testing quite a bit more straightforward, and certainly
some libs may integrate it. It also helps isolate change-sensitive code
from other.

[merging split thread]

On 01/29/2015 03:27 AM, Jorge Timón wrote:
> I'm curious, what is the purpose of libbitcoin-consensus?
> https://github.com/libbitcoin/libbitcoin-consensus

When I first saw libconsensus being discussed I brought it up with Amir
and I offered to factor out our implementation as well, for the same
reasons I mention above. I didn't get to it until recently because it
wasn't a pressing issue, but I wanted to understand the volume of code
before commenting further here. So now it exists. Plans are to have
libbitcoin take a dependency on it in our 3.0 time frame.

> It seems to me that having 100 different reimplementations of a
> consensus library defeats the entire purpose of avoiding hard to
> detect and subtle differences between implementations.
> If it was in js for example I could understand, but also in C++...

Currently there are 2 robust implementations in C/C++ that I'm aware of.
But I'm not sure what purpose is being defeated. Libbitcoin's purpose is
not to provide an API on top of essential code from the Foundation. It
exists to provide a viable alternative.

> Probably I'm missing something.

https://bitcoinmagazine.com/6234/what-libbitcoin-and-sx-are-and-why-they-matter/

e

> On Sat, Jan 24, 2015 at 12:05 AM, Eric Voskuil <eric@???> wrote:
>> Hi Jorje,
>>
>> There are several potential problems with libbitcoin taking a dependency
>> on a big chunk of the Satoshi implementation. However the most
>> straightforward to address is the OpenSSL dependency:
>>
>> https://github.com/bitcoin/bitcoin/blob/master/libbitcoinconsensus.pc.in#L11
>>
>> After spending months getting OpenSSL out of libbitcoin I couldn't
>> imagine bringing it back.
>>
>> We do use bitcoin/libsep256k1, which will eventually help in consensus
>> on crypto. And that compact C lib has recently removed all dependencies
>> (GMP is now optional and OpenSSL is only used for regression testing).
>>
>> I think it makes a lot more sense to verify consensus in the same way
>> libsecp256k1 is verified, through a comprehensive test matrix. We used
>> to perform testing against OpenSSL directly, now we have a large matrix
>> of test vectors that we generated from OpenSSL:
>>
>> https://github.com/libbitcoin/libbitcoin/blob/master/test/script_number.hpp
>>
>> Community-published and verified test vectors are reusable regardless of
>> language or implementation. As an example, our implementation for script
>> is here:
>>
>> https://github.com/libbitcoin/libbitcoin/blob/master/test/script.hpp#L33
>>
>> In fact we tend to use vectors from public sources for just this reason,
>> for example:
>>
>> https://github.com/libbitcoin/libbitcoin/blob/master/test/ec_keys.cpp#L55
>>
>> https://github.com/libbitcoin/libbitcoin-explorer/blob/master/test/commands/hd-private.cpp
>>
>> The best way to ensure consensus is for everyone to use the same
>> implementation. However this is ultimately at odds with the higher
>> objective of decentralization.
>>
>> e
>>
>> On 01/23/2015 12:50 PM, Jorge Timón wrote:
>>> If libbitcoin doesn't produce non-standard signatures it cannot create
>>> a fork as William says.
>>> The risk is that libbitcoin nodes get fork because they think that an
>>> invalid signature is valid.
>>> By using libconsensus (which is a dynamic library that 0.10 produces)
>>> you can avoid any danger with regard to these changes completely.
>>> Currently libconsensus only exposes a C function to verify a script in
>>> a transaction:
>>>
>>> https://github.com/bitcoin/bitcoin/blob/master/src/script/bitcoinconsensus.h#L55
>>>
>>> C was chosen so that it's easier to call from other languages like python.
>>> The rest of the consensus checks will have to keep being done by libbitcoin.
>>> Hopefully by 0.11 libconsensus will be able to check a full
>>> transaction, although the interface for the relevant utxo subset is
>>> not clear (I'm working on those code movements with so if you have
>>> suggestions on what parameters Consensus::CheckTxInputs() should take
>>> I would love to hear them).
>>> But not being able to fork due to signature or script checks is
>>> already a huge deal IMO.
>>> It is a small dependency (and will be smaller when libsecp256k1 is
>>> more mature and bitcoin core drops SSL for signature checking) and it
>>> is specially easy to use by C or C++ alternative implementations like
>>> libbitcoin.
>>>
>>> I can't find the where you check scripts in libbitcoin, only where you
>>> check signatures:
>>> https://github.com/libbitcoin/libbitcoin/search?utf8=%E2%9C%93&q=check_signature
>>> But it seems it's only used on the tests so maybe your VerifyScript()
>>> is in some other project that consumes libbitcoin?
>>>
>>> Anyway, I know it's not the only solution, but I highly recommend
>>> libconsensus as a dependency for any bitcoin alternative
>>> implementation that can afford to use it (like it's clearly the case
>>> for any implementation that already uses C or C++).
>>>
>>> I will be glad to help with this if you want to do it but I don't know
>>> libbitcoin's code.
>>>
>>>
>>> On Fri, Jan 23, 2015 at 3:04 AM, William Swanson <swansontec@???> wrote:
>>>> Libbitcoin will never produce non-standard signatures, so there is no
>>>> chance of us *creating* a fork.
>>>>
>>>> Besides this, there aren't any mining implementations running on top
>>>> of libbitcoin, so our consensus rules can't possibly lead to bad
>>>> blocks. The flip side is that we are at risk of being locked out if we
>>>> somehow reject a block that the rest of the network thinks is valid.
>>>>
>>>> I don't know if libsecp256k1 accepts non-standard signatures or not.
>>>> If libsecp256k1 rejects non-standard signatures, then we are already
>>>> in danger. If somebody mines a block with one of these signatures,
>>>> libbitcoin will reject the block and be unable to continue, even
>>>> though the rest of the network is fine. Hopefully this isn't the case.
>>>>
>>>> If libsecp256k1 is willing to accept non-strict signatures, then we
>>>> are fine in any case. If the network accepts this rule and becomes
>>>> more strict than us, we certainly won't disagree with any mined
>>>> blocks. This would need to be tightened up if we were to ever try
>>>> mining on top of libbitcoin, though.
>>>>
>>>> -William
>>>>
>>>>
>>>> On Thu, Jan 22, 2015 at 2:39 PM, Noel Maersk <veox@???> wrote:
>>>>> As you know, 0.10 is around the corner, and I was idly wondering about
>>>>> libbitcoin's compatibility with that. Chances are a noticeable portion
>>>>> of the network will be switching to 0.10.
>>>>>
>>>>> gmaxwell has also asked on #darkwallet@freenode:
>>>>>
>>>>>> < gmaxwell> genjix: Care to review
>>>>>> http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg06744.html
>>>>>> < gmaxwell> ?
>>>>>
>>>>> I didn't review it myself yet, hence asking.
>>>>>
>>>>> The proposal is "to make non-DER signatures illegal (they've been
>>>>> non-standard since v0.8.0)".
>>>>>
>>>>> We're not using OpenSSL in the new libbitcoin-server. However, what
>>>>> about Obelisk, older libbitcoin versions, and code that relies on that?
>>