On Fri, Jan 30, 2015 at 6:50 AM, Eric Voskuil <eric@???> wrote:
> Hi Jorge,
>
> On 01/25/2015 05:18 AM, Jorge Timón wrote:
>> It wouldn't be a big chunk of the bitcoin implementation but a minimal
>> library extracted from the satoshi implementation.
>
> I suppose that we could differ on what constitutes large.
>
> It's a sizeable percentage of libbitcoin, essentially covering most of
> the code outside of network, wallet, configuration and build.

Well, I'm afraid we will have to disagree on what constitutes "most"
of the code in bitcoin core.
If you think the script interpreter and the transaction primitive is
"most of the code outside of network, wallet, configuration and build"
I'm sorry but I can only disagree.
As said I really think that "most of the code outside of network,
wallet, configuration and build" is not consensus related code but
rather policy-related code.
I'm only starting but please take, for example a look at this:

https://github.com/jtimon/bitcoin/blob/consensus_policy/src/policy.cpp
and https://github.com/jtimon/bitcoin/blob/consensus_policy/src/consensus/consensus.cpp

310 lines in policy.cpp vs 97 in consensus.cpp. Note that what is in
consensus.cpp is NOT in libconsensus yet, because it's clear what the
best interface should be. In that sense VerifyScript() was much
simpler.
Of course this is far from being a proof of my claims. But you're not
proved yours either.

> ...
>> Having many language agnostic test vector is great but you can never
>> test all possible cases as libsecp256k1 developers very well know.
>
> This is of course an argument for implementation as standard. This isn't
> IMO the best way to advance a standard. Generally there is formalization
> based on the deliberation among interested parties, test cases and more
> than one reference implementation. More about this below.

Well, a "standard" for consensus-related code as you propose may be a
good idea as well, I don't know.
But bitcoin has unique requirements when it comes to consensus-related
code acting identically and that's what libconsensus is about.

>> I disagree that libconsensus is against decentralization. Just like a
>> wide use of libsecp256k1 is not against decentralization.
>
> I'm much less concerned about one of multiple implementations of
> well-defined standards rooted in math than definition of an ultimately
> subjective standard based on whatever gets implemented by single
> organization.

Consensus::VerifyScript() is rooted in math as well, it's just an
interpreter of a language with crypto features.
Ideally once completed libconsensus should be as said touched rarely
and with many organizations involved in those changes, not just one.
Ideally libconsensus would be the only part of bitcoin core that
everybody cares about.

>> Consensus failures don't increase decentralization in any way that I'm aware of.
>
> One implementation across the entire ecosystem certainly increases
> centralization.

I believe there will be still be reimpletations for software where
having a C lib as a dependency is problematic, for example, javascript
where the target platform may be an html5 interpreter.
It is always very risky though since they have to be "bug by bug
identical" to prevent consensus failures.

> If multiple implementations strengthen Bitcoin it's in large part
> because there are multiple implementations of consensus.

I completely disagree. It is because there are multiple
implementations of everything else (which is most of the code IMO).
For example, bitcoinj having its own interpreter doesn't "strengthen
Bitcoin" in any way that I'm aware of, I believe it only introduces
high risks that come from subtle differences between C and java.

>> Besides, once finished libconsensus is expected to be very stable,
>> only changing when consensus rules change (ie softforks or hardforks).
>
> Who decides when the rules are to change?

Certainly not libconsensus maintainers.
Miners decide in the case of softforks, everyone needs to agree in the
case of hardforks.
Please try to come up with an example of something that could go wrong
with different bitcoin implementations sharing libconsensus code, it
is very hard for me to argue about "decentralization" and "strength"
in this abstract and vague way.
Please explain to me in which ways libconsensus implies risks that
libsecp256k1 doesn't, for the shake of your examples.

>> I would at least suggest to use libconsensus for testing
>> your script interpreter implementation and your signatures checks.
>
> Consider for a moment what this implies. In order to run any such
> testing we need test vectors to run through both implementations. There
> is no other way to perform such testing, and notably there is no need
> for shared code.

Well, even running random bitstrings in both interpreters would be
useful, as any difference in results would be a symptom of an
extremely dangerous bug.
There is no "need" for shared code in a strict sense, of course.
Libconsensus should just be a tool for bitcoin developers to shoot
themselves in the foot less often.
Of course you can reproduce all consensus checks in elisp or any
language of your choice if there's no C compiler for your plattform or
something.

> The choice of vectors is important, and needs to be informed by an
> understanding of design, including common and corner cases. We could
> write a test harness to shove our own through libconsensus, but this
> speaks to my earlier point.
>
> The community should ideally drive consensus around conformance test
> cases and docs, which already happens in other areas. Each
> implementation can then easily be tested for conformance with no need
> for integration or for each implementation to test against another.

As said tests are great but consensus code has unique risks that I
don't think are comparable to other risks in other areas.
In any case ,this sounds good to me, but it also sounds complementary
to libconsensus, not a replacement.

>> Bringing back OpenSSL via libconsus (thus temporarily) only for some
>> tests that are optional at build time seems something very reasonable.
>> When libconsensus script interpreter and libbitcoin's don't produce
>> exactly the same results for the same inputs...something is probably
>> wrong (a bug in one of them is a certainty in that case).
>
> Yes, but the question of OpenSSL is moot in this case because there is
> no need for integration.

Well, you argued that libbitcoin didn't wanted to depend again on
openSSL (even though libsecp256k1's use for bitcoin signature checking
is still discouraged by its developers) and I just said that depending
on it only optionally at build time for testing is another option (an
option that say javascript libraries could consider as welll).

>> Anyway, If you're not interested in libconsensus at this point, I
>> don't want to be a distraction you from the reimplementing the
>> softfork proposal (which apparently in libbitcoin's case will just
>> mean some code from a standard script check to the consensus script
>> check) which is was this thread was about.
>
> No worries at all - it's not a distraction. I realize you put a good
> amount of effort into libconsensus, and I think it's worthwhile. It will
> make conformance testing quite a bit more straightforward, and certainly
> some libs may integrate it. It also helps isolate change-sensitive code
> from other.
>
> [merging split thread]
>
> On 01/29/2015 03:27 AM, Jorge Timón wrote:
>> I'm curious, what is the purpose of libbitcoin-consensus?
>> https://github.com/libbitcoin/libbitcoin-consensus
>
> When I first saw libconsensus being discussed I brought it up with Amir
> and I offered to factor out our implementation as well, for the same
> reasons I mention above. I didn't get to it until recently because it
> wasn't a pressing issue, but I wanted to understand the volume of code
> before commenting further here. So now it exists. Plans are to have
> libbitcoin take a dependency on it in our 3.0 time frame.

Isolating your own consensus checks seems a very reasonable start for
taking libconsensus as a dependency, even if that wasn't your
intention for any time frame.
But thanks for explaining.

>> It seems to me that having 100 different reimplementations of a
>> consensus library defeats the entire purpose of avoiding hard to
>> detect and subtle differences between implementations.
>> If it was in js for example I could understand, but also in C++...
>
> Currently there are 2 robust implementations in C/C++ that I'm aware of.
> But I'm not sure what purpose is being defeated. Libbitcoin's purpose is
> not to provide an API on top of essential code from the Foundation. It
> exists to provide a viable alternative.
>
>> Probably I'm missing something.
>
> https://bitcoinmagazine.com/6234/what-libbitcoin-and-sx-are-and-why-they-matter/

Certainly the importance of alternative implementions and libbitcoin
is not what I'm missing.
I very much agree that having alternative implementations is healthy.
In my mind libconsensus is only the small part of all the code where
it is *not* healthy at all to have differences.
Please point out anything that you see in libconsensus that you think
doesn't belong there.
In my opinion libconsensus itself should be more like libbitcoin in some senses.
For example, having a single function for the whole interpreter makes
it far less readable.
I will reopen 5153 (now you can see only a very limited version of
what it was) because each operator should have its own function like
it does in libbitcoin.
libconsensus doesn't mean that suddenly libbitcoin doesn't makes sense
or anything like that.
On the contrary, my hope was that libbitcoin could be an early user of
libconsensus and could help define its interface and functionality.
libbitcoin, as a full implementation of the bitcoin protocol can help
to make sure that libconsensus is properly done and not coupled with
bitcoind, which is my strongest concern for defining libconsensus's
API.
I think starting with VerifyScript it's very good because its
interface was easier than it will be for fully checking transactions
or blocks, for example (but please help fix verifyscript's interface
if it's flawed).
I believe that libbitcoin eventually depending on libconsensus will be
very good for libconsensus because it needs more software depending on
it than the reference implementation to make sense.
It's an opportunity for consensus decisions to get more review from
developers from other implementations without burdening them with the
extra effort of filtering out bitcoin-qt-specific changes they may not
care about or reimplementing the changes themselves if they don't want
or need to.
Having different implementations with diverse features and policies is
essential to make sure libconsensus is as minimal and agnostic as it
should be.
I believe depending on libconsensus could also be good for libbitcoin
as it removes any chance of a consensus failure and more conservative
users (say, miners) may consider libbitcoin safer for them.
libbitcoin can always ignore libconsensus changes and implement
different rules on top of it just like it can do with libsecp256k1 if
the official distribution was compromised in any way.
Keeping your own consensus checks in parallel is very reasonable and
isolating the equivalent of the current libconsensus in
libbitcoin-consensus like you've done it's something great IMO.
Please don't take this as an attempt to force anything into libbitcoin
but rather as a request for collaboration in the creation of
libconsensus.
Encapsulating consensus code in libbitcoin can provide you very useful
insight that is very relevant to libconsensus.
Currently there's little couplings between libconsensus and bitcoind
(CFeeRate and CTxOut::IsDust(), for example) and it would be great if
you guys could identify more things you don't want in libconsensus.
Decoupling other consensus checks for transactions and blocks from
bitcoind is more challenging since some of the relevant functions are
currently coupled with bitcoind's utxo storage (CCoinsViewCache in
coins.h).
That obviously doesn't belong in libbitcoin, but as said it's not
clear what the interface for those should exactly be.
Help is welcomed and I think that by also isolating consensus checks
(maybe you're closer to that than bitcoind) and potentially using
libconsensus, libbitcoin will be able to provide it and make sure
libconsensus has a clean interface.

I'll write a post for bitcoin-dev mailing list about libconsensus
explaining my view of libconsensus' present and future, asking for
opinions and ideas.

> e
>
>> On Sat, Jan 24, 2015 at 12:05 AM, Eric Voskuil <eric@???> wrote:
>>> Hi Jorje,
>>>
>>> There are several potential problems with libbitcoin taking a dependency
>>> on a big chunk of the Satoshi implementation. However the most
>>> straightforward to address is the OpenSSL dependency:
>>>
>>> https://github.com/bitcoin/bitcoin/blob/master/libbitcoinconsensus.pc.in#L11
>>>
>>> After spending months getting OpenSSL out of libbitcoin I couldn't
>>> imagine bringing it back.
>>>
>>> We do use bitcoin/libsep256k1, which will eventually help in consensus
>>> on crypto. And that compact C lib has recently removed all dependencies
>>> (GMP is now optional and OpenSSL is only used for regression testing).
>>>
>>> I think it makes a lot more sense to verify consensus in the same way
>>> libsecp256k1 is verified, through a comprehensive test matrix. We used
>>> to perform testing against OpenSSL directly, now we have a large matrix
>>> of test vectors that we generated from OpenSSL:
>>>
>>> https://github.com/libbitcoin/libbitcoin/blob/master/test/script_number.hpp
>>>
>>> Community-published and verified test vectors are reusable regardless of
>>> language or implementation. As an example, our implementation for script
>>> is here:
>>>
>>> https://github.com/libbitcoin/libbitcoin/blob/master/test/script.hpp#L33
>>>
>>> In fact we tend to use vectors from public sources for just this reason,
>>> for example:
>>>
>>> https://github.com/libbitcoin/libbitcoin/blob/master/test/ec_keys.cpp#L55
>>>
>>> https://github.com/libbitcoin/libbitcoin-explorer/blob/master/test/commands/hd-private.cpp
>>>
>>> The best way to ensure consensus is for everyone to use the same
>>> implementation. However this is ultimately at odds with the higher
>>> objective of decentralization.
>>>
>>> e
>>>
>>> On 01/23/2015 12:50 PM, Jorge Timón wrote:
>>>> If libbitcoin doesn't produce non-standard signatures it cannot create
>>>> a fork as William says.
>>>> The risk is that libbitcoin nodes get fork because they think that an
>>>> invalid signature is valid.
>>>> By using libconsensus (which is a dynamic library that 0.10 produces)
>>>> you can avoid any danger with regard to these changes completely.
>>>> Currently libconsensus only exposes a C function to verify a script in
>>>> a transaction:
>>>>
>>>> https://github.com/bitcoin/bitcoin/blob/master/src/script/bitcoinconsensus.h#L55
>>>>
>>>> C was chosen so that it's easier to call from other languages like python.
>>>> The rest of the consensus checks will have to keep being done by libbitcoin.
>>>> Hopefully by 0.11 libconsensus will be able to check a full
>>>> transaction, although the interface for the relevant utxo subset is
>>>> not clear (I'm working on those code movements with so if you have
>>>> suggestions on what parameters Consensus::CheckTxInputs() should take
>>>> I would love to hear them).
>>>> But not being able to fork due to signature or script checks is
>>>> already a huge deal IMO.
>>>> It is a small dependency (and will be smaller when libsecp256k1 is
>>>> more mature and bitcoin core drops SSL for signature checking) and it
>>>> is specially easy to use by C or C++ alternative implementations like
>>>> libbitcoin.
>>>>
>>>> I can't find the where you check scripts in libbitcoin, only where you
>>>> check signatures:
>>>> https://github.com/libbitcoin/libbitcoin/search?utf8=%E2%9C%93&q=check_signature
>>>> But it seems it's only used on the tests so maybe your VerifyScript()
>>>> is in some other project that consumes libbitcoin?
>>>>
>>>> Anyway, I know it's not the only solution, but I highly recommend
>>>> libconsensus as a dependency for any bitcoin alternative
>>>> implementation that can afford to use it (like it's clearly the case
>>>> for any implementation that already uses C or C++).
>>>>
>>>> I will be glad to help with this if you want to do it but I don't know
>>>> libbitcoin's code.
>>>>
>>>>
>>>> On Fri, Jan 23, 2015 at 3:04 AM, William Swanson <swansontec@???> wrote:
>>>>> Libbitcoin will never produce non-standard signatures, so there is no
>>>>> chance of us *creating* a fork.
>>>>>
>>>>> Besides this, there aren't any mining implementations running on top
>>>>> of libbitcoin, so our consensus rules can't possibly lead to bad
>>>>> blocks. The flip side is that we are at risk of being locked out if we
>>>>> somehow reject a block that the rest of the network thinks is valid.
>>>>>
>>>>> I don't know if libsecp256k1 accepts non-standard signatures or not.
>>>>> If libsecp256k1 rejects non-standard signatures, then we are already
>>>>> in danger. If somebody mines a block with one of these signatures,
>>>>> libbitcoin will reject the block and be unable to continue, even
>>>>> though the rest of the network is fine. Hopefully this isn't the case.
>>>>>
>>>>> If libsecp256k1 is willing to accept non-strict signatures, then we
>>>>> are fine in any case. If the network accepts this rule and becomes
>>>>> more strict than us, we certainly won't disagree with any mined
>>>>> blocks. This would need to be tightened up if we were to ever try
>>>>> mining on top of libbitcoin, though.
>>>>>
>>>>> -William
>>>>>
>>>>>
>>>>> On Thu, Jan 22, 2015 at 2:39 PM, Noel Maersk <veox@???> wrote:
>>>>>> As you know, 0.10 is around the corner, and I was idly wondering about
>>>>>> libbitcoin's compatibility with that. Chances are a noticeable portion
>>>>>> of the network will be switching to 0.10.
>>>>>>
>>>>>> gmaxwell has also asked on #darkwallet@freenode:
>>>>>>
>>>>>>> < gmaxwell> genjix: Care to review
>>>>>>> http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg06744.html
>>>>>>> < gmaxwell> ?
>>>>>>
>>>>>> I didn't review it myself yet, hence asking.
>>>>>>
>>>>>> The proposal is "to make non-DER signatures illegal (they've been
>>>>>> non-standard since v0.8.0)".
>>>>>>
>>>>>> We're not using OpenSSL in the new libbitcoin-server. However, what
>>>>>> about Obelisk, older libbitcoin versions, and code that relies on that?
>>>
>
>
> _______________________________________________
> Libbitcoin mailing list
> Libbitcoin@???
> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/libbitcoin
>