On 01/30/2015 08:18 AM, Jorge Timón wrote:
> On Fri, Jan 30, 2015 at 6:50 AM, Eric Voskuil <eric@???> wrote:
>> Hi Jorge,
>>
>> On 01/25/2015 05:18 AM, Jorge Timón wrote:
>>> It wouldn't be a big chunk of the bitcoin implementation but a minimal
>>> library extracted from the satoshi implementation.
>>
>> I suppose that we could differ on what constitutes large.
>>
>> It's a sizeable percentage of libbitcoin, essentially covering most of
>> the code outside of network, wallet, configuration and build.
>
> Well, I'm afraid we will have to disagree on what constitutes "most"
> of the code in bitcoin core.
> If you think the script interpreter and the transaction primitive is
> "most of the code outside of network, wallet, configuration and build"
> I'm sorry but I can only disagree.
> As said I really think that "most of the code outside of network,
> wallet, configuration and build" is not consensus related code but
> rather policy-related code.
> I'm only starting but please take, for example a look at this:
>
> https://github.com/jtimon/bitcoin/blob/consensus_policy/src/policy.cpp
> and https://github.com/jtimon/bitcoin/blob/consensus_policy/src/consensus/consensus.cpp
>
> 310 lines in policy.cpp vs 97 in consensus.cpp. Note that what is in
> consensus.cpp is NOT in libconsensus yet, because it's clear what the
> best interface should be. In that sense VerifyScript() was much
> simpler.
> Of course this is far from being a proof of my claims. But you're not
> proved yours either.

I said that it's a sizeable portion of *libbitcoin*.

>> ...
>>> Having many language agnostic test vector is great but you can never
>>> test all possible cases as libsecp256k1 developers very well know.
>>
>> This is of course an argument for implementation as standard. This isn't
>> IMO the best way to advance a standard. Generally there is formalization
>> based on the deliberation among interested parties, test cases and more
>> than one reference implementation. More about this below.
>
> Well, a "standard" for consensus-related code as you propose may be a
> good idea as well, I don't know.
> But bitcoin has unique requirements when it comes to consensus-related
> code acting identically and that's what libconsensus is about.

I'm not aware of any unique requirements. Many systems rely on
protocols/standards that if not properly implemented can lead to loss of
value. The purpose of protocols is to define these requirements so that
there can be multiple implementations.

>>> I disagree that libconsensus is against decentralization. Just like a
>>> wide use of libsecp256k1 is not against decentralization.
>>
>> I'm much less concerned about one of multiple implementations of
>> well-defined standards rooted in math than definition of an ultimately
>> subjective standard based on whatever gets implemented by single
>> organization.
>
> Consensus::VerifyScript() is rooted in math as well, it's just an
> interpreter of a language with crypto features.
> Ideally once completed libconsensus should be as said touched rarely
> and with many organizations involved in those changes, not just one.
> Ideally libconsensus would be the only part of bitcoin core that
> everybody cares about.
>
>>> Consensus failures don't increase decentralization in any way that I'm aware of.
>>
>> One implementation across the entire ecosystem certainly increases
>> centralization.
>
> I believe there will be still be reimpletations for software where
> having a C lib as a dependency is problematic, for example, javascript
> where the target platform may be an html5 interpreter.
> It is always very risky though since they have to be "bug by bug
> identical" to prevent consensus failures.

In which case the need for conformance testing remains and the reasoning
for a common implementation doesn't hold up.

>> If multiple implementations strengthen Bitcoin it's in large part
>> because there are multiple implementations of consensus.
>
> I completely disagree. It is because there are multiple
> implementations of everything else (which is most of the code IMO).
> For example, bitcoinj having its own interpreter doesn't "strengthen
> Bitcoin" in any way that I'm aware of, I believe it only introduces
> high risks that come from subtle differences between C and java.

It's very possible that higher level language implementations will catch
failures in what is considered the "reference" implementation.

OpenSSL has been the reference implementation for the curve, but it
turns out that there are problems that are only being discovered due to
alternative implementations.

>>> Besides, once finished libconsensus is expected to be very stable,
>>> only changing when consensus rules change (ie softforks or hardforks).
>>
>> Who decides when the rules are to change?
>
> Certainly not libconsensus maintainers.
> Miners decide in the case of softforks, everyone needs to agree in the
> case of hardforks.

It is not the case that everyone needs to agree. Bitcoin is not that
fragile. If it was it would not stand a chance of survival. Miners can
forking the code and doing what they want is the example of disagreement.

But with one implementation that choice is much harder. At some point
down the road, when the bulk of the world's transactions rely on the
blockchain, there is likely to be significant controversy of such choices.

A good example is what is happening presently with the Swift network and
the associated saber-rattling. Who gets to make these choices? The
people who can control the network operators. Such operators are
relatively few and exist in regulated meatspace. They do what they must
to continue to do business, including cutting others off the network.

Bitcoin can end up being no different. The majority of hashing power may
easily fall prey to the same attacks. The resilience of Bitcoin against
such attacks is in the broad distribution of mining.

https://blockchain.info/pools

For an alternative consensus to arise in this likely future scenario,
the "unknown" miners need to be able to form their own consensus. This
requires credible alternative implementations and maintainers with
experience to support them and help drive towards agreement.

As we see presently, for an implementation to be credible it needs to be
in-use. That can take a long time to develop, but ultimately it brings
more people to the table to guide such choices, and may even help to
forestall significant conflict down the road.

> Please try to come up with an example of something that could go wrong
> with different bitcoin implementations sharing libconsensus code,

It has already happened that a bug has been pushed out broadly,
requiring a temporary shut-down of the entire network. Imagine a
scenario where there are many disparate code bases. The impact of a
failure caused by any one of them is minimized.

> it is very hard for me to argue about "decentralization" and "strength"
> in this abstract and vague way.
> Please explain to me in which ways libconsensus implies risks that
> libsecp256k1 doesn't, for the shake of your examples.

The curve library implies the same technical risk, but is independent of
the political risk.

>>> I would at least suggest to use libconsensus for testing
>>> your script interpreter implementation and your signatures checks.
>>
>> Consider for a moment what this implies. In order to run any such
>> testing we need test vectors to run through both implementations. There
>> is no other way to perform such testing, and notably there is no need
>> for shared code.
>
> Well, even running random bitstrings in both interpreters would be
> useful, as any difference in results would be a symptom of an
> extremely dangerous bug.

Fuzzing is simply a means for generating test vectors.

> There is no "need" for shared code in a strict sense, of course.
> Libconsensus should just be a tool for bitcoin developers to shoot
> themselves in the foot less often.
> Of course you can reproduce all consensus checks in elisp or any
> language of your choice if there's no C compiler for your plattform or
> something.
>
>> The choice of vectors is important, and needs to be informed by an
>> understanding of design, including common and corner cases. We could
>> write a test harness to shove our own through libconsensus, but this
>> speaks to my earlier point.
>>
>> The community should ideally drive consensus around conformance test
>> cases and docs, which already happens in other areas. Each
>> implementation can then easily be tested for conformance with no need
>> for integration or for each implementation to test against another.
>
> As said tests are great but consensus code has unique risks that I
> don't think are comparable to other risks in other areas.
> In any case ,this sounds good to me, but it also sounds complementary
> to libconsensus, not a replacement.

I'm not arguing for a libconsensus replacement, just pointing out that
comparative testing is not an integration scenario.

>>> Bringing back OpenSSL via libconsus (thus temporarily) only for some
>>> tests that are optional at build time seems something very reasonable.
>>> When libconsensus script interpreter and libbitcoin's don't produce
>>> exactly the same results for the same inputs...something is probably
>>> wrong (a bug in one of them is a certainty in that case).
>>
>> Yes, but the question of OpenSSL is moot in this case because there is
>> no need for integration.
>
> Well, you argued that libbitcoin didn't wanted to depend again on
> openSSL (even though libsecp256k1's use for bitcoin signature checking
> is still discouraged by its developers) and I just said that depending
> on it only optionally at build time for testing is another option (an
> option that say javascript libraries could consider as welll).

My "in this case" was a reference to the conformance testing (vs.
integration) scenario. In the integration scenario OpenSSL remains a
current technical issue, but ultimately it's not the central issue.

>>> Anyway, If you're not interested in libconsensus at this point, I
>>> don't want to be a distraction you from the reimplementing the
>>> softfork proposal (which apparently in libbitcoin's case will just
>>> mean some code from a standard script check to the consensus script
>>> check) which is was this thread was about.
>>
>> No worries at all - it's not a distraction. I realize you put a good
>> amount of effort into libconsensus, and I think it's worthwhile. It will
>> make conformance testing quite a bit more straightforward, and certainly
>> some libs may integrate it. It also helps isolate change-sensitive code
>> from other.
>>
>> [merging split thread]
>>
>> On 01/29/2015 03:27 AM, Jorge Timón wrote:
>>> I'm curious, what is the purpose of libbitcoin-consensus?
>>> https://github.com/libbitcoin/libbitcoin-consensus
>>
>> When I first saw libconsensus being discussed I brought it up with Amir
>> and I offered to factor out our implementation as well, for the same
>> reasons I mention above. I didn't get to it until recently because it
>> wasn't a pressing issue, but I wanted to understand the volume of code
>> before commenting further here. So now it exists. Plans are to have
>> libbitcoin take a dependency on it in our 3.0 time frame.
>
> Isolating your own consensus checks seems a very reasonable start for
> taking libconsensus as a dependency, even if that wasn't your
> intention for any time frame.
> But thanks for explaining.
>
>>> It seems to me that having 100 different reimplementations of a
>>> consensus library defeats the entire purpose of avoiding hard to
>>> detect and subtle differences between implementations.
>>> If it was in js for example I could understand, but also in C++...
>>
>> Currently there are 2 robust implementations in C/C++ that I'm aware of.
>> But I'm not sure what purpose is being defeated. Libbitcoin's purpose is
>> not to provide an API on top of essential code from the Foundation. It
>> exists to provide a viable alternative.
>>
>>> Probably I'm missing something.
>>
>> https://bitcoinmagazine.com/6234/what-libbitcoin-and-sx-are-and-why-they-matter/
>
> Certainly the importance of alternative implementions and libbitcoin
> is not what I'm missing.
> I very much agree that having alternative implementations is healthy.
> In my mind libconsensus is only the small part of all the code where
> it is *not* healthy at all to have differences.
> Please point out anything that you see in libconsensus that you think
> doesn't belong there.
> In my opinion libconsensus itself should be more like libbitcoin in some senses.
> For example, having a single function for the whole interpreter makes
> it far less readable.
> I will reopen 5153 (now you can see only a very limited version of
> what it was) because each operator should have its own function like
> it does in libbitcoin.
> libconsensus doesn't mean that suddenly libbitcoin doesn't makes sense
> or anything like that.
> On the contrary, my hope was that libbitcoin could be an early user of
> libconsensus and could help define its interface and functionality.
> libbitcoin, as a full implementation of the bitcoin protocol can help
> to make sure that libconsensus is properly done and not coupled with
> bitcoind, which is my strongest concern for defining libconsensus's
> API.

My first recommendation is to extract libconsensus into an independent
git repository, as I have done with libbitcoin-consensus. As a rule, if
one wants to publish an API all consumers of that API should be treated
equally, including the "native" implementations. Otherwise the
implementer fails to fully appreciate situations faced by users.

I think that you will very quickly recognize one of the core technical
problems with the current approach when you do this. You will be faced
with a choice to (1) dramatically increase the surface area of the API,
or (2) reimplement a large portion of the API in the calling code.

> I think starting with VerifyScript it's very good because its
> interface was easier than it will be for fully checking transactions
> or blocks, for example (but please help fix verifyscript's interface
> if it's flawed).
> I believe that libbitcoin eventually depending on libconsensus will be
> very good for libconsensus because it needs more software depending on
> it than the reference implementation to make sense.

This is true for libconsensus, but it certainly has value even without
external uptake. I think that factoring it out fully and cleanly would
make it much more attractive to other implementations.

> It's an opportunity for consensus decisions to get more review from
> developers from other implementations without burdening them with the
> extra effort of filtering out bitcoin-qt-specific changes they may not
> care about or reimplementing the changes themselves if they don't want
> or need to.

Agree.

> Having different implementations with diverse features and policies is
> essential to make sure libconsensus is as minimal and agnostic as it
> should be.
> I believe depending on libconsensus could also be good for libbitcoin
> as it removes any chance of a consensus failure and more conservative
> users (say, miners) may consider libbitcoin safer for them.
> libbitcoin can always ignore libconsensus changes and implement
> different rules on top of it just like it can do with libsecp256k1 if
> the official distribution was compromised in any way.
> Keeping your own consensus checks in parallel is very reasonable and
> isolating the equivalent of the current libconsensus in
> libbitcoin-consensus like you've done it's something great IMO.
> Please don't take this as an attempt to force anything into libbitcoin
> but rather as a request for collaboration in the creation of
> libconsensus.

Absolutely, there is no problem. I appreciate you reaching out to
discuss it with us.

> Encapsulating consensus code in libbitcoin can provide you very useful
> insight that is very relevant to libconsensus.
> Currently there's little couplings between libconsensus and bitcoind
> (CFeeRate and CTxOut::IsDust(), for example) and it would be great if
> you guys could identify more things you don't want in libconsensus.
> Decoupling other consensus checks for transactions and blocks from
> bitcoind is more challenging since some of the relevant functions are
> currently coupled with bitcoind's utxo storage (CCoinsViewCache in
> coins.h).
> That obviously doesn't belong in libbitcoin, but as said it's not
> clear what the interface for those should exactly be.
> Help is welcomed and I think that by also isolating consensus checks
> (maybe you're closer to that than bitcoind) and potentially using
> libconsensus, libbitcoin will be able to provide it and make sure
> libconsensus has a clean interface.

I think that the independent repo approach I gave above would be a good
starting point for identifying an interface.

I am very much against the idea of conditionally compiling libbitcoin
source for various dependencies. We had several such conditions in the
build previously and they have all been removed.

The proper way to switch dependencies is to define a common interface
and switch at link time. This could be implemented using an internal
stub for the interface, as secp256k1 did originally for OpenSSL and GMP
WRT big number. But that was only necessary because both are independent
and mature libs with heavily utilized public interfaces. In consensus
case we have the opportunity to define the interface. The problem may be
however the choice between a large interface and quite redundant code.

> I'll write a post for bitcoin-dev mailing list about libconsensus
> explaining my view of libconsensus' present and future, asking for
> opinions and ideas.
>
>> e
>>
>>> On Sat, Jan 24, 2015 at 12:05 AM, Eric Voskuil <eric@???> wrote:
>>>> Hi Jorje,
>>>>
>>>> There are several potential problems with libbitcoin taking a dependency
>>>> on a big chunk of the Satoshi implementation. However the most
>>>> straightforward to address is the OpenSSL dependency:
>>>>
>>>> https://github.com/bitcoin/bitcoin/blob/master/libbitcoinconsensus.pc.in#L11
>>>>
>>>> After spending months getting OpenSSL out of libbitcoin I couldn't
>>>> imagine bringing it back.
>>>>
>>>> We do use bitcoin/libsep256k1, which will eventually help in consensus
>>>> on crypto. And that compact C lib has recently removed all dependencies
>>>> (GMP is now optional and OpenSSL is only used for regression testing).
>>>>
>>>> I think it makes a lot more sense to verify consensus in the same way
>>>> libsecp256k1 is verified, through a comprehensive test matrix. We used
>>>> to perform testing against OpenSSL directly, now we have a large matrix
>>>> of test vectors that we generated from OpenSSL:
>>>>
>>>> https://github.com/libbitcoin/libbitcoin/blob/master/test/script_number.hpp
>>>>
>>>> Community-published and verified test vectors are reusable regardless of
>>>> language or implementation. As an example, our implementation for script
>>>> is here:
>>>>
>>>> https://github.com/libbitcoin/libbitcoin/blob/master/test/script.hpp#L33
>>>>
>>>> In fact we tend to use vectors from public sources for just this reason,
>>>> for example:
>>>>
>>>> https://github.com/libbitcoin/libbitcoin/blob/master/test/ec_keys.cpp#L55
>>>>
>>>> https://github.com/libbitcoin/libbitcoin-explorer/blob/master/test/commands/hd-private.cpp
>>>>
>>>> The best way to ensure consensus is for everyone to use the same
>>>> implementation. However this is ultimately at odds with the higher
>>>> objective of decentralization.
>>>>
>>>> e
>>>>
>>>> On 01/23/2015 12:50 PM, Jorge Timón wrote:
>>>>> If libbitcoin doesn't produce non-standard signatures it cannot create
>>>>> a fork as William says.
>>>>> The risk is that libbitcoin nodes get fork because they think that an
>>>>> invalid signature is valid.
>>>>> By using libconsensus (which is a dynamic library that 0.10 produces)
>>>>> you can avoid any danger with regard to these changes completely.
>>>>> Currently libconsensus only exposes a C function to verify a script in
>>>>> a transaction:
>>>>>
>>>>> https://github.com/bitcoin/bitcoin/blob/master/src/script/bitcoinconsensus.h#L55
>>>>>
>>>>> C was chosen so that it's easier to call from other languages like python.
>>>>> The rest of the consensus checks will have to keep being done by libbitcoin.
>>>>> Hopefully by 0.11 libconsensus will be able to check a full
>>>>> transaction, although the interface for the relevant utxo subset is
>>>>> not clear (I'm working on those code movements with so if you have
>>>>> suggestions on what parameters Consensus::CheckTxInputs() should take
>>>>> I would love to hear them).
>>>>> But not being able to fork due to signature or script checks is
>>>>> already a huge deal IMO.
>>>>> It is a small dependency (and will be smaller when libsecp256k1 is
>>>>> more mature and bitcoin core drops SSL for signature checking) and it
>>>>> is specially easy to use by C or C++ alternative implementations like
>>>>> libbitcoin.
>>>>>
>>>>> I can't find the where you check scripts in libbitcoin, only where you
>>>>> check signatures:
>>>>> https://github.com/libbitcoin/libbitcoin/search?utf8=%E2%9C%93&q=check_signature
>>>>> But it seems it's only used on the tests so maybe your VerifyScript()
>>>>> is in some other project that consumes libbitcoin?
>>>>>
>>>>> Anyway, I know it's not the only solution, but I highly recommend
>>>>> libconsensus as a dependency for any bitcoin alternative
>>>>> implementation that can afford to use it (like it's clearly the case
>>>>> for any implementation that already uses C or C++).
>>>>>
>>>>> I will be glad to help with this if you want to do it but I don't know
>>>>> libbitcoin's code.
>>>>>
>>>>>
>>>>> On Fri, Jan 23, 2015 at 3:04 AM, William Swanson <swansontec@???> wrote:
>>>>>> Libbitcoin will never produce non-standard signatures, so there is no
>>>>>> chance of us *creating* a fork.
>>>>>>
>>>>>> Besides this, there aren't any mining implementations running on top
>>>>>> of libbitcoin, so our consensus rules can't possibly lead to bad
>>>>>> blocks. The flip side is that we are at risk of being locked out if we
>>>>>> somehow reject a block that the rest of the network thinks is valid.
>>>>>>
>>>>>> I don't know if libsecp256k1 accepts non-standard signatures or not.
>>>>>> If libsecp256k1 rejects non-standard signatures, then we are already
>>>>>> in danger. If somebody mines a block with one of these signatures,
>>>>>> libbitcoin will reject the block and be unable to continue, even
>>>>>> though the rest of the network is fine. Hopefully this isn't the case.
>>>>>>
>>>>>> If libsecp256k1 is willing to accept non-strict signatures, then we
>>>>>> are fine in any case. If the network accepts this rule and becomes
>>>>>> more strict than us, we certainly won't disagree with any mined
>>>>>> blocks. This would need to be tightened up if we were to ever try
>>>>>> mining on top of libbitcoin, though.
>>>>>>
>>>>>> -William
>>>>>>
>>>>>>
>>>>>> On Thu, Jan 22, 2015 at 2:39 PM, Noel Maersk <veox@???> wrote:
>>>>>>> As you know, 0.10 is around the corner, and I was idly wondering about
>>>>>>> libbitcoin's compatibility with that. Chances are a noticeable portion
>>>>>>> of the network will be switching to 0.10.
>>>>>>>
>>>>>>> gmaxwell has also asked on #darkwallet@freenode:
>>>>>>>
>>>>>>>> < gmaxwell> genjix: Care to review
>>>>>>>> http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg06744.html
>>>>>>>> < gmaxwell> ?
>>>>>>>
>>>>>>> I didn't review it myself yet, hence asking.
>>>>>>>
>>>>>>> The proposal is "to make non-DER signatures illegal (they've been
>>>>>>> non-standard since v0.8.0)".
>>>>>>>
>>>>>>> We're not using OpenSSL in the new libbitcoin-server. However, what
>>>>>>> about Obelisk, older libbitcoin versions, and code that relies on that?
>>>>
>>
>>
>> _______________________________________________
>> Libbitcoin mailing list
>> Libbitcoin@???
>> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/libbitcoin
>>