{"id":20756,"date":"2018-12-18T21:48:51","date_gmt":"2018-12-18T21:48:51","guid":{"rendered":"http:\/\/ci027cfe70c00c26c3"},"modified":"2018-12-18T21:48:51","modified_gmt":"2018-12-18T21:48:51","slug":"synching-data-between-bitcoin-nodes-about-get-easier","status":"publish","type":"post","link":"https:\/\/bitcoinmagazine.com\/technical\/synching-data-between-bitcoin-nodes-about-get-easier","title":{"rendered":"Syncing Data Between Bitcoin Nodes Is About to Get Easier"},"content":{"rendered":"
<\/div>
<\/figure>\n

Minisketch<\/a> is a new solution that\u2019s trying to solve an old problem.<\/p>\n

Spearheaded by Blockstream co-founder Pieter Wuille, Bitcoin Core contributor and fellow Blockstream co-founder Gregory Maxwell, and Blockstream software engineer Gleb Naumenko, the open-source initiative is designed to achieve set reconciliation between the mempools of each full node. <\/p>\n

\u201cSet reconciliation, in short, is the problem of trying to figure out what the differences are between two sets stored on different computers, while minimizing how much data needs to be exchanged between them. In particular, it\u2019s trying to do so while sending less data than the entire set of data,\u201d Wuille told Bitcoin Magazine<\/em><\/p>\n

For Bitcoin, this means discerning the differences in transaction data between nodes. Maxwell likened set reconciliation to the process of syncing your phone\u2019s contact list with another person who has many of the same contacts.<\/p>\n

\u201cYou could send them your whole list but it won\u2019t fit on a postcard and would be pretty wasteful in any case, since they already know most of the contacts \u2026 It is possible, in fact, to communicate your whole set of contacts to them by sending only as much information as the size of the difference between your lists even without any idea in advance of what the actual differences are,\u201d Maxwell told Bitcoin Magazine<\/em>.<\/p>\n

In short, set reconciliation would reduce the bandwidth required to run a full node on the Bitcoin network by minimizing how much data each node transmits to each other. This would effectively allow nodes to sync up the data in their mempools more efficiently.<\/p>\n

Breaking Down Set Reconciliation<\/h3>\n

The problem that minisketch wants to amend isn\u2019t blockchain specific.<\/p>\n

Set reconciliation is a constant frustration that any distributed system grapples with. Writ large, it simply means that two or more parties on a distributed network hold different sets of data, and to reconcile this, they must figure out which pieces of the data they\u2019re missing \u2014 as well as which pieces they have that the other party lacks.<\/p>\n

For Bitcoin, these data are transactions. These transactions are relayed from node to node until they are picked up by miners to be included into new blocks.<\/p>\n

The problem is that the order of transactions can vary from mempool to mempool (the queues that list incoming transactions to be included into new blocks). This means that there can be (and usually is) a discrepancy in the order of transactions between mempools and newly relayed blocks.<\/p>\n

\u201cBitcoin nodes have a \u2026 problem when relaying transactions between each other. Any given node will have mostly the same transactions as any one of its peers \u2014 received from other links, but not exactly the same. Today nodes waste a lot of bandwidth just figuring out who needs to be sent what [data],\u201d Maxwell said.<\/p>\n

How Minisketch Closes the Gap<\/h3>\n

An implementation of the PinSketch algorithm, minisketch constructs a library of data to be used for constructing set sketches (i.e., for this use case, sets of transaction data). Nodes and miners can then use these sets for compact set reconciliation.<\/p>\n

Simply put, the solution would allow node operators to compare notes on transaction data. It would allow them to sketch (create) sets (lists) of transactions, and the program would cross-check these sets to see which data occurs in one but not both sets. Instead of spending time and energy revealing all of this data to each other, though, with minisketch, nodes only need to know the difference between their transaction sets to sketch a full set.<\/p>\n

As Wuille explained, it would look like this in practice:<\/p>\n

\u201cIf we simplify it to just a single difference, it\u2019s easy to see how this may work:<\/p>\n