- Improved Initial Block Download (IBD) Performance
- Faster new block validation and relay
- Optional fee bumping
- Hashes for verification
We are pleased to release Bitcoin Core 0.14.0, which significantly speeds up the processing of historic blocks by newly started nodes and the validation and relay of new blocks by online nodes. An optional new feature (disabled by default) is also provided to allow wallet users to rebroadcast one of their previously-sent unconfirmed transactions with a higher fee, which may allow it to confirm more quickly.
A short summary of the major new features is provided below, with links to more details later in this post.
Improved Initial Block Download (IBD) performance: a full node that has been started for the first time can now validate all blocks up until the present faster than before. This is at least the sixth time IBD performance has been significantly improved by developers to help ensure that new users aren’t discouraged from running a full node because synchronizing the ever-growing block chain takes too long.
Faster new block validation and relay: miners in particular will benefit from improvements made in four existing Bitcoin Core features. The signature cache has been upgraded for machines with many CPU cores—a test on a system with 16 cores shows a 40% speed increase in processing a new block. The BIP152 CompactBlocks implementation will now relay some blocks before they’ve been fully validated, allowing those blocks to propagate across the peer-to-peer (P2P) network faster than before. The P2P network code has also generally been refactored to allow multiple actions to happen at the same time (concurrency) as well as to increase throughput, eliminating many potential delays in processing new blocks. Finally, unconfirmed transactions in each node’s mempool can now be saved to and restored from disk when the node is restarted.
Optional fee bumping: Bitcoin Core’s wallet now allows users to optionally send transactions that opt-in to fee-based transaction replacement. This allows users to increase the fee their transactions pay even after they’ve broadcast an earlier version of the transaction. This feature is disabled by default.
Improved Initial Block Download (IBD) Performance
Over time, the constantly-increasing size of the block chain has forced new first-time nodes to process larger and large amounts of data before they can be used with a wallet to receive and send payments. Many previous Bitcoin and Bitcoin Core releases have included major improvements designed to eliminate the pain of this Initial Block Download (IBD), also called the initial sync. For example:
|Release||Major improvements in IBD|
|0.5.0||Skip verification of historic (checkpointed) signatures|
|0.8.0||Switch to LevelDB & parallel signature validation|
|0.10.0||Headers-first sync and parallel block download|
|0.11.0||Optional block file pruning to save disk space|
|0.12.0||New fast signature validation library written from scratch (libsecp256k1)|
|0.13.1||Segwit to allow skipping download of historic signatures in the future|
This 0.14.0 release includes two more improvements that significantly increase IBD speed:
- Assumed valid blocks
- Memory-sharing between mempool and UTXO DB cache
Those two optimizations are described in more detail below. A test of the speed of the previous release (Bitcoin Core 0.13.2) compared to the speed of this Bitcoin Core 0.14.0 release was performed using Amazon EC2 virtual private servers, type
t2.xlarge with four cores and 16 GB memory at a cost of $0.188 USD per hour (excluding block storage costs). All Bitcoin Core settings were left at their defaults.
Bitcoin Core 0.13.2 took 1 day, 12 hours, and 40 minutes to complete IBD for a total cost of $6.89.
Bitcoin Core 0.14.0 took 0 days, 6 hours, and 24 minutes to complete IBD for a total cost of $1.20.
Under these testing conditions, Bitcoin Core 0.14.0 was 5.7 times faster at IBD than the previous version of Bitcoin Core.
Note: Bitcoin Core’s defaults are chosen to allow it to run on a large variety of hardware, including older machines with small amounts of memory. For users who want a faster sync than six hours, consider starting Bitcoin Core with the
-dbcache= option higher than its
300 (MB) default. Many modern computers should be able to sync in about 3 hours using Bitcoin Core 0.14.0 and 8 GB of memory (
Assumed valid blocks
Bitcoin 0.3.2 introduced a mechanism called checkpoints to prevent denial-of-service attacks during IBD by ensuring that new full nodes couldn’t be tricked into spending excessive amounts of effort validating alternative block chains that were different than the best-known chain at certain points in time.
Bitcoin 0.5.0 built upon those checkpoints to speed up IBD by skipping verification of signatures in blocks that were earlier in the block chain than the most recent checkpoint.
Over time, other improvements in Bitcoin Core’s security (such as headers-first sync and minimum chainwork) has reduced the need for checkpoints, while at the same time many Bitcoin developers have expressed the desire to remove checkpoints because they confuse new developers about the security model—checkpoints make it look like Bitcoin developers are deciding which chain is valid even though the intent was always to only accept the chain which the software had already decided was valid.
Assumed valid blocks is a new feature that separates the signature-skipping optimization from the checkpoints anti-denial-of-service mechanism so they can each be dealt with independently.
An assumed valid block is a block that individual users consider to be valid, including being part of a valid block chain. This is easy for anyone to test in a completely repeatable (deterministic) way: take any consensus-compatible Bitcoin implementation and run it on the whole block chain including the block in question. If the software doesn’t reject the block or any preceding block, it’s valid.
If someone who starts a new full node for the first time knows about any valid blocks, they can then provide the highest-height one of those blocks to Bitcoin Core 0.14.0 and the software will skip verifying signatures in the blocks before the assumed valid block. Since verifying signatures consumes a lot of CPU during IBD, using assumed valid blocks can significantly speed up IBD. All blocks after the assumed valid block will still have their signatures checked normally.
A critical difference between checkpoints and assumed valid blocks is that Bitcoin 0.3.2 required all checkpointed blocks to be part of the block chain but Bitcoin Core 0.14.0 does not require any assumed valid blocks to be part of the chain. If no assumed valid block provided by the user (or the system defaults) is part of the block chain, Bitcoin Core will simply verify all signatures for historic blocks. Or, if there’s a block chain fork and the valid block chain with the most proof of work doesn’t include a known assumed valid block, Bitcoin Core will still switch to the new best block chain even though it means abandoning the assumed valid block.
New users to Bitcoin probably won’t know about any valid blocks, but they probably also won’t know all the consensus rules—so they can simply use the copy of the full node software they download. Bitcoin Core 0.14.0 ships with a default assumed valid block that is set during the release process by having multiple well-known developers each confirm that a certain block is known to be valid by them (example).
Anyone who wants to verify all signatures using Bitcoin Core can still do so by starting the program using
-assumevalid=0. Anyone who wants to specify an alternative assumed valid block can specify the block identifier as a parameter to
assumevalid; for example:
The default assumed valid block in Bitcoin Core 0.14.0 is #453354, 16
Februrary 2017, with hash
Memory-sharing between mempool and UTXO DB cache
During IBD, Bitcoin Core doesn’t use it’s mempool because until you have
the most recent blocks there’s no way to verify most recently-created
transactions. This means that the memory allocated to the mempool
has historically simply been unallocated, meaning Bitcoin Core ran with
less memory during IBD than it did normally.
In Bitcoin Core 0.14.0, unused mempool memory is shared with the UTXO
database cache, which increases the number of Unspent Transaction
Outputs (UTXOs) that can be cached in fast memory rather than needing to
be stored and retrieved from a much slower disk drive.
Faster new block validation and relay
Four significant improvements in Bitcoin Core 0.14.0 will be especially interesting to miners and other users who need to receive and process new blocks as fast as possible.
Improved signature cache
The first feature is an update of the signature cache to use cuckoo hashing. The signature cache allows Bitcoin Core to save (cache) the result of verifying signatures in an unconfirmed transaction so that when the same transaction appears in a new block, Bitcoin Core doesn’t need to verify the signatures again. Because signature verification is typically the most computationally-expensive part of processing a new block, using the signature cache significantly improves the speed at which new blocks can be processed by nodes that have been online for a while.
The existing signature cache in Bitcoin Core 0.13.2 and below works well for systems with fewer than eight CPU cores, but for systems with eight or more CPU cores a bottleneck (lock contention) appears that prevents the system from using all available cores as effectively as possible. The upgrade to using cuckoo hashing to provide a “cuckoo cache” eliminates this problem and allows more cores to be used effectively.
In a test on a system with 16 cores, a new block was able to be added (connected) to the block chain 40% faster using 0.14.0 than a previous version of Bitcoin Core. For systems with fewer than 8 cores, there is no major performance increase, although the cuckoo cache does allow caching more signatures than before for the same amount of memory, so there can be a slight performance improvement.
Earlier BIP152 compactblock relay
The second feature improved in Bitcoin Core 0.14.0 is the BIP152 compactblock implementation. The previous implementation supported both of BIP152’s two opt-in modes:
A low-bandwidth mode that attempts to send the minimum data necessary to relay a new block, including waiting for the receiving node to request
that specific new block.
A high-bandwidth mode that sends new block data without waiting for the receiving node to request that specific block. This risks sending the receiving node the same data that another node sent it—a waste of bandwidth—but helps ensure that blocks are transferred quickly.
The upgraded implementation enhances the high-bandwidth node by starting the relay of a new block before the block has been fully validated. In the best case, removing the validation delay can allow new blocks to propagate across multiple hops on the peer-to-peer network several times faster than they could before. In the worst case, some additional bandwidth is wasted transferring invalid blocks. In either case, the security model remains the same since all nodes will still reject invalid blocks.
P2P code refactor focused on concurrency and throughput
The third feature improved in Bitcoin Core 0.14.0 is the entire set of P2P networking code, which has been refactored to support increased concurrency and throughput. The concurrency improvements help allow newly-received blocks (such as BIP152 compactblocks) to be processed ahead of lower-priority traffic, ensuring that blocks are relayed and validated as fast as possible.
The refactor also now allows network activity to continue in the background during message processing, notably providing an improvement in IBD speed that complements the headers-first sync introduced in Bitcoin Core 0.10.0 (see above for more information).
Mempool saved to disk
A fourth feature which helps support the signature cache and compactblocks implementations is that the memory pool (mempool) of unconfirmed transactions received by each node is now saved to disk during regular shutdown, and is then loaded back into memory when the node starts back up.
Combined with compact blocks, this can save the node from having to re-download all those unconfirmed transactions when they are received in a newly-produced block. Combined with the signature cache, this allows the node to cache the signature verification of those unconfirmed transactions so that new blocks including those transactions can be validated more quickly.
Optional fee bumping
An optional feature in Bitcoin Core 0.14.0 (disabled by default) causes all new transactions produced by the the wallet to opt-in to transaction replacement, specifically BIP125 opt-in Replace By Fee (RBF).
When this feature is enabled by starting Bitcoin Core with the
-walletrbf option, an additional RPC (
bumpfee) is made available that allows a previously-sent unconfirmed transaction to be resent with a higher fee. Miners who support either opt-in RBF or full RBF will usually replace the lower-fee transaction with the higher-fee transaction in their queues, and the higher fee will encourage miners to mine the new version of the transaction more quickly.
The next major planned release will be scheduled for approximately six months after the 0.14.0 release (but it will not be released until fully tested).
If you are interested in contributing to Bitcoin Core, please see our contributing page and the document How to contribute code to Bitcoin Core. If you don’t know where to get started or have any other questions, please stop by our IRC chatroom and we’ll do our best to help you.
Hashes for verification
These are the SHA-256 hashes of the released files: