Ethereum Block 10860844 & BC 0.9.63

DEVELOPERS: If you are testing BC as a developer, a community hosted snap shot is now available ( It is not recommended to sync with BC using this snapshot.

The Event

The BC multichain relies on rovers to connect to the blockchains that it consumes in order to build the multichain. Each rover only sends it’s block to the BC miner if the rover itself believes it is a valid and unique block. On Friday, September 18, 2020 7:16:11 AM GMT-04:00 the Ethereum Rover stopped transmitting blocks after Ethereum block #10860843 last seen in the multichain at multichain height 1771220

In addition to network and consensus checks, the Ethereum rover for the BC multichain runs three stages of validation on all inbound blocks. These stages are in addition to the validation requirements of the Ethereum foundation to assert the validity of blocks.

  1. Is the block constructed correctly?
  2. Is the block referencing it’s chain (uncles) correctly?
  3. Is the block generate a valid trie?

If a block fails any one of these it is rejected by the Ethereum rover and is never sent to the miner. It is an invalid block and therefore cannot be a part of the BC multichain.

At 7:21 AM GMT-04:00 (5 minutes following Ethereum #10860843) a monitoring system run by core began sending alerts that the Ethereum blockchain was no longer iterating on the multichain. This did not mean that the BC multichain was down, instead it meant that the new blocks being created were all referencing THE SAME Ethereum block. Essentially, BC was no longer adding Ethereum blocks. This was a critical issue, as Ethereum is included in the BC multichain due to its status as one of the most widely accepted representations of blockchain technology.

The Cause

Exactly 10 seconds after 10860843, #10860844 was accepted by the Ethereum network and REJECTED by BC Rovers. It is unclear why the Ethereum network accepted this block, as after several days of debugging it was determined the Patricia Merkle Trie (PMT) test fails on block #10860844.

The PMT test was tracked to a popular NPM package with over 100k downloads per week, used by Metamask and most in-browser Ethereum clients. After rewriting the PMT test manually, it was confirmed that as stated in the Ethereum documentation, the library was in fact CORRECT and the block itself does not pass. The PMT in reference can be found here:

The BC multichain continued as expected but without Ethereum in the chain, BC’s network difficulty was greatly affected. The Ethereum rover began to consume enormous amounts of memory as it searched the Ethereal network for thousands, then tens of thousands of blocks. The way the Rover was designed (fixed in 0.9.63) had all blocks stored in RAM until a correct block was found, which ultimately lead to BC miners crashing, restarting, and then never syncing with the Ethereum network.

The Solution

The solution was deployed in two stages:

  1. The first stage involved fixing the Ethereum Rover such that after a period of time it would accept the block as long as no valid alternative was discovered. The PMT test failed on the block and the Ethereum network had moved on and built on top this block. The Ethereum rover on BC needed to adapt to this behavior–both in memory management and in adopting the edge of the network. The updated Ethereum rover can also now support both ETH1 and ETH2.

  2. The second stage implemented this functionality on all other BC rovers (BTC, LSK, WAV, & NEO).

Additionally, as most of the miners were unable to even sync with Ethereum due to memory constraints, engineering took this opportunity to add low level performance upgrades to UTXO evaluation, syncing, peer evaluation and edge cases to P2P reported by miners and the community.

All of this has now been rolled into 0.9.63. The final testing can now continue for Borderless. It is highly encouraged to download the Borderless Client now and test very small trades or amounts.

Freedom through cryptography.