Solana Halted by Bug Linked to Certain Cold Storage Transactions

The Solana network suffered its latest outage Wednesday, felled for over four hours by a bug in how the blockchain processes a niche type of transaction that’s designed for offline use cases.

Validators began restarting the network only after disabling these “durable nonce transactions,” Solana Labs communications chief Austin Federa told CoinDesk. Those transactions will remain nixed until developers identify and patch the exact culprit that threw Solana’s consensus mechanism off-kilter.

That may have ramifications for offline custodians whose transactions fall under this category, perhaps even freezing their ability to move funds until the patch is in, validators said. CoinDesk has begun reaching out to exchanges to ask about their Solana transaction setup.

Even so, at press time Wednesday a number of exchanges were reporting problems with Solana deposits and withdrawals. Among them: Binance, Coinbase and Crypto.com.

The chain’s native SOL token was already trading lower Wednesday when the outage began around noon Eastern time; it continued its 24-hour slide and was down nearly 13% around 8:30 p.m. ET, trading at $39.98, according to CoinMarketCap.

Durable nonces

Federa said durable nonces represented “an incredibly small percentage” of transactions on Solana until recently. The technology has been growing in popularity among exchanges. In cryptography, a nonce is a random number used for a specific purpose.

“This was probably a bug that existed for a while but never really became an issue because it isn't something that most people use,” Federa said.

Durable nonces on Solana are designed for token holders with complex offline signing setups that can’t always prep their transactions fast enough for the speedy network.

For example, a custodian that signs Solana transactions with two air-gapped computers might not be able to finish the job within a single block. Normal transactions on Solana would fail in this scenario. Durable nonces give the token holder time to work.

What happened on Wednesday was a failure in Solana’s ability to handle durable nonces. Instead of treating these niche inbounds as a single transaction, the network’s validators double-counted them as a single transaction at two different block heights, Federa said. This impossible situation effectively broke Solana’s consensus mechanism.

In a tweet, Laine from Stakewiz, a Solana validator operator, said the bug was “known” and was being fixed prior to Wednesday’s events. It “hadn’t been triggered in this form previously,” Laine said.

The network was slowly coming back to life on Wednesday evening as key infrastructure pieces such as RPC nodes resumed work.