Solana Network Stumbles, On-Chain Trading Slows After ‘Forking’ Incident

Infrastructure operators on the Solana network rushed to right the struggling blockchain early Saturday after a tech issue throttled users’ ability to trade crypto, transfer assets or do anything on-chain.

The blockchain started “forking” (creating conflicting versions of its transaction history) at around 12:53 a.m. ET, according to Solana’s Discord server. Shortly after that, validators’ RAM began increasing while the chain’s transaction throughput fell off a cliff.

Those tech troubles combined to effectively freeze nearly all on-chain activity on the Solana network. By 2 a.m., the network was processing about 93 transactions per second (TPS), well below the rate near 5000 TPS some 15 minutes prior, according to the data site Solana Explorer.

It was a crisis reminiscent of the chain’s rocky tech incidents in 2022, when a series of halts and slowdowns prompted reforms to how Solana manages inbound traffic. Saturday’s incident had no immediate culprit, though validator operators and network engineers suspected a bug in the new version of Solana code that had come online hours before.

Without a definite bug to squash, some validators began downgrading to the previous version in hopes of reviving Solana’s throughput, said the pseudonymous SolBlaze, who runs a liquid staking pool and is active in developer circles. Later, Solana employees began advocating for the downgrade, too.

Within hours a supermajority of validators had switched back to the old software in their attempt to restore Solana’s operations. But it did little to solve the still unknown problem that was weighing down performance. The effort then turned to a more drastic solution: restarting the chain to the point immediately prior to the forking.

“Coordinating a restart attempt means that the chain will be offline completely, which is always the last resort,” SolBlaze said.

UPDATE (Feb. 25, 2023 04:09 UTC): Added details.