Solana Goes Dark for 7 Hours as Bots Swarm ‘Candy Machine’ NFT Minting Tool

Solana stakeholders rushed to right the network Saturday night after what one insider called “insane amount of data” flooded the proof-of-stake chain, knocking validators out of consensus and grinding block production to a halt.

Bots had swarmed the popular non-fungible token (NFT) minting tool known as Candy Machine earlier Saturday with an unprecedented tsunami of inbound traffic: four million transaction requests and 100 gigabits of data every second – a record for the network, one person at the Solana Foundation said.

For reasons not yet clear, this swarm pushed validators out of consensus. Block production became impossible and the network went dark at 4:32 p.m. ET. By 11 p.m. ET, validators (coordinating through Solana’s Discord channels and a Google doc created by one of the validators) restarted the cluster at slot 131973970.

Co-founder Anatoly Yakovenko, who said he was traveling during much of the fracas, credited the validator community for spearheading mainnet recovery. He'd caught flak on Twitter Saturday for allegedly being "MIA" during a network crisis.

Unlike last September’s 17-hour outage, Saturday’s hard fork restart did not resolve with new-and-improved code populating across the validators. It simply picked up where the network flopped seven hours prior.

In preparing for the restart validators mulled whether to implement code that would temporarily block Candy Machine transactions. Some debated in the Discord whether such a move constituted censorship. Regardless, it would only be effective if two-thirds of validators opted in. Few appeared to do so on Saturday night.

Other parts of the ecosystem quickly moved to bolster their defenses. At 11:36 p.m. ET, Metaplex, the key steward of Solana NFT infrastructure and one closely intertwined with Candy Machine, tweeted it would soon deploy a 0.01 SOL “botting penalty” to help NFT projects stanch excessive traffic.

Solana ecosystem services like Phantom wallet and decentralized exchange Mango Markets struggled to stand up in the aftermath as RPC node providers plodded back online.

The outage contributed to a bloody, albeit brief, drawdown in SOL markets. Solana’s native token crashed to a 24-hour low of $83.13 about three hours into the outage before climbing back toward $89, according to CoinGecko.

Solana core developers have yet to diagnose what went wrong Saturday or how the apparent botting attack overcame existing safeguards to stymie consensus, a member of the foundation told CoinDesk.