Avalanche operational again after 5 hour outage
Avalanche nodes recently began the Durango upgrade, which is scheduled to activate on its mainnet next month
Furkan Cubuk/Shutterstock, modified by Blockworks
Avalanche’s C-Chain stopped producing blocks at around 6:30 am ET Friday, due to a client code bug, developers have learned. The network resumed normal operations roughly 5 hours later.
Most transactions on the network take place on the C-Chain, however some subnets were also affected since all validators are required to process C-Chain transactions.
In an X post, Ava Labs co-founder Kevin Sekniqi attributed the likely cause to “a gossip-related mempool management bug, which is purely a code-related bug, and not an issue with performance handling,” meaning the outage was not related to a spike in usage of the network.
The initial working theory was that it was an “edge case” related to inscriptions.
Shortly after the outage began, Avalanche’s network status page reported that “developers across the community are currently investigating.”
Avalanche validator operators have been in the process of upgrading their node software to a new version that would bring a major upgrade called Durango online.
The upgrade is expected to activate on March 6, 2024, and as of the time of the outage, roughly 17% of the 1,838 network validators were already running the new client production code version 1.11.0, according to data provider snowpeer.io.
That release of AvalancheGo was published on Wednesday, but it’s not thought to be the cause, according to Sekniqi.
“I do not think it is related to v1.11, it just seems to have hit an edge case in mempool processing,” Sekniqi told Blockworks. “The bug hit majority of nodes, it seems.”
By 11:00 am ET, Ava Labs engineering vice president reported the bug had been identified, writing on X that it resulted from “logic added in v1.10.18 that lead to validators sending an excessive amount of gossip to each other.”
Validators are encouraged to update to a newly released version 1.11.1. Over a quarter of the network had done so by 11:45 am ET, while the percentage of validators running the buggy version dipped to 19%, allowing the network to come back online snowpeer.io showed.
Subnet validators currently must also validate the C-Chain, which is why the outage also affected subnets, although this architecture is slated to change. That can’t come soon enough according to Avalanche director Omer Demirel.
“Today’s outage is a good example of why we need ACP-13 (Subnet-Only Validators), which will improve the Avalanche network’s fault tolerance by relieving subnets from primary network validation duty,” Demirel posted to X.
The outage bears resemblance to one that knocked Solana offline earlier this month.
Read more: Solana price dips as outage requires a network restart
In that instance, a software patch followed by a coordinated restart was required before service was restored several hours later.
According to community manager Leandro Matos on the Avalanche Discord, the outage is unusual. “The chain only had one minor outage since 2020 which lasted a few minutes,” he said.
Avalanche’s native currency AVAX as down 3.2% on the day to around $35.60 as of 11:30 am ET.
Updated Feb. 23, 2024 at 8:38, 9:24, 10:27 and 11:32 am ET with additional context and 11:46 am ET revising the headline.
Get the news in your inbox. Explore Blockworks newsletters:
- The Breakdown: Decoding crypto and the markets. Daily.
- 0xResearch: Alpha in your inbox. Think like an analyst.
