Chainalysis Testimony Raises the Question: Do We Know How Well Any Such Software Works?

Editor-in-chief’s note: An earlier form of this pseudonymously written opinion piece was retracted due to its containing a perceived personal attack on a Chainalysis employee. We did this because it violated our policy, which is based on the principle that people have the right to know the identity of their critics, of not publishing personalized attacks under pseudonymous bylines. It was not our intent to besmirch the reputation of the writer, who has for some time used the same pseudonym and built a reputation around it. To resolve this, in collaboration with the writer, we are republishing a further edited version of the piece, removing language that unreasonably targeted the individual in question and making clear that its sole intent is to question the accuracy of blockchain demystification software, and not attack any particular individual.

Elizabeth Bisbee, head of investigations at Chainalysis Government Solutions, testified she was “unaware” of scientific evidence for the accuracy of Chainalysis’ Reactor software on June 23, 2023, an unreleased transcript of the hearing shared with CoinDesk shows.

The fact that Chainalysis’ blockchain demystification tools have become so widespread is a serious threat to the crypto ecosystem. Although industry insiders have raged against Chainalysis since it was founded, often accusing it of violating people’s financial privacy, there may be a better argument to make against the company and analysis firms like it: it’s within the realm of possibility that these “probabilistic” machines don’t work as well as advertised. This could lead to unjustified account restrictions when blockchain surveillance tools are used for compliance, and in the worst case land unsuspecting individuals on the radar of law enforcement agencies without probable cause.

That’s precisely the argument that renowned lawyer Tor Ekeland is making in his latest defense of an accused early bitcoin adopter, and why he was quizzing a Chainalysis executive on the stand.

Bisbee was testifying in a case between the U.S. government and Roman Sterlingov, the alleged creator of the once popular Bitcoin Fog cryptocurrency mixer used to anonymize bitcoin transactions. Chainalysis’ Reactor software was used to track cryptocurrency payments in Sterlingov’s criminal investigation, and is now being challenged by Sterlingov’s defense.

Sterlingov is represented by Ekeland, who has made a career out of defending hackers and technology providers. Ekeland said Chainalysis’ Reactor is “a black box algorithm” that “relies on junk science.”

In a hearing aimed to establish the admissibility of expert testimony, Bisbee was pressed for details on the accuracy of the Reactor software Chainalysis sells to governments for law enforcement purposes, including what evidence the company has that suggests it works.

Bisbee said she was unable to provide the court with statistical error rates for Chainalysis’ Reactor software. She said she unaware of any scientific peer-reviewed papers or “anything published anywhere” attesting to the accuracy of Chainalysis Reactor.

Instead, Chainalysis reportedly judges its software’s accuracy using customer feedback, she said.

See also: Crypto Criminals Stealing Much Less Money in 2023

None of this is to cast aspersions on Bisbee's personal understanding of the software. For example, Bisbee’s statements are in line with a blog post published by Chainalysis competitor Coinbase, which describes blockchain analytics as “more of an art than a science.” Coinbase offers blockchain analytics services to law enforcement via its Tracer software.

A statement issued by Chainalysis on July 18th confirms that Bisbee is not only unaware of margins of error rates for Chainalysis’ Reactor software, but that Chainalysis has failed to gather and record false positive and negative rates for its software overall.

Unfortunately for Chainalysis, we still live in a democracy in which criminal convictions prerequisite the existence of scientific evidence.