Crypto White Hat Platform Immunefi Banned 15 ChatGPT-Generated Bug Reports. Here's Why

OpenAI’s ChatGPT has stirred the hornet’s nest since its launch, with the Twitterverse readily commenting on how the software could eventually create code, write stories, steal jobs and even fight wildfires.

But the crypto platform Immunefi is having none of that. The bounty-paying white hat – a term for individuals who hack products with the goal of patching security instead of theft – banned 15 users last week who submitted ChatGPT-regenerated reports on the platform, albeit to mixed sentiment.

Immunefi developers told CoinDesk that the platform’s reasoning on the subject is crystal clear for now.

“There’s a difference between something like GitHub Copilot and ChatGPT. With the former, you are driving the process and the role of Copilot is offering useful suggestions in context, which you accept or reject as you write your program,” Immunefi told CoinDesk in a Twitter message. “With the latter, you are using a single prompt to generate something that looks like a well-written bug report, but is nonsense when analyzed further.”

“This wastes the white hat’s time, Immunefi’s time and the projects’ time. That’s why we ban ChatGPT reports,” they added.

“It’s really an art,” developers further said, referring to the act of carefully writing a well-crafted bug report that clearly lays out issues and solutions for any exploit or bug.

Immunefi went a step ahead and asked ChatGPT itself why the software shouldn’t be used for generating bug reports, to a “satisfying response.”

The bug bounty platform has over $135 million in rewards available for white hats who find out vulnerabilities on decentralized finance (DeFi) platforms. It claims to have paid out more than $60 million in bounties and says the service has saved an estimated $25 billion in user funds.

As such, Immunefi said it would continue to monitor ChatGPT-generated reports even as they eventually get more sophisticated. “There are a lot of very obvious tells. If you play around with ChatGPT, you’ll start to see a lot of patterns in the output,” developers said.

“For example, no regular white hat would go out of their way to waste their time submitting a long, nicely written and well-structured bug report that has no relation to a project’s smart contracts at all,” they added.

But that’s not to say they would ever not consider unbanning the use of AI-generated bug reports in the future.

“We will keep an eye out for the development of AI tools, and we will consider unbanning their use if they are ever capable of generating real bug reports.” For now, however, it’s ban time.