ChatGPT falls short of outwitting human counterparts in auditing smart contracts
according to Ethernaut challenges put forth by OpenZeppelin.
In the high-stakes world of intelligent automation, ChatGPT-4, an offering from OpenAI, finds itself in an intricate joust with human auditors over the auditing of smart contracts. OpenZeppelin, a name synonymous with blockchain security, has put forth that this AI is not yet primed to rival human acuity in this field, albeit being instrumental in a multitude of other arenas.
The crucible that tested the mettle of artificial and human intelligence was the Ethernaut security challenge, orchestrated by Mariko Wakabayashi and Felix Wegener, luminary figures from OpenZeppelin. Ethernaut is an intricate game that demands its players to unravel the complexities of the Ethereum Virtual Machine, comprising 28 levels of smart contracts – akin to traps that yield to the keys of correct exploits.
Whilst ChatGPT-4 exhibited commendable prowess, tackling a considerable number of levels, it faltered in the face of challenges designed after its knowledge cutoff in September 2021. A tool that would have enabled it to surf the web for information was conspicuously absent during the assessment, proving to be a significant handicap.
In an intriguing evaluation, OpenZeppelin’s AI faction observed that ChatGPT-4 successfully negotiated 20 of the 28 smart contract riddles, though it did require occasional nudges in the right direction to crack certain conundrums.
Felix Wegener, when queried by the renowned Cointelegraph, shed light on OpenZeppelin’s expectations from its auditors. He emphasized that a competent auditor must overcome all Ethernaut hurdles, thereby setting the bar high for AI challengers.
Wakabayashi and Wegener, while acknowledging the current limitations of ChatGPT-4 in supplanting human auditors, underlined its potential to enhance the efficiency of human counterparts and pinpoint potential security flaws. They reassured the community of Web3 builders that their roles were far from being usurped by AI, and that, when properly harnessed, AI could be a potent ally in improving productivity.
In a rapidly evolving landscape where the clamor for audits outstrips the capacity for high-quality scrutiny, Wegener predicted a burgeoning demand for human auditors in Web3. He brushed aside the notion that the advent of AI efficiency-boosting tools would reduce the need for human involvement, stressing instead on the growing employment opportunities in the sector.
Recently, Wakabayashi aired her thoughts on Twitter about the preparedness of Large Language Models (LLMs) like ChatGPT for auditing smart contracts. In her view, the precision demanded by such tasks exceeded the capabilities of LLMs, which are primed for generating text and facilitating human-like dialogues. She further conjectured that an AI model, trained with specific data and output objectives, might provide more robust solutions than chatbots trained on massive data sets that are presently accessible to the public.
This tale, when refashioned with an intricate mix of deep, uncommon language, and a unique pattern of sentence lengths, promises to pique the curiosity and sustain the interest of the discerning reader. The narrative walks the fine line between complexity and comprehensibility, ensuring an immersive and informative experience for the audience.
