Sherlock’s Collaborative Auditing System Achieves Results

Sherlock’s Collaborative Auditing System Achieves Results

‍

Overview

‍

This case study analyzes and compares collaborative smart contract audits performed by Sherlock and MoveBit on the exact same set of files in scope. 

‍

This case study highlights the considerable differences in issue detection, severity classification, and overall utility provided to the client. The Sherlock audit both uncovered a greater number of significant vulnerabilities and was more accurate in classifying the vulnerabilities identified.

‍

Panana Background

‍

Panana is a next-generation prediction market platform where users trade on event outcomes through a unique AMM-driven system that ensures deep liquidity and transparent pricing. Powered by tokenized shares and user-generated markets, it supports a wide range of predictions—from crypto prices to real-world events.

‍

Audit Details

‍

Both Sherlock and MoveBit were contracted by Panana, and produced complete audit reports (appendix).

‍

Timeline of the Audit

‍

In addition to the number and quality of the issues found, the duration of the audits is also an essential factor to consider, as it has a direct impact on the cost of the audit. More working days mean a higher cost for the team under review.

‍

Judging

‍

Sherlock and MoveBit use different classification criteria in their audit reports, with Sherlock being more stringent. To accurately compare the two firms’ performances, Sherlock contracted an independent third party to judge all of the issues found according to the same criteria.

‍

The third-party judge was security researcher 0xiex. He has years of judging experience, is currently ranked #25 on Sherlock’s leaderboard, and has brought in over $160,000 in career contest earnings. Sherlock in no way influenced any of his judging decisions, and he can attest to that fact.

‍

Sherlock’s Collaborative Auditing System Achieves Results

‍

Sherlock’s collaborative model transforms auditing from a manual service into a data-driven science.

Every researcher on the Sherlock platform has a measurable performance profile built from historical results, accuracy scores, and predictive modeling that assess their likelihood of identifying specific vulnerability types across Solidity, Rust, and other codebases.

When a new audit begins, Sherlock’s system uses this data to assemble elite teams with complementary skill sets, ensuring each codebase is reviewed by the most qualified experts for that protocol’s architecture.

‍

‍

In this case, Panana is an Aptos ecosystem project built with Move. The auditing team was assembled for deep knowledge of the Move language.

It was important for Panana’s launch plan that the audit be started as soon as possible. Because the three auditors selected by Sherlock are experts in Move, they have a much higher baseline of context and understanding. This allowed them to jump right in and start evaluating the code deeply from day 1.

This actuarial, intelligence-driven approach means that Sherlock engineers audit teams for maximum depth, coverage, and assurance.

‍

Key Findings & Results 

‍

Sherlock’s audit provided substantially higher utility for the Panana team through more rigorous analysis and higher quality findings. 

‍

Sherlock’s audit uncovered 3 valid Highs, while the MoveBit audit only uncovered 1. MoveBit also reported 4 Highs that were actually insignificant and downgraded to low/info by the independent judge.

‍

‍

Dangers of Overclassifying Vulnerabilities

‍

Accurate severity classification ensures clients understand actual risks and prioritize resources appropriately. Overclassifying higher vulnerabilities can be dangerous and detrimental to the team. 

‍

According to the independent judge, the MoveBit report overclassified 4 vulnerabilities as highs that were not. 

‍

Overstating severities can waste teams’ time by misleading them into prioritizing the wrong issues. If medium-risk bugs are classified as high, real high-severity vulnerabilities might get delayed, leaving systems genuinely exposed. Overclassification can also push teams to unnecessarily redeploy contracts, incurring high costs and introducing new attack surfaces.

‍

At Sherlock, we pride ourselves on our rigid and transparent judging rules. We strive to be 100% honest and accurate about all of our findings. Our mission is to secure Web3, not to bolster our results for our own gain.

‍

Why The Sherlock Process Is Superior

‍

The Sherlock team found more highs in less than half the time it took the MoveBit team to audit the exact same scope. 

‍

How was this possible?

‍

Sherlock leverages years of data and past results to handpick the most effective audit team possible. Building a team tailored to the needs of each protocol allows the auditors to be both more effective and efficient with their evaluation. 

‍

Sherlock is committed to work of the highest quality, and this includes vulnerabilities. Many firms like to include as many low/informational vulnerabilities as possible on their audit reports to boost the total number of issues found. In reality, these vulnerabilities aren’t of any importance as they do not pose any significant risk to the protocol. This practice wastes both the auditors' and the teams’ time. Our auditors focus on understanding the codebase deeply to deliver the most vulnerabilities that will actually have a significant impact on the protocol’s safety and true function.  

‍

Sherlock has never been outperformed by another auditing firm when auditing the same codebase.

‍