The brief history of Web3 projects is littered with multimillion-dollar Web3 hacks and exploits plastered across Discord, Telegram, and Crypto Twitter. Yet, in the few years that Arbitrary Execution has been around, we have yet to find a Web3 company that does not want to be secure, although we have noticed several factors that make it challenging for Web3 companies to apply security best practices. As the Web3 industry continues to grow, it could continue to tread the same path of billions of dollars lost to hacks, or companies can choose to assess the security of their projects, identify deficiencies, and take steps to remedy identified shortcomings.
At Arbitrary Execution, researching and assessing security is what we do. It may sound obvious, but securing a project starts with the knowledge of what is secure and what is not. It is unreasonable to expect that Web3 companies will drop everything, spend a few years becoming security experts, and then get back to creating successful projects, while also staying up to date on new security tools, methods, and approaches. What companies and developers need is a way to quickly evaluate Web3 security best practices and apply them to their projects.
There are a few well-worn security paths that have become mantras in Web3, such as "not your keys, not your coins," but for most situations, people only have a vague intuition of what "secure" looks like. To remedy this, we need to establish criteria that distinguish what is secure from what is merely hopeful. In this blog post, we will review some existing approaches to "measure" security.
We think it's important to review existing frameworks that others have created to assess the risk of Web3 projects. The following frameworks present, or formerly presented (in the case of abandoned frameworks), scoring criteria to quantify risk, each having its own advantages and disadvantages.
Smart Contract Security Verification Standard
The framework provided by Composable Security, called the Smart Contract Security Verification Standard (SCSVS), is a checklist that developers, architects, auditors, and vendors can use to verify that smart contracts adhere to security best practices. The framework was initially published in 2019 in the securing GitHub repository and most recently published by Composable Security.
The checklists provided by the SCSVS are extensive, providing a great pre-audit checklist for smart contracts. The lists provide specific, low-level considerations for creating smart contracts that avoid common pitfalls and vulnerabilities, even going so far as to recommend gas optimizations such that users would not be deterred from performing transactions with contracts.
The checklist presupposes knowledge about a number of topics, such that it would be immediately beneficial to an auditor, but would require a greater investment by a software developer to understand. The checks indicate the presence or absence of different features, but they do not provide direction on how to achieve the desired outcome. The framework implies that documentation exists, but does not lay out requirements for a whitepaper that includes specific math, diagrams, etc. that would be critical for an outside party to understand the intended operation of a protocol. The coverage of secure deployments is quite sparse, as is the mention of security best practices for monitoring, defending, and mitigating risks after deployment.
The framework provided by Defi Safety was last published in December 2021 and appears to be a descendant of the SecurEth Guidelines. The framework claims to be patterned off software best practices from the aerospace industry.
Access to the framework is free and open to all, providing many valid criteria for assessing security. The categories presented cover many important aspects of security, and the overall goal of the project is to increase transparency for users about the potential risks in decentralized finance protocols. The process documentation clearly defines each criterion as well as provides specifications, scoring guidance, and tips on how to improve scores.
There are several criteria that do not appear to have a direct bearing on the security of a software project. For example, it is unclear why having the entire development history of a project is included. Next, the focus of the framework appears to be creating a single "score" that summarizes the trustworthiness or security of the project. To achieve this "score," weights have been assigned to each of the criteria, but those weights appear to be somewhat arbitrary. The weighted categories are then summed up to create the final "score," but directly comparing disparate categories is ill-advised. For example, the weights in the framework effectively set up an equivalence between a project listing its executing code addresses and a project having a full test suite that covers all its code.
CryptoCurrency Certification Consortium
The framework provided by the CryptoCurrency Certification Consortium (C4) includes extensive policy documentation, formal examinations, and certification processes for auditing information systems that will interact with cryptocurrencies. It seeks to provide sets of standards for professionals specializing in blockchain technology, allowing prospective employers and companies to easily recognize individuals who have the proper expertise.
The material that addresses key management processes is extensive. For example, the standards recommend keys that are geographically distributed, organizationally distributed, and redundant. This is a high bar, but certainly applicable if and when private keys have administrative or privileged access granted to them.
As the gatekeeper of the standards and certifications, one must go through C4 to become a certified auditor or to request a certified auditor to receive a grade on a project. The standards pertain to information systems, not Web3 projects. While there is an overlap between those two applications, there is a lot left uncovered for Web3 companies.
The framework provided by DeFi Score was introduced in September 2019 and then archived in GitHub in September 2022. The framework attempted to assess risk in decentralized lending protocols, with the goal of transitioning its methods to a public good in the future.
The approach acknowledges that the "security of smart contracts are extremely important when evaluating the risk of users losing funds that are stored in a smart contract when interacting with many DeFi platforms." As some examples of valid areas of concern, the framework acknowledges the impacts of administrative private keys, oracle dependencies, audits, and bounty programs.
Much as the name implies, the DeFi Score framework seeks to establish a "score" that can be used to assess the risk in decentralized lending protocols. The pitfalls to this single "score" approach are the same as those mentioned in the Defi Safety case. The documentation for DeFi Score acknowledges that it is an opinion-based framework that started with arbitrary weights for scoring. While the framework mentions security, its treatment is broad and descriptive rather than detailed and prescriptive.
We believe the existing Web3 security criteria all fall short in one way or another. However, we acknowledge that security is an incremental process. The existing security criteria have helped shape the foundation for a new set of security criteria. A clear set of security criteria acts as a standard that teams can use to identify areas that need attention and improvement. Criteria should have accompanying milestones that projects may use to guide their planning processes. Above all, these criteria should be both descriptive and prescriptive. That is, they should both tell companies what security best practices are, and they should tell companies how to achieve better security.
For Web3 companies to incorporate security into every phase of their projects, they need to know what they're aiming for and what their security objectives are. Borrowing from the world of project management, we propose security criteria that are patterned after SMART goals. For a metric to be valuable, it needs to be Specific, Measurable, Attainable, Relevant, and Time-Bound, hence SMART.
As an example, some criteria are naturally predisposed to this model, such as having a test coverage criterion for your smart contract code:
- Specific - Have a test suite for code
- Measurable - The ratio of number of branches visited during tests versus total number of branches existing in code
- Achievable - Write test code and test cases
- Relevant - Tests will allow your developers to understand your code better, find bugs, and detect regressions in code when new features are added
- Time-Bound - Create and run before code audit
We understand that applying security best practices in all phases of your project requires time and effort, especially when you are uncertain of where to even find credible information on what those practices are. Soon, we will be rolling out well-considered, practical materials that you can start applying to your project to secure it from conception through deployment. Stay tuned for new developments from Arbitrary Execution!