The deployment of Web3 protocols is lacking with respect to security. A glance at the Rekt leaderboard showcases the projects that have had security vulnerabilities introduced during deployment, or were due in part to insufficient deployment processes which enabled vulnerabilities. Deployment of smart contracts and other off-chain components is still largely a manual process, fraught with peril and security vulnerabilities caused by human error. Additionally, deployment is typically not considered to be in-scope during a security audit and most Web3 teams lack dedicated Development Operations (DevOps) personnel.
Previous blog posts have introduced the Design and Develop phases of the Web3 Secure Development Life Cycle, but in this blog post, we present the Deploy phase. The Deploy phase of the Web3 SDLC encompasses all processes related to on-chain smart contract deployment, off-chain deployment of accompanying components, and upgrade processes. Within the Deploy phase are three key areas:
- Administration and Data Management
- Deployment Automation
- Verification and Testing
We will define each area, as well as provide high-level points to consider.
Administration and Data Management
Many protocols include administrative functions to help manage smart contracts. The actions that these administrative functions take vary widely, and include operations such as adjusting a key parameter, moving fees to a treasury, and more. Administrative functions are typically privileged, and as such require access control. Access control varies per protocol, and can require custom methods to be implemented to govern who can use administrative functions. A smaller protocol might only need a single role that is used for every protected function, while a more complex protocol that operates as a DAO might require multiple roles for specific purposes or even a fully-fledged governance proposal and voting system. Identifying and creating consistent processes for smart contract administration is vital to ensuring smart contracts are managed when necessary, and that no malicious actors can tamper with the protocol in general.
On the topic of governance, identifying relevant stakeholders, their roles and responsibilities, and the conditions upon which they may act is also important. While governance may not apply to every protocol, for the protocols it does apply to, it is key that privileged stakeholders who may influence the governance process do so in consistent and expected ways. This increases transparency with a project and its users, and also reduces the likelihood that security vulnerabilities are introduced should stakeholders need to interject into the governance process of a protocol.
For every project, it is essential to carefully consider, define, and document your private key protection strategy. The risk of loss due to scenarios like developers going rogue or team members getting phished can hardly be understated. Financial losses for some projects have totaled into the hundreds of millions of dollars due to key, wallet, or seed phrase compromises that subsequently result in smart contract ownership takeovers. The technology around private key management is constantly evolving and improving, and there are many ways to improve security. Storing keys on cold wallets, using a Multi-party Computation (MPC) wallet, and using smart contract multi-signature constructs are just a few of the ways to improve key management from less secure mediums such as hot wallets or local environment variable files that may contain deployment private keys. Additionally, ensuring sensitive information that is not a private key is appropriately stored and handled is also important to securing a Web3 project.
Within Administration and Data Management, it is also important to ensure post-deployment artifacts are published and readily available. In the event that external users need to interact with a protocol, having artifacts such as contract ABIs on hand reduces the reliance on third-party platforms.
One of the biggest defenses against failure in any domain is having a plan that is well thought out and has been tested and proven over time. This is nowhere more clearly seen than in the area of smart contract deployment, where even one simple oversight or omission can lead to irreversible consequences. In most traditional software environments, a misstep during the deployment process can often be easily corrected with a subsequent redeployment. On the blockchain, much of what is deployed is immutable, and mistakes made there are sometimes enshrined for all time.
Having automated processes for deployment ensures consistent behavior that is less likely to be influenced by human error. This is also true for upgrades, which in addition to automated deployment process contract upgrades should also have automation in place to ensure the integrity and expected behavior of the contract have not been compromised or incorrectly changed. An automated deployment process goes beyond just basic scripting and creates a procedure that can be reviewed, tested, and verified at each step. Encapsulating your deployment into a Continuous Integration-Continuous Development (CI-CD) pipeline creates an audit trail so that each release can be traced to the exact build environment that generated it. This allows developers, auditors and users to reproduce any build to fix a bug, run tests, or validate the build.
Verification and Testing
Having proper deployment verification and testing that simulates real blockchain network conditions is vital to ensuring a successful deployment. Even well-audited projects that have followed all the security best practices up to this point, with comprehensive unit test suites have faltered at smart contract deployment. Proper integration testing with real-world simulations are not encompassed under unit tests, and are certainly not covered by smart contract mocks that development teams are used to testing with.
Protocols should never be developed in a vacuum, isolated from running on real node software, and not the software that is packaged into popular development frameworks. Modern Web3 tooling has made it extremely easy for developers to run real blockchain software nodes forked from a mainnet locally on their development machines, giving them the opportunity to deploy and test a project’s on-chain code in an operating environment that very closely mimics the live blockchain that contracts will be deployed to. While there are many elements of running on a live blockchain that cannot be easily simulated on a local blockchain, these areas are even more important to test, and integrating tooling to enable this testing is worth the effort.
Additionally, using real node software affords opportunities to test compatibility with precompile contracts that exist exclusively within that node software (in most cases). Mocks of precompile or other blockchain-specific contracts do not adequately simulate real-world conditions and are oftentimes mocked incorrectly, which provide a false sense of security and can subsequently lead to deployment failures or even introduce security vulnerabilities during deployment.
An alternative to testing on a local blockchain is to deploy your project to a live testnet version of the respective blockchain. While there are still a number of elements that can only be thoroughly tested on the mainnet version of a blockchain, the testnet version of the blockchain should be functionally identical to the mainnet version, and in most cases include duplicates of contracts that exist on the mainnet. Therefore, test results should be closer to what would be expected on a mainnet relative to local forks that cannot easily simulate external contract traffic.
This blog post provides a broad overview of the key areas to consider within the Deploy phase, as well as some high-level guidance. We will continue to expand upon the SDLC in the future, so make sure you're following us to get alerts when we publish new Web3 SDLC content.
Why You Can Trust Arbitrary Execution
Arbitrary Execution (AE) is an engineering-focused organization that specializes in securing decentralized technology. Our team of security researchers leverage their offensive security expertise, tactics, techniques, and hacker mindset to help secure the crypto ecosystem. In the two years since the company’s inception, Arbitrary Execution has performed more than 40 audits of Web3 protocols and projects, as well as created tools that continuously monitor the blockchain for anomalous activity. For more information on Arbitrary Execution's professional services, contact email@example.com. Follow us on Twitter and LinkedIn for updates on our latest projects, including the Web3 SDLC.