By now you should have applied security best practices in the Design phase, written your code and subjected it to rigorous internal, external, and automated review in the Develop phase, and securely launched your project in the Deploy phase. All of your hard work has led you to the Defend phase of the Web3 SDLC. Much as we might like to use the fire-and-forget approach, most projects will need to plan for security after their project goes live as well. Because the vast majority of blockchains are public in nature, attackers can seek out and exploit vulnerabilities in your code anytime after it’s deployed.

The Defend phase is divided into four categories:

  • Monitoring
  • Incident Response
  • Bug Bounties
  • Insurance

These categories build upon the work performed in the earlier phases, such that you will already have a head start on some topics here. This blog post will provide explanations for each category and some examples to illustrate their implementation.

Monitoring

In the Design phase we performed threat modeling to identify our risks. Then in the Develop phase we incorporated features into our code to reduce those risks. However, some threats simply cannot be mitigated by changes in design or implementation, and we need to stay vigilant by monitoring transactions that interact with the project. Monitoring consists of receiving newly mined transactions, filtering transactions based on a set of trigger conditions, and sending out notifications or even initiating a response if those conditions are met. We will present how to go about establishing requirements for monitoring and the available options for monitoring platforms, data sources, and notifications.

We consider most monitoring to fall into one of two main categories: operational and threat. Both types of monitoring use the same types of resources to accomplish their purposes, but they often differ in the complexity of their filtering conditions. Operational monitoring can be thought of as watching the normal activity of a project and providing summary notifications that are easy to understand. Examples of this may include token deposits or withdrawals, loans taken, swaps performed, and any other “normal” or expected transactions. Threat monitoring, on the other hand, is for alerting you to any anomalous activity. Threat monitoring will typically use much more specific conditions for sending alerts, while operational monitoring will provide visibility into the broader patterns that emerge from the normal activity of the project.

There are many ways that a project could choose to host its monitoring resources, from managing its own local infrastructure, to managing cloud-hosted infrastructure, to using a monitoring platform that manages its own infrastructure. The right option for any company will depend upon many factors, but the management of infrastructure is quite demanding, especially for infrastructure that will perform critical monitoring.

Data sources will directly impact what conditions can be monitored on your platform. At a minimum, monitoring will need on-chain data, including emitted events, function calls, internal transactions (trace data), and access to state variables through blockchain JSON-RPC endpoints. Importantly, for many platforms this on-chain data is typically only available for mined or validated transactions, which is to say that they have already been added to the blockchain. Therefore, many monitoring platforms can only notify a project after a condition has occurred, rather than in advance of it occurring. It is possible to monitor transactions in the mempool, where they have not yet been mined or validated, but this requires access to data from a mempool data provider.

Once a monitored condition is detected, alerts or notifications should be sent out through communications channels such as email, chat applications, or dashboards. Messages meant for human consumption need to be brief and to the point, as alert fatigue can set in and impact the response of a team. Dashboards can be a great option for operational monitoring, as they are highly effective at summarizing the data associated with a project, be it in the form of charts, tables, or interactive graphics.

Incident Response

Any company with a Web3 project should hope for the best and prepare for the worst. Rather than cobbling together an ad hoc plan when something occurs, a company should have an established response plan to efficiently direct their actions. Their plan should clearly define the roles and responsibilities of concerned parties, whether internal or external to the company. The plan should also outline what actions must be taken, be they on-chain transactions or public communications, and should specify which actions are manual or automated. There may be a temptation to remedy an issue as quickly as possible, but hastiness may result in greater problems, so a response plan—practiced frequently and followed to the letter—is the best approach.

While fully automated responses seem appealing, there should be a great deal of caution applied when considering the level of authority that such responses require. For example, if an automated response has the authority to "pause" a smart contract and it does so at an incorrect time, it may cause a loss of revenue for the company or it may cause users to abandon the project. If the monitoring conditions and the authority of an automated response are both narrowly scoped, then it may be a great option.

Triage and Root-Cause Analysis

A good incident response will start with a quick assessment of the available data to determine the severity and root cause of the problem. This may require a great deal of skill and expertise, as each incident may use its own unique attack vector, making the forensic process extremely laborious to the uninitiated. The result of this assessment will feed back into the response plan to help prepare remediation efforts, public statements, and coordination with any other affected companies and projects.

Bug Bounties

Once your contracts are deployed, a bug in your code could prove costly if exploited by malicious actors. A bug bounty program provides rewards for white hat hackers and security researchers who search for and report bugs found in your production code. While it may be tough to prove, bug bounty programs may even cause hackers to pursue the ethical route of reporting vulnerabilities rather than exploiting them.

The recommended size of bug bounties may be a matter of debate between protocols, security researchers, and bounty platforms. In brief, your bounties should be proportional to the total value that your protocol manages. If you manage $1B USD worth of ERC20 tokens, having a $5K USD bounty for critical vulnerabilities won’t provide much incentive to researchers and bounty hunters.

In addition to bug bounties that are open for longer periods of time, there are also audit contests. These contests allow a company to set the duration of the contest, the total reward amount, and the scope in terms of which contracts are eligible for finding vulnerabilities. Contests typically draw many security researchers with a wide range of skill levels and protocol-specific knowledge, but the fixed prize pool and the competitive nature of the rewards structure means that the company does not take on additional costs if more competitors participate. The downside of this approach is that one cannot guarantee that there will be many seasoned researchers and auditors participating in any given contest. However, some audit contest platforms account for this by having an experienced auditor oversee and participate in the contest.

Insurance

For times when an exploit occurs and no response plan can recover the funds or mitigate the damage, there are options available that are similar to insurance. Typically referred to as "cover," these options hold funds staked by members to cover a group of projects against a specific range of risks. Just as with insurance, there are claims processes to follow, and each insurer may have their own specific requirements, fee structure, and risk assessments for determining your insurability and rates.

Defend Together

There is one critical defense asset that can help guide you through all of these categories: a security partner. Having a team of seasoned security researchers, auditors, and smart contract monitoring experts will bolster your Defend phase like no other single factor. Your security partner can embed security personnel into your team to walk with you through each best practice, guide you to the best tools, help you to a secure, confident launch of your project, and advise you as your user community grows.

Why You Can Trust Arbitrary Execution

Arbitrary Execution (AE) is an engineering-focused organization that specializes in securing decentralized technology. Our team of security researchers leverage their offensive security expertise, tactics, techniques, and hacker mindset to help secure the crypto ecosystem. In the two years since the company’s inception, Arbitrary Execution has performed more than 40 audits of Web3 protocols and projects, as well as created tools that continuously monitor the blockchain for anomalous activity. For more information on Arbitrary Execution's professional services, contact info@arbitraryexecution.com. Follow us on Twitter and LinkedIn for updates on our latest projects, including the Web3 SDLC.