From 99578feeb7bc782504dd803698e0101e2ce166a7 Mon Sep 17 00:00:00 2001 From: Monica Jin Date: Wed, 6 Nov 2024 11:26:47 +0100 Subject: [PATCH 1/3] add examples to exploits README --- smartbugs-curated/0.4.x/README.md | 108 ++++++++++++++++++++++++------ 1 file changed, 86 insertions(+), 22 deletions(-) diff --git a/smartbugs-curated/0.4.x/README.md b/smartbugs-curated/0.4.x/README.md index 2ef9e4f..5b03767 100644 --- a/smartbugs-curated/0.4.x/README.md +++ b/smartbugs-curated/0.4.x/README.md @@ -1,33 +1,97 @@ -# Automatic exploits for smartbugs-curated +# Automatic Exploit Testing for SmartBugs Contracts -This directory contains attacks which expoit the vulnerabilities in contracts from smartbugs-curated. +This repository contains automated exploit tests for vulnerabilities found in the SmartBugs-curated smart contract dataset. The tests are implemented using the Hardhat testing framework. -These attacks are automated using Hardhat testing framework. +## Overview -## Directory structure -- `contracts/`: Contains Solidity smart contracts. - + `dataset/`: Dataset directory corresponding to smartbugs-curated contracts. - + `/`: intermediate directory where `` corresponds to the category of vulnerabilities. - - `_attack.sol`: attacker contract w.r.t. `dataset//.sol` +The project demonstrates various security vulnerabilities in Ethereum smart contracts by providing: +- Vulnerable contracts from the SmartBugs-Curated dataset +- Exploit contracts that demonstrate the vulnerabilities +- Automated tests that verify the exploits -- `test/`: Contains test files. Hardhat uses Mocha testing framework by default. - + ``: intermediate directory where `` corresponds to the category of vulnerabilities. - - `_test.js`: test simulating the attack `_attack.sol` on `.sol`. +## Project Structure -- `artifacts/`: Stores compiled contract artifacts generated by Hardhat. +``` +. +├── contracts/ +│ ├── dataset/ # Original vulnerable contracts from SmartBugs +│ └── / # Exploit contracts organized by vulnerability type +│ └── *_attack.sol # Exploit contract for each vulnerability +│ +├── test/ +│ └── / # Test files organized by vulnerability type +│ └── *_test.js # Test scripts that demonstrate exploits +│ +├── artifacts/ # Compiled contract files (generated) +└── hardhat.config.js # Hardhat configuration +``` -- `hardhat.config.js`: Configuration file for Hardhat. - -## How to get started +## Getting Started ### Prerequisites -- Nodejs with npm7+ (tested using node:v20.14.0 and npm:10.7.0) + +- Node.js (v20.14.0 or later) +- npm (v10.7.0 or later) ### Installation -1. Make sure that Nodejs is installed. -2. Clone repo and go into `smartbugs-curated/0.4.x` directory. -3. Run `npm ci`. -### Running tests -- To test all the exploits run `npx hardhat test`. -- To test a specific exploit run `npx hardhat test `. +1. Clone the repository + +```bash +git clone +cd smartbugs-curated/0.4.x +``` + +2. Install dependencies + +```bash +npm ci +``` + +### Running Tests + +Run all exploit tests: +```bash +npx hardhat test +``` + +Run a specific exploit test: +```bash +npx hardhat test test//_test.js +``` + +## Example + +To test a specific reentrancy vulnerability: +```bash +npx hardhat test test/reentrancy/simple_dao_test.js +``` + +## Test Results + +Each test file contains two types of tests: + +### 1. Sanity Tests +Verify normal contract behavior without exploiting vulnerabilities. These ensure the contract works as intended under normal circumstances. + +### 2. Exploit Tests +Demonstrate the vulnerability by executing attack sequences that exploit the security weakness. + +Example output: +```bash + Reentrancy Attack for simpleDAO.sol + ✔ sanity check: reentrancy/simpleDAO.sol (632ms) + ✔ should successfully drain funds through reentrancy attack + + 2 passing (651ms) + +``` + +## Contributing + +Feel free to contribute by: +1. Adding new exploit contracts +2. Improving existing tests +3. Enhancing documentation + +Please ensure all new exploits follow the existing directory structure and naming conventions. \ No newline at end of file From ff73868f1662c4dac578c613e364952a585f5c3f Mon Sep 17 00:00:00 2001 From: Monica Jin Date: Wed, 6 Nov 2024 11:49:02 +0100 Subject: [PATCH 2/3] refctor main readme --- README.md | 130 ++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 101 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index fe50865..ed7e30c 100644 --- a/README.md +++ b/README.md @@ -1,31 +1,103 @@ # HACK-LABS -Contains actual smart contract attacker contracts for major benchmarks, starting with smartbugs-curated. - -It includes an evaluation [tool](https://github.com/ASSERT-KTH/solidity-hack-labs/tree/main/evaluator) for automated patch assessment. -The tool allows you to test patched contracts from the dataset against these exploits to evaluate the effectiveness of the applied patches. - -### Smartbugs-Curated Dataset -Total contracts: 143 - -Details per category (following the [DASP taxonomy](https://dasp.co/)): - -| Vulnerability | Total Contracts | Exploits | -|--------------------------|-----------------|-----------| -| Reentrancy | 31 | 26 | -| Access Control | 18 | 16 | -| Arithmetic | 15 | 13 | -| Unchecked Low Level Calls| 52 | 20 | -| Denial Of Service | 6 | 4 | -| Bad Randomness | 8 | 4 | -| Front Running | 4 | 3 | -| Time Manipulation | 5 | 3 | -| Short Addresses | 1 | 0 | -| Others | 3 | 2 | -| Total | 143 | 91 | - -Details: - -- 1 duplicate contract: - - 0x627fa62ccbb1c1b04ffaecd72a53e37fc0e17839.sol on reentrancy and unchecked_low_level - - Not exploitable for reentrancy +[![Test Suite](https://github.com/ASSERT-KTH/solidity-hack-labs/actions/workflows/ci.yml/badge.svg)](https://github.com/ASSERT-KTH/solidity-hack-labs/actions/workflows/ci.yml) + +A comprehensive framework for testing smart contract security patches against real-world exploits. This project provides: +1. A reproducible dataset of smart contract exploits for SmartBugs-Curated dataset +2. An automated tool for evaluating security patches + +## 🎯 Project Structure + +``` +. +├── smartbugs-curated/ # Exploit dataset and test framework +│ └── 0.4.x/ # Solidity 0.4.x contracts and exploits +│ ├── contracts/ # Vulnerable contracts and exploits +│ └── test/ # Automated exploit tests +│ +└── evaluator/ # Patch evaluation tool + └── src/ # Source code for patch testing +``` + +## 📊 Exploit Dataset Coverage + +Our dataset is based on the [DASP taxonomy](https://dasp.co/) and covers the following vulnerabilities: + +| Vulnerability | Total Contracts | Working Exploits | Coverage % | +|--------------------------|-----------------|------------------|------------| +| Reentrancy | 31 | 26 | 83.9% | +| Access Control | 18 | 16 | 88.9% | +| Arithmetic | 15 | 13 | 86.7% | +| Unchecked Low Level Calls| 52 | 20 | 38.5% | +| Denial Of Service | 6 | 4 | 66.7% | +| Bad Randomness | 8 | 4 | 50.0% | +| Front Running | 4 | 3 | 75.0% | +| Time Manipulation | 5 | 3 | 60.0% | +| Short Addresses | 1 | 0 | 0.0% | +| Others | 3 | 2 | 66.7% | +| **Total** | **143** | **91** | **63.6%** | + +### Coverage Limitations and Technical Constraints + +The following table summarizes why certain contracts in our dataset could not be exploited: + +| Challenge Type | Number of Contracts | +|----------------------------|-------------------| +| Not Exploitable | 31 | +| Missing Methods | 9 | +| Mislabeled Solidity Version| 7 | +| Exceeded Time Limit | 3 | +| Incompatible Solidity Version| 1 | +| Honeypot Contracts | 1 | +| Hash Cracking Required | 1 | + +For a detailed breakdown of non-exploitable contracts and their specific reasons, see [not-exploitable.md](./not-exploitable.md). + +## 🚀 Quick Start + +### Testing Exploits +Navigate to the exploit dataset: +```bash +cd smartbugs-curated/0.4.x +npm ci +npx hardhat test +``` + +### Evaluating Patches +Navigate to the evaluator tool: +```bash +cd evaluator +python -m venv .venv +source .venv/bin/activate +pip install -e ".[dev]" +``` + +Test a patch: +```bash +python src/main.py \ + --format solidity \ + --patch ./your-patch.sol \ + --contract-file ./target-contract.sol \ + --main-contract ContractName +``` + +## 📚 Documentation + +- [Exploit Dataset Documentation](./smartbugs-curated/0.4.x/README.md) +- [Patch Evaluator Documentation](./evaluator/README.md) + +## 🤝 Contributing + +Contributions are welcome! Please check our contribution guidelines before submitting: +1. Follow the existing project structure +2. Include tests for new exploits +3. Document any new features or changes + +## 📝 Notes + +- Dataset contains one duplicate contract (`0x627fa62ccbb1c1b04ffaecd72a53e37fc0e17839.sol`) listed under both reentrancy and unchecked_low_level categories +- All exploits are provided for educational and testing purposes only + +## 🙏 Acknowledgements + +- [SmartBugs](https://github.com/smartbugs/smartbugs) - For the smart contract vulnerability dataset \ No newline at end of file From 1e981f40bd00189d83161accb3452c1065790875 Mon Sep 17 00:00:00 2001 From: Monica Jin Date: Wed, 6 Nov 2024 11:50:19 +0100 Subject: [PATCH 3/3] fix: smartbugs ack --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ed7e30c..6d848a9 100644 --- a/README.md +++ b/README.md @@ -100,4 +100,4 @@ Contributions are welcome! Please check our contribution guidelines before submi ## 🙏 Acknowledgements -- [SmartBugs](https://github.com/smartbugs/smartbugs) - For the smart contract vulnerability dataset \ No newline at end of file +- [SmartBugs](https://github.com/smartbugs) - For the smart contract vulnerability dataset \ No newline at end of file