Skip to content

Commit

Permalink
fix: reset invalid fixtures as new scopes arrive
Browse files Browse the repository at this point in the history
  • Loading branch information
antazoey committed Oct 21, 2024
1 parent d43ea77 commit 6149aa8
Show file tree
Hide file tree
Showing 19 changed files with 1,504 additions and 107 deletions.
79 changes: 70 additions & 9 deletions docs/userguides/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,11 +70,6 @@ def test_authorization(my_contract, owner, not_owner):
my_contract.authorized_method(sender=not_owner)
```

```{note}
Ape has built-in test and fixture isolation for all pytest scopes.
To disable isolation add the `--disable-isolation` flag when running `ape test`
```

## Fixtures

Now that we have discussed the full flow of a test, let's dive deeper into the specific parts, starting with `pytest.fixtures`.
Expand Down Expand Up @@ -200,12 +195,10 @@ You also have access to the `project` you are testing. You will need this to dep
```python
import pytest
@pytest.fixture
def owner(accounts):
return accounts[0]
@pytest.fixture
def my_contract(project, owner):
# ^ use the 'project' fixture from the 'ape-test' plugin
Expand All @@ -226,6 +219,72 @@ def my_contract(Contract):

It has the same interface as the [ChainManager](../methoddocs/managers.html#ape.managers.chain.ChainManager).

## Isolation

By default, tests run with chain-isolation.
This means, at the start of each test, a snapshot is taken.
After each test completes, the chain reverts to that snapshot from the beginning of the test.

By default, every `pytest` fixture is `function` scoped, meaning it will be replayed each time it is requested (no result-caching).
For example, if you deploy a contract in a function-scoped fixture, it will be re-deployed each time the fixture gets used in your tests.
To only deploy once, you can use different scopes, such as `"session"`, `"package"`, `"module"`, or `"class"`, and you **must** use these fixtures right away, either via `autouse=True` or using them in the first collected tests.
Otherwise, higher-scoped fixtures that arrive late in a Pytest session will cause the snapshotting system to have to rebase itself, which can be costly.
For example, if you define a session scoped fixture that deploys a contract and makes transactions, the state changes from those transactions remain in subsequent tests, whether those tests use that fixture or not.
However, if a new fixture of a session scope comes into play after module, package, or class scoped snapshots have already been taken, those lower-scoped fixtures are now invalid and have to re-run after the session fixture to ensure the session fixture remains in the session-snapshot.

In the following example, the `my_contract` fixture gets deployed upon its first usage, which happens in the test `test_my_contract_0()`.
During the test `test_something_else()`, it may not have been deployed yet, as it was not requested, and it is defined before the other tests.
Then, during `test_my_contract_1()`, instead of deploying again, it uses the cached result from the session-scoped fixture and the chain still has it in its state because the fixture is session-scoped and runs before the test-isolation.

```python
import pytest
@pytest.fixture(scope="session")
def my_contract(accounts, project):
owner = accounts[0]
contract = project.MyContract.deploy(sender=owner)
# Can also do stateful transactions in a session-scoped fixture.
contract.initialize(sender=owner)
return contract
def test_something_else():
...
def test_my_contract_0(my_contract):
my_contract.myMethod()
def test_my_contract_1(my_contract):
my_contract.myMethod()
```

To disable isolation, run `ape test` with the `--disable-isolation` flag.
When isolation is disabled, the blockchain's state persists as the tests run.
This will be more performant and less complex, but will also cause non-deterministic results in your tests as each test inherits the state of whatever was run before it.

This may be further complicated when running with other pytest plugins such as `pytest-xdist` or `pytest-split` which re-arranges the order that tests are executed in (not recommended to use these plugins together with ape until more proper integrations are developed).

```shell
ape test --disable-isolation
```

```{warning}
Be mindful if, when, and how you define non-function scoped fixtures.
Pytest activates fixtures in the order they are used.
If a session scoped fixture comes into play after package, module, or class scoped fixtures, the isolation logic has to invalidate each of those scopes and replay them after the session scoped, which causes any benefits of package, module, or class scopes to be void.
If you are using higher-scoped fixtures for parametrized fixtures with lower-scoped fixtures, each itertion of the parametried fixture invalidates the lower-level fixtures each time, rendering everything to behave as function scoped until the end of the parametrized fixtures first run-through.
```

If you are using chain-isolation and have a higher-scoped fixture that you know is for-sure not chain-altering, you can use `ape.fixture` and the `chain_isolation` flag, and it may improve performance:

```python
import ape
from ape_tokens import tokens
@ape.fixture(scope="session", chain_isolation=False, params=("WETH", "DAI", "BAT"))
def token_addresses(request):
return tokens[request].address
```

## Ape testing commands

```bash
Expand Down Expand Up @@ -345,7 +404,8 @@ You may also supply an `re.Pattern` object to assert on a message pattern, rathe
import ape
import re

# Matches explicitly "foo" or "bar"
# Matches
# "foo" or "bar"
with ape.reverts(re.compile(r"^(foo|bar)$")):
...
```
Expand Down Expand Up @@ -396,8 +456,9 @@ You may also supply an `re.Pattern` object to assert on a dev message pattern, r

```python
import ape
import re

# Matches explictly "dev: foo" or "dev: bar"
# Matches "dev: foo" or "dev: bar"
with ape.reverts(dev_message=re.compile(r"^dev: (foo|bar)$")):
...
```
Expand Down
2 changes: 2 additions & 0 deletions src/ape/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@

from ape.managers.project import ProjectManager as Project
from ape.pytest.contextmanagers import RevertsContextManager
from ape.pytest.fixtures import fixture
from ape.utils import ManagerAccessMixin as _ManagerAccessMixin

# Wiring together the application
Expand Down Expand Up @@ -59,6 +60,7 @@
"config",
"convert",
"Contract",
"fixture",
"networks",
"project",
"Project", # So you can load other projects
Expand Down
2 changes: 1 addition & 1 deletion src/ape/exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -622,7 +622,7 @@ def __init__(self, snapshot_id: "SnapshotID"):
# Is block hash
snapshot_id = humanize_hash(cast(Hash32, snapshot_id))

super().__init__(f"Unknown snapshot ID '{str(snapshot_id)}'.")
super().__init__(f"Unknown snapshot ID '{snapshot_id}'.")


class QueryEngineError(ApeException):
Expand Down
12 changes: 12 additions & 0 deletions src/ape/pytest/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,18 @@ class ConfigWrapper(ManagerAccessMixin):

def __init__(self, pytest_config: PytestConfig):
self.pytest_config = pytest_config
if not self.verbosity:
# Enable verbose output if stdout capture is disabled
self.verbosity = self.pytest_config.getoption("capture") == "no"
# else: user has already changes verbosity to an equal or higher level; avoid downgrading.

@property
def verbosity(self) -> int:
return self.pytest_config.option.verbose

@verbosity.setter
def verbosity(self, value):
self.pytest_config.option.verbose = value

@cached_property
def supports_tracing(self) -> bool:
Expand Down
Loading

0 comments on commit 6149aa8

Please sign in to comment.