Decide on testing strategies

[webknjaz]

SUMMARY

We need to decide and document how we do testing.

Right now we have a strong linter suite integrated and running in the CI.
It's mostly enough for its purposes but I should note that some of the checks simply don't support Cython so if we discover good Cython linters we should totally add them into the suite. We have some support for the coverage collection but Cython plugin is currently disabled.

We also have pytest which at the moment does almost nothing. It's a smoke test in its current incarnation, it only checks imports and object construction. Also, 2/3 of the smoke tests are failing on object instantiation.

Besides, we have a few more smoke tests in the form of imports after building dists here and there.

Build/test/lint steps are wrapped with tox to be easier and better reproducible. This is also very helpful when mapping them to CI and making sure that they run there in the same manner as locally.

Now, pytest-based tests need to be specified more clearly. As I see it, we should have unit and integration tests. I think we can cover both types using pure pytest (maybe with plugins). For integration tests, we can use OpenSSH that is usually already pre-installed on the systems that we have interest in supporting.

@ganeshrn also mentioned adding some docker containers for integration tests but I don't have a clear understanding of why we can't test things w/o them. This needs clarification. FWIW it's totally possible to run Docker on Linux workers in GitHub Actions Workflows if really needed.

ISSUE TYPE

• Feature Idea