TestingStrategy
- Functional and non-functional user story testing
- Unit, integration, and system testing for the developed code
- Testing for any usability and performance scenarios
- User Acceptance testing against a simulated environment using a VM
- Security testing
- Penetration testing
For each sprint the test strategy is split broadly into development testing, system testing, and user acceptance testing. These will be carried out by various members of the project team at different stages:
Builds will progress through the various stages when they succeed at all previous levels. For example, builds passing all unit and integration will be passed to system testing, etc.
Generally the distribution of the number of tests and the required time for execution should roughly follow the pyramid explained in this blog post and shown in the diagram below:
The aim is to detect as many defects as possible in development and then most of the remaining ones in system testing. This will relieve the load on deployment testing which is typically the most difficult to orchestrate.
Development testing is primarily managed and maintained by the developers. Developers have a key role in the testing process in addition to the tester; unit and integration testing are the sole responsibility of developers and an important part of the development activity.
- Automated Unit and Integration tests will be created and monitored around all the features where business logic materialises.
- Pull requests will require peer review and approval. Other formal and informal peer reviews can be performed throughout the project to ensure coding standards and good practice guidelines are followed.
- Code will have automated code analysis and unit test code coverage tools to monitor other quality metrics.
Most system testing will be story-based against pieces of development as they come through. Where possible there will also be automated end-to-end (E2E) regression tests created that run using the latest code.
One advantage of automating some of these tests is that they can be run frequently eg as part of every build rather than waiting for a deployment of the system. Due to the current lack of a build agent these will likely need to be run manually at regular intervals rather than for each new build.
System testing will also include some unscripted testing, whereby if developers and testers during development identify previously unflagged possible risk areas then these may be investigated. Any discovered bugs would subsequently be reported.
If the story is still open then the developer will be informed of the defect and will be expected to fix it as part of the completion of the story. If bugs are found that appear to be unrelated to any currently open stories/development then a new item will be recorded in the backlog and prioritised accordingly.
In Sprint review meetings, the project team will demonstrate the user story implementation in the Test environment (this may be synonymous with the Dev environment in the early stages). Business users will be talked through the acceptance criteria for each committed story in the sprint using the latest version of master. Post-review meeting, the business users may perform their user acceptance testing by following SMG guidelines or the demo scenarios to run the code themselves to evaluate the functionality and usability of the application. Any issue or suggestions identified in the user evaluation period shall be communicated to the development team for further action; most likely story creation and prioritisation.
Due to the application's modular architecture it may be possible to deploy and test parts of the application separately. This will be done at earlier stages wherever possible. This testing will not be destructive in nature - any destructive testing shall take place in System Testing with use of a developer environment.