Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial unit testing implementation #89

Merged
merged 6 commits into from
May 10, 2021

Conversation

nordmoen
Copy link
Collaborator

@nordmoen nordmoen commented May 4, 2021

This PR contains the initial implementation for unit testing in BLOM. The PR introduces a new sub-directory, tests, which contains unit tests to run. Currently only fuk95 is complete and running. The README has been updated to reflect the unit testing setup.

This PR also contains an updated CI configuration which will run both build and test for all targets and separates out OpenMP to catch problems earlier. In addition this PR skips MacOS builds with MPI as that has given some strange dependency issues. Tests are currently not run with MPI as that is not working locally either.

The next steps for unit testing would be to parse the output of the completed runs and compare the result with known good output, so that deviations from this can automatically be detected.

@AleksiNummelin
Copy link
Collaborator

AleksiNummelin commented May 5, 2021

I think this is a good start, we can get the physical tests in later. A minor point is that some of the new idealized cases (like the single_column and channel, maybe even fuk95) could be used for real science as well, not just for testing. So I wonder if tests directory should be renamed to examples or something similar, or if there should be another directory called examples and then in the testsdirectory one would just have the ci.yml and in future some validation data/scripts related to the physical tests? This way the tests directory would just be a case directory for testing while the actual fortran files would live in examples/<example_name> directory. @nordmoen would this approach be possible/problematic in terms of CI/meson?

@nordmoen
Copy link
Collaborator Author

nordmoen commented May 5, 2021

It doesn't really matter what the test folder is called as long as it is added with subdir in the main meson.build files. Currently the test folder only contains meson.build files to create an executable which is then added as a Meson test, so no Fortran files have been moved there.

The ci.yml file must live under .github/workflow folder or else it will not work.

I think we maybe should add an additional example/ folder instead of using the test/ folder. The unit tests, that should be run by CI, should not be changed by users so that we can always be certain that the tests are equivalent on all machines.

@matsbn
Copy link
Contributor

matsbn commented May 10, 2021

I tried the test procedure on Betzy and it worked well! We have an ongoing discussion on the directory structure of the repository (#86), but it seems fairly straightforward to relocate things when that has been settled.

One thing I would have preferred is that the output from tests is relative to the "builddir" given as argument to Meson. I typically locate the "builddir" outside of the repository and writes output on my scratch area. The updated .gitignores ensures that the test output does not interfere with git, but cleaning up after a test is a bit cumbersome since the "limits" file should be preserved in the test run folder.

Change the 'workdir' variable (the folder where tests are run) to the
current build directory. This necessitates a copy of the 'limits'
configuration file as well.
@nordmoen
Copy link
Collaborator Author

One thing I would have preferred is that the output from tests is relative to the "builddir" given as argument to Meson. I typically locate the "builddir" outside of the repository and writes output on my scratch area. The updated .gitignores ensures that the test output does not interfere with git, but cleaning up after a test is a bit cumbersome since the "limits" file should be preserved in the test run folder.

I change the configuration around so that this is indeed the case.

@matsbn
Copy link
Contributor

matsbn commented May 10, 2021

Thanks for making that change, @nordmoen!

@matsbn matsbn merged commit 26db3ed into NorESMhub:master May 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants