Merge branch 'dev' into expose_aws_region

.github/PULL_REQUEST_TEMPLATE/release.md

@@ -1,6 +1,7 @@
 Prepare for release of HDMF [version]
 
 ### Before merging:
+- [ ] Make sure all PRs to be included in this release have been merged to `dev`.
 - [ ] Major and minor releases: Update package versions in `requirements.txt`, `requirements-dev.txt`,
   `requirements-doc.txt`, `requirements-opt.txt`, and `environment-ros3.yml` to the latest versions,
   and update dependency ranges in `pyproject.toml` and minimums in `requirements-min.txt` as needed.
@@ -10,23 +11,28 @@ Prepare for release of HDMF [version]
   and any other locations as needed
 - [ ] Update `pyproject.toml` as needed
 - [ ] Update `README.rst` as needed
-- [ ] Update `src/hdmf/common/hdmf-common-schema` submodule as needed. Check the version number and commit SHA manually
+- [ ] Update `src/hdmf/common/hdmf-common-schema` submodule as needed. Check the version number and commit SHA
+  manually. Make sure we are using the latest release and not the latest commit on the `main` branch.
 - [ ] Update changelog (set release date) in `CHANGELOG.md` and any other docs as needed
 - [ ] Run tests locally including gallery tests, and inspect all warnings and outputs
-  (`pytest && python test_gallery.py`)
+  (`pytest && python test_gallery.py`). Try to remove all warnings.
 - [ ] Run PyNWB tests locally including gallery and validation tests, and inspect all warnings and outputs
-  (`cd pynwb; python test.py -v > out.txt 2>&1`)
+  (`cd pynwb; git checkout dev; git pull; python test.py -v > out.txt 2>&1`)
 - [ ] Run HDMF-Zarr tests locally including gallery and validation tests, and inspect all warnings and outputs
-  (`cd hdmf-zarr; pytest && python test_gallery.py`)
+  (`cd hdmf-zarr; git checkout dev; git pull; pytest && python test_gallery.py`)
 - [ ] Test docs locally and inspect all warnings and outputs `cd docs; make clean && make html`
-- [ ] Push changes to this PR and make sure all PRs to be included in this release have been merged
-- [ ] Check that the readthedocs build for this PR succeeds (build latest to pull the new branch, then activate and
-  build docs for new branch): https://readthedocs.org/projects/hdmf/builds/
+- [ ] After pushing this branch to GitHub, manually trigger the "Run all tests" GitHub Actions workflow on this
+  branch by going to https://github.com/hdmf-dev/hdmf/actions/workflows/run_all_tests.yml, selecting
+  "Run workflow" on the right, selecting this branch, and clicking "Run workflow". Make sure all tests pass.
+- [ ] Check that the readthedocs build for this PR succeeds (see the PR check)
 
 ### After merging:
 1. Create release by following steps in `docs/source/make_a_release.rst` or use alias `git pypi-release [tag]` if set up
 2. After the CI bot creates the new release (wait ~10 min), update the release notes on the
    [GitHub releases page](https://github.com/hdmf-dev/hdmf/releases) with the changelog
-3. Check that the readthedocs "latest" and "stable" builds run and succeed
-4. Update [conda-forge/hdmf-feedstock](https://github.com/conda-forge/hdmf-feedstock) with the latest version number
-   and SHA256 retrieved from PyPI > HDMF > Download Files > View hashes for the `.tar.gz` file. Re-render as needed
+3. Check that the readthedocs "stable" build runs and succeeds
+4. Either monitor [conda-forge/hdmf-feedstock](https://github.com/conda-forge/hdmf-feedstock) for the
+   regro-cf-autotick-bot bot to create a PR updating the version of HDMF to the latest PyPI release, usually within
+   24 hours of release, or manually create a PR updating `recipe/meta.yaml` with the latest version number
+   and SHA256 retrieved from PyPI > HDMF > Download Files > View hashes for the `.tar.gz` file. Re-render and update
+   dependencies as needed.

.github/workflows/project_action.yml

@@ -20,15 +20,15 @@ jobs:
       - name: Add to Developer Board
         env:
           TOKEN: ${{ steps.generate_token.outputs.token }}
-        uses: actions/add-to-project@v0.5.0
+        uses: actions/add-to-project@v0.6.1
         with:
           project-url: https://github.com/orgs/hdmf-dev/projects/7
           github-token: ${{ env.TOKEN }}
 
       - name: Add to Community Board
         env:
           TOKEN: ${{ steps.generate_token.outputs.token }}
-        uses: actions/add-to-project@v0.5.0
+        uses: actions/add-to-project@v0.6.1
         with:
           project-url: https://github.com/orgs/hdmf-dev/projects/8
           github-token: ${{ env.TOKEN }}

.github/workflows/run_all_tests.yml

@@ -41,8 +41,8 @@ jobs:
           - { name: windows-python3.12            , test-tox-env: pytest-py312-pinned          , python-ver: "3.12", os: windows-latest }
           - { name: windows-python3.12-upgraded   , test-tox-env: pytest-py312-upgraded        , python-ver: "3.12", os: windows-latest }
           - { name: windows-python3.12-prerelease , test-tox-env: pytest-py312-prerelease      , python-ver: "3.12", os: windows-latest }
-          - { name: macos-python3.8-minimum       , test-tox-env: pytest-py38-minimum          , python-ver: "3.8" , os: macos-latest }
-          - { name: macos-python3.9               , test-tox-env: pytest-py39-pinned           , python-ver: "3.9" , os: macos-latest }
+          - { name: macos-python3.8-minimum       , test-tox-env: pytest-py38-minimum          , python-ver: "3.8" , os: macos-13 }
+          - { name: macos-python3.9               , test-tox-env: pytest-py39-pinned           , python-ver: "3.9" , os: macos-13 }
           - { name: macos-python3.10              , test-tox-env: pytest-py310-pinned          , python-ver: "3.10", os: macos-latest }
           - { name: macos-python3.11              , test-tox-env: pytest-py311-pinned          , python-ver: "3.11", os: macos-latest }
           - { name: macos-python3.11-optional     , test-tox-env: pytest-py311-optional-pinned , python-ver: "3.11", os: macos-latest }
@@ -105,7 +105,7 @@ jobs:
           - { name: windows-gallery-python3.11-optional  , test-tox-env: gallery-py311-optional-pinned , python-ver: "3.11", os: windows-latest }
           - { name: windows-gallery-python3.12-upgraded  , test-tox-env: gallery-py312-upgraded        , python-ver: "3.12", os: windows-latest }
           - { name: windows-gallery-python3.12-prerelease, test-tox-env: gallery-py312-prerelease      , python-ver: "3.12", os: windows-latest }
-          - { name: macos-gallery-python3.8-minimum      , test-tox-env: gallery-py38-minimum          , python-ver: "3.8" , os: macos-latest }
+          - { name: macos-gallery-python3.8-minimum      , test-tox-env: gallery-py38-minimum          , python-ver: "3.8" , os: macos-13 }
           - { name: macos-gallery-python3.11-optional    , test-tox-env: gallery-py311-optional-pinned , python-ver: "3.11", os: macos-latest }
           - { name: macos-gallery-python3.12-upgraded    , test-tox-env: gallery-py312-upgraded        , python-ver: "3.12", os: macos-latest }
           - { name: macos-gallery-python3.12-prerelease  , test-tox-env: gallery-py312-prerelease      , python-ver: "3.12", os: macos-latest }
@@ -233,7 +233,7 @@ jobs:
 
       - name: Install run dependencies
         run: |
-          pip install -e .
+          pip install .
           pip list
 
       - name: Conda reporting

.github/workflows/run_coverage.yml

@@ -55,18 +55,18 @@ jobs:
 
       - name: Install package
         run: |
-          python -m pip install -e .  # must install in editable mode for coverage to find sources
+          python -m pip install .
           python -m pip list
 
       - name: Run tests and generate coverage report
         run: |
-          pytest --cov
-          python -m coverage xml  # codecov uploader requires xml format
-          python -m coverage report -m
+          # coverage is configured in pyproject.toml
+          pytest --cov --cov-report=xml --cov-report=term  # codecov uploader requires xml format
 
       - name: Upload coverage to Codecov
         uses: codecov/codecov-action@v4
         with:
           fail_ci_if_error: true
+          file: ./coverage.xml
         env:
           CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

.github/workflows/run_hdmf_zarr_tests.yml

@@ -32,8 +32,7 @@ jobs:
           git clone https://github.com/hdmf-dev/hdmf-zarr.git --recurse-submodules
           cd hdmf-zarr
           python -m pip install -r requirements-dev.txt  # do not install the pinned install requirements
-          # must install in editable mode for coverage to find sources
-          python -m pip install -e .  # this will install a different version of hdmf from the current one
+          python -m pip install .  # this will install a different version of hdmf from the current one
           cd ..
           python -m pip uninstall -y hdmf  # uninstall the other version of hdmf
           python -m pip install .  # reinstall current branch of hdmf

.github/workflows/run_pynwb_tests.yml

@@ -32,8 +32,7 @@ jobs:
           git clone https://github.com/NeurodataWithoutBorders/pynwb.git --recurse-submodules
           cd pynwb
           python -m pip install -r requirements-dev.txt  # do not install the pinned install requirements
-          # must install in editable mode for coverage to find sources
-          python -m pip install -e .  # this will install a different version of hdmf from the current one
+          python -m pip install .  # this will install a different version of hdmf from the current one
           cd ..
           python -m pip uninstall -y hdmf  # uninstall the other version of hdmf
           python -m pip install .  # reinstall current branch of hdmf

.github/workflows/run_tests.yml

@@ -28,7 +28,7 @@ jobs:
           - { name: linux-python3.12-upgraded    , test-tox-env: pytest-py312-upgraded  , python-ver: "3.12", os: ubuntu-latest , upload-wheels: true }
           - { name: windows-python3.8-minimum    , test-tox-env: pytest-py38-minimum    , python-ver: "3.8" , os: windows-latest }
           - { name: windows-python3.12-upgraded  , test-tox-env: pytest-py312-upgraded  , python-ver: "3.12", os: windows-latest }
-          - { name: macos-python3.8-minimum      , test-tox-env: pytest-py38-minimum    , python-ver: "3.8" , os: macos-latest }
+          - { name: macos-python3.8-minimum      , test-tox-env: pytest-py38-minimum    , python-ver: "3.8" , os: macos-13 }
           - { name: macos-python3.12-upgraded    , test-tox-env: pytest-py312-upgraded  , python-ver: "3.12", os: macos-latest }
     steps:
       - name: Checkout repo with submodules
@@ -243,7 +243,7 @@ jobs:
 
       - name: Install run dependencies
         run: |
-          pip install -e .
+          pip install .
           pip list
 
       - name: Conda reporting

.pre-commit-config.yaml

@@ -18,7 +18,7 @@ repos:
 #     hooks:
 #     -   id: black
 -   repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.2.0
+    rev: v0.3.3
     hooks:
     -   id: ruff
 # -   repo: https://github.com/econchick/interrogate

CHANGELOG.md

@@ -1,10 +1,24 @@
 # HDMF Changelog
 
-## HDMF 3.12.3 (Upcoming)
+## HDMF 3.14.0 (Upcoming)
+
+### Enhancements
+- Updated `_field_config` to take `type_map` as an argument for APIs. @mavaylon1 [#1094](https://github.com/hdmf-dev/hdmf/pull/1094)
+- Added `TypeConfigurator` to automatically wrap fields with `TermSetWrapper` according to a configuration file. @mavaylon1 [#1016](https://github.com/hdmf-dev/hdmf/pull/1016)
+- Updated `TermSetWrapper` to support validating a single field within a compound array. @mavaylon1 [#1061](https://github.com/hdmf-dev/hdmf/pull/1061)
+- Updated testing to not install in editable mode and not run `coverage` by default. @rly [#1107](https://github.com/hdmf-dev/hdmf/pull/1107)
+- Add `post_init_method` parameter when generating classes to perform post-init functionality, i.e., validation. @mavaylon1 [#1089](https://github.com/hdmf-dev/hdmf/pull/1089)
 
 ### Bug fixes
 - Exposed `aws_region` to `HDF5IO` and downstream passes to `h5py.File`. @codycbakerphd [#1040](https://github.com/hdmf-dev/hdmf/pull/1040)
 
+## HDMF 3.13.0 (March 20, 2024)
+
+### Enhancements
+- Unwrap `TermSetWrapper` within the builder to support different backends more efficiently. @mavaylon1 [#1070](https://github.com/hdmf-dev/hdmf/pull/1070)
+- Added docs page that lists limitations of support for the HDMF specification language. @rly [#1069](https://github.com/hdmf-dev/hdmf/pull/1069)
+- Added warning when using `add_row` or `add_column` to add a ragged array to `DynamicTable` without an index parameter. @stephprince [#1066](https://github.com/hdmf-dev/hdmf/pull/1066)
+
 ## HDMF 3.12.2 (February 9, 2024)
 
 ### Bug fixes
@@ -131,8 +145,8 @@ will increase the minor version number to 3.10.0. See the 3.9.1 release notes be
 ## HDMF 3.6.0 (May 12, 2023)
 
 ### New features and minor improvements
-- Updated `ExternalResources` to have `FileTable` and new methods to query data. the `ResourceTable` has been removed along with methods relating to `Resource`. @mavaylon [#850](https://github.com/hdmf-dev/hdmf/pull/850)
-- Updated hdmf-common-schema version to 1.6.0. @mavaylon [#850](https://github.com/hdmf-dev/hdmf/pull/850)
+- Updated `ExternalResources` to have `FileTable` and new methods to query data. the `ResourceTable` has been removed along with methods relating to `Resource`. @mavaylon1 [#850](https://github.com/hdmf-dev/hdmf/pull/850)
+- Updated hdmf-common-schema version to 1.6.0. @mavaylon1 [#850](https://github.com/hdmf-dev/hdmf/pull/850)
 - Added testing of HDMF-Zarr on PR and nightly. @rly [#859](https://github.com/hdmf-dev/hdmf/pull/859)
 - Replaced `setup.py` with `pyproject.toml`. @rly [#844](https://github.com/hdmf-dev/hdmf/pull/844)
 - Use `ruff` instead of `flake8`. @rly [#844](https://github.com/hdmf-dev/hdmf/pull/844)
@@ -146,7 +160,7 @@ will increase the minor version number to 3.10.0. See the 3.9.1 release notes be
   [#853](https://github.com/hdmf-dev/hdmf/pull/853)
 
 ### Documentation and tutorial enhancements:
-- Updated `ExternalResources` how to tutorial to include the new features. @mavaylon [#850](https://github.com/hdmf-dev/hdmf/pull/850)
+- Updated `ExternalResources` how to tutorial to include the new features. @mavaylon1 [#850](https://github.com/hdmf-dev/hdmf/pull/850)
 
 ## HDMF 3.5.6 (April 28, 2023)
 
@@ -186,13 +200,13 @@ will increase the minor version number to 3.10.0. See the 3.9.1 release notes be
 
 ### Bug fixes
 - Fixed issue with conda CI. @rly [#823](https://github.com/hdmf-dev/hdmf/pull/823)
-- Fixed issue with deprecated `pkg_resources`. @mavaylon [#822](https://github.com/hdmf-dev/hdmf/pull/822)
-- Fixed `hdmf.common` deprecation warning. @mavaylon [#826]((https://github.com/hdmf-dev/hdmf/pull/826)
+- Fixed issue with deprecated `pkg_resources`. @mavaylon1 [#822](https://github.com/hdmf-dev/hdmf/pull/822)
+- Fixed `hdmf.common` deprecation warning. @mavaylon1 [#826]((https://github.com/hdmf-dev/hdmf/pull/826)
 
 ### Internal improvements
 - A number of typos fixed and Github action running codespell to ensure that no typo sneaks in [#825](https://github.com/hdmf-dev/hdmf/pull/825) was added.
-- Added additional documentation for `__fields__` in `AbstactContainer`. @mavaylon [#827](https://github.com/hdmf-dev/hdmf/pull/827)
-- Updated warning message for broken links. @mavaylon [#829](https://github.com/hdmf-dev/hdmf/pull/829)
+- Added additional documentation for `__fields__` in `AbstactContainer`. @mavaylon1 [#827](https://github.com/hdmf-dev/hdmf/pull/827)
+- Updated warning message for broken links. @mavaylon1 [#829](https://github.com/hdmf-dev/hdmf/pull/829)
 
 ## HDMF 3.5.1 (January 26, 2023)
 
@@ -211,9 +225,9 @@ will increase the minor version number to 3.10.0. See the 3.9.1 release notes be
 - Added ``HDMFIO.__del__`` to ensure that I/O objects are being closed on delete. @oruebel[#811](https://github.com/hdmf-dev/hdmf/pull/811)
 
 ### Minor improvements
-- Added support for reading and writing `ExternalResources` to and from denormalized TSV files. @mavaylon [#799](https://github.com/hdmf-dev/hdmf/pull/799)
-- Changed the name of `ExternalResources.export_to_sqlite` to `ExternalResources.to_sqlite`. @mavaylon [#799](https://github.com/hdmf-dev/hdmf/pull/799)
-- Updated the tutorial for `ExternalResources`. @mavaylon [#799](https://github.com/hdmf-dev/hdmf/pull/799)
+- Added support for reading and writing `ExternalResources` to and from denormalized TSV files. @mavaylon1 [#799](https://github.com/hdmf-dev/hdmf/pull/799)
+- Changed the name of `ExternalResources.export_to_sqlite` to `ExternalResources.to_sqlite`. @mavaylon1 [#799](https://github.com/hdmf-dev/hdmf/pull/799)
+- Updated the tutorial for `ExternalResources`. @mavaylon1 [#799](https://github.com/hdmf-dev/hdmf/pull/799)
 - Added `message` argument for assert methods defined by `hdmf.testing.TestCase` to allow developers to include custom error messages with asserts. @oruebel [#812](https://github.com/hdmf-dev/hdmf/pull/812)
 - Clarify the expected chunk shape behavior for `DataChunkIterator`. @oruebel [#813](https://github.com/hdmf-dev/hdmf/pull/813)
 
@@ -354,7 +368,7 @@ the fields (i.e., when the constructor sets some fields to fixed values). @rly
 - Plotted results in external resources tutorial. @oruebel (#667)
 - Added support for Python 3.10. @rly (#679)
 - Updated requirements. @rly @TheChymera (#681)
-- Improved testing for `ExternalResources`. @mavaylon (#673)
+- Improved testing for `ExternalResources`. @mavaylon1 (#673)
 - Improved docs for export. @rly (#674)
 - Enhanced data chunk iteration speeds through new ``GenericDataChunkIterator`` class.  @CodyCBakerPhD (#672)
 - Enhanced issue template forms on GitHub. @CodyCBakerPHD (#700)
@@ -430,7 +444,7 @@ the fields (i.e., when the constructor sets some fields to fixed values). @rly
 - Allow passing ``index=True`` to ``DynamicTable.to_dataframe()`` to support returning `DynamicTableRegion` columns
   as indices or Pandas DataFrame. @rly (#579)
 - Improve ``DynamicTable`` documentation. @rly (#639)
-- Updated external resources tutorial. @mavaylon (#611)
+- Updated external resources tutorial. @mavaylon1 (#611)
 
 ### Breaking changes and deprecations
 - Previously, when using ``DynamicTable.__getitem__`` or ``DynamicTable.get`` to access a selection of a
@@ -515,7 +529,7 @@ the fields (i.e., when the constructor sets some fields to fixed values). @rly
   - Add experimental namespace to HDMF common schema. New data types should go in the experimental namespace
     (hdmf-experimental) prior to being added to the core (hdmf-common) namespace. The purpose of this is to provide
     a place to test new data types that may break backward compatibility as they are refined. @ajtritt (#545)
-  - `ExternalResources` was changed to support storing both names and URIs for resources. @mavaylon (#517, #548)
+  - `ExternalResources` was changed to support storing both names and URIs for resources. @mavaylon1 (#517, #548)
   - The `VocabData` data type was replaced by `EnumData` to provide more flexible support for data from a set of
     fixed values.
   - Added `AlignedDynamicTable`, which defines a `DynamicTable` that supports storing a collection of sub-tables.

README.rst

@@ -33,8 +33,8 @@ Overall Health
 .. image:: https://github.com/hdmf-dev/hdmf/actions/workflows/ruff.yml/badge.svg
     :target: https://github.com/hdmf-dev/hdmf/actions/workflows/ruff.yml
 
-.. image:: https://github.com/hdmf-dev/hdmf/actions/workflows/check_external_links.yml/badge.svg
-    :target: https://github.com/hdmf-dev/hdmf/actions/workflows/check_external_links.yml
+.. image:: https://github.com/hdmf-dev/hdmf/actions/workflows/check_sphinx_links.yml/badge.svg
+    :target: https://github.com/hdmf-dev/hdmf/actions/workflows/check_sphinx_links.yml
 
 .. image:: https://github.com/hdmf-dev/hdmf/actions/workflows/run_pynwb_tests.yml/badge.svg
     :target: https://github.com/hdmf-dev/hdmf/actions/workflows/run_pynwb_tests.yml