Releases: andreped/GradientAccumulator
v0.2.1
v0.2.0
What's Changed
- AGC and mixed precision are now compatible
- Support for both float16 and bfloat16 on GPU and TPU, respectively
- Set default accum_steps in GAModelWrapper to 1
- Improved documentation regarding usage of AGC, GPU/TPU, and recommended model format (SavedModel)
- Support for 3D operations in AGC, such as Conv3D, by @dbouget in #17
- Corrected mean reduction to before gradient computation @dbouget in #2
New Contributors
Full Changelog: v0.1.5...v0.2.0
v0.1.5
Changes:
- Added mixed precision support (only
float16
currently, which is compatible with NVIDIA GPUs) - Added adaptive gradient clipping support (normalization-free approach which works with GA)
- Added CI test for AGC
Full Changelog: v0.1.4...v0.1.5
v0.1.4
Zenodo DOI release and updated README to contain updated documentation regarding installation and usage.
Changes:
- Renamed
n_gradients
toaccum_steps
. - Added citation policy and Zenodo citation
Full Changelog: v0.1.3...v0.1.4
v0.1.3
GradientAccumulator is now available on PyPI :
https://pypi.org/project/gradient-accumulator/#files
Changes:
- Added experimental mixed precision support
- Added support for TF >= 2.2
- Added support for Python >3.6
- Added pytest to CI for unit testing
- Added CI test for mixed precision
- Added CI test for multi-input-output models
- Added CI test for optimizer invariance
- Added CI test for basic mnist training
- Added CI test to verify that we get expected result for GA vs regular batch training
Full Changelog: v0.1.2...v0.1.3
v0.1.2
Changes:
- Fixed critical bug regarding gradient updates (use MEAN reduction, instead of SUM reduction)
- Now, GA yields identical results compared to regular batch training
- Added unit tests with pytest to yield AssertionError if results are different
- Added compatibility with
sample_weight
- now GAModelWrapper should be fully compatible with model.compile/fit
Full Changelog: v0.1.1...v0.1.2
v0.1.1
Changes:
- Swapped optimizer wrapper solution with Model wrapper solution
- Enables adding gradient accumulation support for "any" tf.keras.Model by simply overloading the train_step method
- Added convenience class GAModelWrapper that handles all this for you - just provide the model!
- Solution should also be more compatible with older TF versions, as train_step overloading was added already in TF 2.2.
Full Changelog: v0.1.0...v0.1.1
v0.1.0
First release of the GradientAccumulator package that enables usage of accumulated gradients in TensorFlow 2.x by simply wrapping an optimizer.
Currently, compatible with Python 3.7-3.9, tested with TensorFlow 2.8.0 and 2.9.1, and cross-platform compatible (Windows, Ubuntu, and macOS).
Full Changelog: https://github.com/andreped/GradientAccumulator/commits/v0.1.0