Skip to content

Releases: andreped/GradientAccumulator

v0.2.1

25 Aug 17:31
Compare
Choose a tag to compare

This is a minor patch release.

What's changed:

  • Fixed typo by renaming use_acg to use_agc.

v0.2.0

24 Jun 16:29
Compare
Choose a tag to compare

What's Changed

  • AGC and mixed precision are now compatible
  • Support for both float16 and bfloat16 on GPU and TPU, respectively
  • Set default accum_steps in GAModelWrapper to 1
  • Improved documentation regarding usage of AGC, GPU/TPU, and recommended model format (SavedModel)
  • Support for 3D operations in AGC, such as Conv3D, by @dbouget in #17
  • Corrected mean reduction to before gradient computation @dbouget in #2

New Contributors

Full Changelog: v0.1.5...v0.2.0

v0.1.5

20 Jun 15:37
Compare
Choose a tag to compare

Changes:

  • Added mixed precision support (only float16 currently, which is compatible with NVIDIA GPUs)
  • Added adaptive gradient clipping support (normalization-free approach which works with GA)
  • Added CI test for AGC

Full Changelog: v0.1.4...v0.1.5

v0.1.4

05 Jun 14:30
Compare
Choose a tag to compare

Zenodo DOI release and updated README to contain updated documentation regarding installation and usage.

Changes:

  • Renamed n_gradients to accum_steps.
  • Added citation policy and Zenodo citation

Full Changelog: v0.1.3...v0.1.4

v0.1.3

05 Jun 13:38
Compare
Choose a tag to compare

GradientAccumulator is now available on PyPI :
https://pypi.org/project/gradient-accumulator/#files

Changes:

  • Added experimental mixed precision support
  • Added support for TF >= 2.2
  • Added support for Python >3.6
  • Added pytest to CI for unit testing
  • Added CI test for mixed precision
  • Added CI test for multi-input-output models
  • Added CI test for optimizer invariance
  • Added CI test for basic mnist training
  • Added CI test to verify that we get expected result for GA vs regular batch training

Full Changelog: v0.1.2...v0.1.3

v0.1.2

03 Jun 16:05
Compare
Choose a tag to compare

Changes:

  • Fixed critical bug regarding gradient updates (use MEAN reduction, instead of SUM reduction)
  • Now, GA yields identical results compared to regular batch training
  • Added unit tests with pytest to yield AssertionError if results are different
  • Added compatibility with sample_weight - now GAModelWrapper should be fully compatible with model.compile/fit

Full Changelog: v0.1.1...v0.1.2

v0.1.1

01 Jun 20:54
Compare
Choose a tag to compare

Changes:

  • Swapped optimizer wrapper solution with Model wrapper solution
  • Enables adding gradient accumulation support for "any" tf.keras.Model by simply overloading the train_step method
  • Added convenience class GAModelWrapper that handles all this for you - just provide the model!
  • Solution should also be more compatible with older TF versions, as train_step overloading was added already in TF 2.2.

Full Changelog: v0.1.0...v0.1.1

v0.1.0

31 May 11:46
Compare
Choose a tag to compare

First release of the GradientAccumulator package that enables usage of accumulated gradients in TensorFlow 2.x by simply wrapping an optimizer.

Currently, compatible with Python 3.7-3.9, tested with TensorFlow 2.8.0 and 2.9.1, and cross-platform compatible (Windows, Ubuntu, and macOS).

Full Changelog: https://github.com/andreped/GradientAccumulator/commits/v0.1.0