Code

Add SophiaG optimizer#844

Code

Open

PeanutButterRat wants to merge 18 commits intofacebookresearch:mainfacebookresearch/fairseq2:mainfrom PeanutButterRat:sophia-optimizerPeanutButterRat/fairseq2:sophia-optimizerCopy head branch name to clipboard

+339-1

Commits

Commits on Oct 28, 2024

• Create sophia.py

  PeanutButterRat

  committed
  d7feed2

• Register SophiaG as a usable optimizer

  PeanutButterRat

  committed
  72e4672

• Add differentiable default

  PeanutButterRat

  committed
  daa4e61

Commits on Oct 30, 2024

• Add finetuning recipe for SophiaG

  PeanutButterRat

  committed
  6205ed3

• Add optimizer field to SophiaG recipe

  PeanutButterRat

  committed
  5ed973c

• Move SophiaG recipe to instruction_finetune.py

  PeanutButterRat

  committed
  f8d1eaa

• Format and add type hints

  PeanutButterRat

  committed
  3f28592

Commits on Nov 1, 2024

• Join sophiag and _single_tensor_sophiag

  PeanutButterRat

  committed
  f006e3b

• Add license header

  PeanutButterRat

  committed
  c7b5ce2

• Add docstrings to SophiaG and SophiaGConfig

  PeanutButterRat

  committed
  e7f3338

Commits on Nov 6, 2024

• Add hessian update and k hyperparameter

  PeanutButterRat

  committed
  e5cdff4

• Fix usage of k hyperparameter

  PeanutButterRat

  committed
  e3c81cd

Commits on Nov 9, 2024

• Add type hints and format SophiaG

  PeanutButterRat

  committed
  d1e6d57

• Remove unused imports

  PeanutButterRat

  committed
  50fd1fb

Commits on Nov 11, 2024

• Explicitly cast step to an int

  PeanutButterRat

  committed
  8e13174

• Create test_sophiag.py

  PeanutButterRat

  committed
  5f30548

Commits on Nov 14, 2024

• Add SophiaG unit test

  PeanutButterRat

  committed
  8ae5403

• Change SophiaG test to check weights

  PeanutButterRat

  committed
  148f647