Support for optimizers which don't have "fused" parameter such as grokadamw and 8bit bnb

[mtasic85]

This fixes #1743

1. This changes bring support for optimizers which do not have "fused" parameter. Examples:

optimizer: grokadamw.GrokAdamW
optimizer: bitsandbytes.optim.AdamW8bit
optimizer: bitsandbytes.optim.PagedAdamW8bit

2. Additionally, pytorch optimizers can be written in following format:

optimizer: torch.optim.AdamW
optimizer: AdamW

3. In case of yaml config file, if value for optimizer: isn't str nor dict, error is thrown.

[rasbt]

Thanks for the PR! I am thinking maybe we can make this even more general so that it would also remove other unsupported arguments if present, something like

Suggested change

	if "." in optimizer:
	class_module, class_name = optimizer.rsplit(".", 1)
	else:
	class_module, class_name = "torch.optim", optimizer
	module = __import__(class_module, fromlist=[class_name])
	optimizer_cls = getattr(module, class_name)
	if "fused" in kwargs and "fused" not in inspect.signature(optimizer_cls).parameters:
	kwargs = dict(kwargs) # copy
	del kwargs["fused"]
	if "." in optimizer:
	class_module, class_name = optimizer.rsplit(".", 1)
	else:
	class_module, class_name = "torch.optim", optimizer
	module = __import__(class_module, fromlist=[class_name])
	optimizer_cls = getattr(module, class_name)
	valid_params = set(inspect.signature(optimizer_cls).parameters)
	kwargs = {key: value for key, value in kwargs.items() if key in valid_params}
	optimizer = optimizer_cls(model_parameters, **kwargs)

[rasbt]

Thanks for the PR, this looks really good to me. What do you think about the suggestions above?

[mtasic85]

@rasbt Can you please check CI/CD logs?

AttributeError: 'GPTNeoXRotaryEmbedding' object has no attribute 'cos_cached'

This should not be issue with this pull request but appears for some reason.

It looks it is related to #1745

[rasbt]

@mtasic85 I noticed that elsewhere too. It's related to the recent transformers release (4.45) yesterday that introduced a backward incompatible change. You are right, it has nothing to do with your PR. And no worries, I will fix this now.

[rasbt]

Alright! So there were two things that broke CI yesterday: a new transformers release and a new jsonargparse release. Should be all addressed now.

[rasbt]

Looks good now, it should be ready to merge. Thanks again for submitting this PR!