Remove MLActivations definitely not usable with recurrent ops

[a-sully]

Follow-up to #664, without going so far as #689

Prior to #664, an MLActivation may be used for either op fusion or with recurrent operators (lstm and gru). Now it's only the latter, and some operators which may be created as an MLActivation no longer make sense to be passed as activations:

• clamp
• gelu (no longer being removed after feedback from reviewers)
• softmax

Note that this PR is not a replacement for #689. That issue still stands on its own. Until we reach consensus on that issue, we should remove the MLActivations which were clearly targeted at the op fusion use case.

---

Preview | Diff

[a-sully]

@huningxin FYI. This should simplify https://crrev.com/c/5495877

This list of operators is based on this code in Chromium's DML implementation. If there are others you think should be added to this PR, please let me know. I started with a minimal list to make this PR as uncontroversial as possible :)

[fdwr]

👍

[huningxin]

@a-sully

@huningxin FYI. This should simplify https://crrev.com/c/5495877

Ack!

This list of operators is based on this code in Chromium's DML implementation.

Unfortunately that implementation may be mis-leading. They are not supported because they are only available in higher DirectML feature level, rather than they are not useful. I opened a Chromium issue regarding to that.

@fdwr

Note {ELU, CELU, RELU, GELU} are supported by DML RNN's including RNN, LSTM, GRU.

I suppose the current criteria is to keep activations that could be used by lstm and gru. So, should gelu be kept?

I don't find DirectML RNN ops supported activations list in MSDN, for example DML_GRU_OPERATOR_DESC. Did I miss anything?

[a-sully]

Added back gelu based on feedback!

I don't find DirectML RNN ops supported activations list in MSDN, for example DML_GRU_OPERATOR_DESC. Did I miss anything?

Yup., that's why I was using the Chromium implementation as the source of truth. It would be nice to have these constraints documented! :P

Please merge at your convenience

[fdwr]

It would be nice to have these constraints documented! :P

Indeed - I'll follow up with our doc writer @stevewhims.

[huningxin]

LGTM!

[fdwr]

I don't find DirectML RNN ops supported activations list in MSDN, for example DML_GRU_OPERATOR_DESC. Did I miss anything?

@huningxin : Until the docs are updated:

✅:
DML_OPERATOR_ACTIVATION_IDENTITY
DML_OPERATOR_ACTIVATION_LINEAR
DML_OPERATOR_ACTIVATION_ELU
DML_OPERATOR_ACTIVATION_RELU
DML_OPERATOR_ACTIVATION_CELU
DML_OPERATOR_ACTIVATION_GELU
DML_OPERATOR_ACTIVATION_SCALED_ELU
DML_OPERATOR_ACTIVATION_LEAKY_RELU
DML_OPERATOR_ACTIVATION_THRESHOLDED_RELU
DML_OPERATOR_ACTIVATION_PARAMETERIZED_RELU
DML_OPERATOR_ACTIVATION_PARAMETRIC_SOFTPLUS
DML_OPERATOR_ACTIVATION_SIGMOID
DML_OPERATOR_ACTIVATION_HARD_SIGMOID
DML_OPERATOR_ACTIVATION_SOFTPLUS
DML_OPERATOR_ACTIVATION_SOFTSIGN
DML_OPERATOR_ACTIVATION_TANH
DML_OPERATOR_ACTIVATION_SCALED_TANH
DML_OPERATOR_ACTIVATION_SHRINK
DML_OPERATOR_ACTIVATION_SWISH
DML_OPERATOR_ACTIVATION_HARD_SWISH

❌:
DML_OPERATOR_ELEMENT_WISE_CLIP
DML_OPERATOR_ACTIVATION_SOFTMAX
DML_OPERATOR_ACTIVATION_SOFTMAX1
DML_OPERATOR_ACTIVATION_LOG_SOFTMAX
DML_OPERATOR_ACTIVATION_LOG_SOFTMAX1
DML_OPERATOR_ACTIVATION_HARDMAX
DML_OPERATOR_ACTIVATION_HARDMAX1