Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add embedding, rms_norm, rope #2517

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add embedding, rms_norm, rope #2517

wants to merge 2 commits into from

Conversation

FindHao
Copy link
Member

@FindHao FindHao commented Oct 18, 2024

Add embedding, rms_norm, rope from liger kernels

Test Plan:

% python run_benchmark.py triton --op embedding,rms_norm,rope --num-inputs 1
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:04<00:00,  4.77s/it]
  x_val    torch_embedding-latency    liger_embedding-latency    inductor_embedding-latency
-------  -------------------------  -------------------------  ----------------------------
      0                   0.102528                   0.035488                      0.034304
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.12it/s]
  x_val    llama_rms-latency    liger_rms-latency    inductor_rms-latency
-------  -------------------  -------------------  ----------------------
      0             0.045856             0.014624                0.014496
  0%|                                                                                                                                                         | 0/1 [00:00<?, ?it/s]`LlamaRotaryEmbedding` can now be fully parameterized by passing the model config through the `config` argument. All other arguments will be removed in v4.46
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.17s/it]
  x_val    apply_rotary_pos_emb-latency    liger_rotary_pos_emb-latency    inductor_rotary_pos_emb_full_op-latency
-------  ------------------------------  ------------------------------  -----------------------------------------
      0                        0.294336                        0.050528                                   0.056128

@facebook-github-bot
Copy link
Contributor

@FindHao has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants