New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MHA_MQA_GQA代码问题 #10

Open

ZhuJiaqi9905 opened this issue Aug 19, 2024 · 0 comments

ZhuJiaqi9905 commented Aug 19, 2024

在attention的计算时的注意力拼接部分。output的维度是想从[batch_size, num_heads, seq_len, head_dim]变为[batch_size, seq_len, model_dim]。因此感觉transpose有误。应该从

## 对注意力输出进行拼接
output = (
    output.transpose(-1, -2)
    .contiguous()
    .view(batch_size, -1, self.head_dim * self.num_heads)
)

改为

## 对注意力输出进行拼接
output = (
    output.transpose(-2, -3)
    .contiguous()
    .view(batch_size, -1, self.head_dim * self.num_heads)
)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment