[Bug]: [1105 main tke regression] tpch 1t 4cn test report 'wait notify message timeout'.

[Ariznawlll]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Branch Name

main

Commit ID

5e4d063

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

job url: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/11666410553/job/32517908805

tpch测试期间日志（UTC时间）
https://grafana.ci.matrixorigin.cn/explore?panes=%7B%221u3%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-main-nightly-5e4d0638a-20241104%5C%22%7D%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221730790063000%22,%22to%22:%221730790183000%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

trigger workflow test on tke

Additional information

No response

[badboynt1]

@m-schen 麻烦看一下

[m-schen]

首先这个肯定是一个偶先的错误，
暂时感觉原因是部分pipeline因为网络 / 本地client not ready等原因发送失败，转为在本地执行导致的卡死。

[m-schen]

首先这个肯定是一个偶先的错误， 暂时感觉原因是部分pipeline因为网络 / 本地client not ready等原因发送失败，转为在本地执行导致的卡死。

检查发现这段逻辑很早就被移除了，目前没有这个逻辑，
感觉是因为网络出问题丢包导致的情况。

[m-schen]

https://grafana.ci.matrixorigin.cn/explore?panes=%7B%221u3%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-main-nightly-5e4d0638a-20241104%5C%22%7D%20%7C%3D%20%60send%20notify%20message%20failed%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221730649600000%22,%22to%22:%221730822399000%22%7D%7D%7D&schemaVersion=1&orgId=1

暂时记录

[m-schen]

root cause是网络出了问题，导致有一个notify message发送失败，因此报出了该错误。

[m-schen]

我会尝试修改让它能马上抛出这个rpc timeout的错误