Rocksdb Background IO error: While pwrite to file at offset ***: Invalid argument

[yongpan0709]

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://www.facebook.com/groups/rocksdb.dev
When go-tpc tpcc runs the TiDB cluster created by operator way, TiKV goes into CrashLoopBackOff. The details of go-tpc is:
./bin/go-tpc tpcc --warehouses 2000 prepare -T 100

Expected behavior

Cluster should go on running. Or throwing some errors.

Actual behavior

2 TiKV nodes go into CrashLoopBackOff.

Steps to reproduce the behavior

As mentioned above. Or refer to facebook#5178

The more infor about this Error:
BCC Info （2022-05-07 18:23:47 +0800）
The COUNT and POS are not aligned by 512 bits.

Stack Info:（2022-05-07 10:23:47 +0000）

TiKV Version:
Release Version: 5.4.0
Edition: Community
Git Commit Hash: b5262299604df88711d9ed4b84d43e9c507749a2
Git Commit Branch: heads/refs/tags/v5.4.0
UTC Build Time: Unknown (env var does not exist when building)
Rust Version: rustc 1.56.0-nightly (2faabf579 2021-07-27)
Enable Features: jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure
Profile: dist_release