Test file_gets_cleans_up_on_cache_eviction failed

[aaronmondal]

Observed in https://github.com/TraceMachina/nativelink/actions/runs/11096476887/job/30826349710?pr=1375

test file_gets_cleans_up_on_cache_eviction ... FAILED

  2024-09-29T22:41:58.120112Z ERROR nativelink_store::filesystem_store: Failed to rename file, err: Error { code: NotFound, messages: ["No such file or directory (os error 2)", "Failed to rename temp file to final path \"/home/runner/.bazel/sandbox/linux-sandbox/670/execroot/_main/_tmp/4570b71ef2441e1548bbbb5c539c5a14/5005206170696388994/content_path/0123456789abcdef000000000000000000010000000000000123456789abcdef-10\""] }, from_path: "/home/runner/.bazel/sandbox/linux-sandbox/670/execroot/_main/_tmp/4570b71ef2441e1548bbbb5c539c5a14/1788747981940690587/temp_path/0123456789abcdef000000000000000000010000000000001200000000000000-10", final_path: "/home/runner/.bazel/sandbox/linux-sandbox/670/execroot/_main/_tmp/4570b71ef2441e1548bbbb5c539c5a14/5005206170696388994/content_path/0123456789abcdef000000000000000000010000000000000123456789abcdef-10"
    at nativelink-store/src/filesystem_store.rs:701
    in nativelink_store::filesystem_store::filesystem_store_emplace_file

test rename_on_insert_fails_due_to_filesystem_error_proper_cleanup_happens ... ok
test get_file_size_uses_block_size ... ok
  2024-09-29T22:41:58.149180Z  WARN nativelink_store::filesystem_store: Failed to delete file, path: "/home/runner/.bazel/sandbox/linux-sandbox/670/execroot/_main/_tmp/4570b71ef2441e1548bbbb5c539c5a14/16993639508665717028/temp_path/0123456789abcdef000000000000000000020000000000001500000000000000-10", err: Error { code: NotFound, messages: ["No such file or directory (os error 2)"] }
    at nativelink-store/src/filesystem_store.rs:516

failures:

---- file_gets_cleans_up_on_cache_eviction stdout ----
Error: Error { code: Internal, messages: ["Sender dropped before sending EOF", "During first read of buf_channel::take()", "Error reading remaining bytes"] }


failures:
    file_gets_cleans_up_on_cache_eviction

test result: FAILED. 18 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.77s

cc @allada @blakehatch Could this be the cause of the cache issue we recently observed?

[MarcusSorealheis]

@aaronmondal are we sure this is a bug?

[aaronmondal]

Yeah, it's a flaky test which most likely means that there is some (rare?) case in which the filesystem store doesn't behave as intended.

[SchahinRohani]

Observed again in the asan workflow here: https://github.com/TraceMachina/nativelink/actions/runs/11486816086/job/31988808419#step:4:347

[PaddyMasum393]

I will solve this one, @MarcusSorealheis, @aaronmondal, @SchahinRohani