sccache server shuts down unexpectedly

[xiaoluzi0050]

The compilation normally finishes in about 4 minutes, but occasionally takes 26 minutes to complete. There have been no changes to the code, and the compilation is done inside a Docker container with an unchanged environment. The last 20 minutes are consistently stuck at sccache execution, as shown in the logs：

427882 [2024-07-31T12:27:17Z TRACE hype]::proto::h1::io] received 0 bytes 427877,1 99%
427883 [2024-07-31T12:27:17Z TRACE hyper::proto::h1::co]n] found EOF on idle connection, closing 427878,1 99%
427884 [2024-07-31T12:27:17Z TRACE hyper::proto::h1::conn] State::close_read() 427879,1 99%
427885 [2024-07-31T12:27:17Z TRACE hyper::proto::h1::conn] flushed({role=client}): State { reading: Closed, writing: Init, keep_alive: Disabled } 427880,1 99%
427886 [2024-07-31T12:27:17Z TRACE hyper::proto::h1::conn] shut down IO complete 427881,1 99%
427887 [2024-07-31T12:27:17Z TRACE mio::poll] deregisteri]g event source from poller 427882,1 99%
427888 [2024-07-31T12:27:17Z TRACE want] sig]al: Closed 427883,1 99%
427889 [2024-07-31T12:27:23Z TRACE hype]::proto::h1::io] received 0 bytes 427884,1 99%
427890 [2024-07-31T12:27:23Z TRACE hyper::proto::h1::co]n] found EOF on idle connection, closing 427885,1 99%
427891 [2024-07-31T12:27:23Z TRACE hyper::proto::h1::conn] State::close_read() 427886,1 99%
427892 [2024-07-31T12:27:23Z TRACE hyper::proto::h1::conn] flushed({role=client}): State { reading: Closed, writing: Init, keep_alive: Disabled } 427887,1 99%
427893 [2024-07-31T12:27:23Z TRACE hyper::proto::h1::conn] shut down IO complete 427888,1 99%
427894 [2024-07-31T12:27:23Z TRACE mio::poll] deregisteri]g event source from poller 427889,1 99%
427895 [2024-07-31T12:27:23Z TRACE want] sig]al: Closed 427890,1 99%
427896 [2024-07-31T12:28:45Z TRACE hype]::client::pool] idle interval checking for expired 427891,1 99%
427897 [2024-07-31T12:28:45Z TRACE hyper::client::pool] idle interval evicting closed for ("http",*oss-cn-shenzhen-internal.aliyuncs.com) 427892,1 99%
427898 [2024-07-31T12:28:45Z TRACE hyper::client::pool] idle interval evicting closed for ("http",*oss-cn-shenzhen-internal.aliyuncs.com) 427893,1 99%
427899 [2024-07-31T12:28:45Z TRACE hyper::client::pool] idle interval evicting closed for ("http",*oss-cn-shenzhen-internal.aliyuncs.com) 427894,1 99%
427900 [2024-07-31T12:28:45Z TRACE hyper::client::pool] idle interval evicting closed for ("http",*oss-cn-shenzhen-internal.aliyuncs.com) 427895,1 99%
427901 [2024-07-31T12:28:45Z TRACE hyper::client::pool] idle interval evicting closed for ("http",*oss-cn-shenzhen-internal.aliyuncs.com) 427896,1 99%
427902 [2024-07-31T12:28:45Z TRACE hyper::client::pool] idle interval evicting closed for ("http",*oss-cn-shenzhen-internal.aliyuncs.com) 427897,1 99%
427903 [2024-07-31T12:28:45Z TRACE hyper::client::pool] idle interval evicting closed for ("http",*oss-cn-shenzhen-internal.aliyuncs.com) 427898,1 99%
427904 [2024-07-31T12:28:45Z TRACE hyper::client::pool] idle interval evicting closed for ("http",*oss-cn-shenzhen-internal.aliyuncs.com) 427899,1 99%
427905 [2024-07-31T12:28:45Z TRACE hyper::client::pool] idle interval evicting closed for ("http",*oss-cn-shenzhen-internal.aliyuncs.com) 427900,1 99%
427906 [2024-07-31T12:28:45Z TRACE hyper::client::pool] idle interval evicting closed for ("http",*oss-cn-shenzhen-internal.aliyuncs.com) 427901,1 99%
427907 [2024-07-31T12:28:45Z TRACE hyper::client::pool] idle interval evicting closed for ("http",*oss-cn-shenzhen-internal.aliyuncs.com) 427902,1 99%
427908 [2024-07-31T12:28:45Z TRACE hyper::client::pool] idle interval evicting closed for ("http",*oss-cn-shenzhen-internal.aliyuncs.com) 427869,1 99%
427908 [2024-07-31T12:28:45Z TRACE hyper::client::pool] idle interval evicting closed for ("http",*oss-cn-shenzhen-internal.aliyuncs.com) 427903,1 99%
427909 [2024-07-31T12:30:15Z TRACE hyper::client::pool] idle interval checking for expired 427904,1 99%
427910 [2024-07-31T12:31:45Z TRACE hyper::client::pool] idle interval checking for expired 427905,1 99%
427911 [2024-07-31T12:33:15Z TRACE hyper::client::pool] idle interval checking for expired 427906,1 99%
427912 [2024-07-31T12:34:45Z TRACE hyper::client::pool] idle interval checking for expired 427907,1 99%
427913 [2024-07-31T12:36:15Z TRACE hyper::client::pool] idle interval checking for expired 427908,1 99%
427914 [2024-07-31T12:37:45Z TRACE hyper::client::pool] idle interval checking for expired 427909,1 99%
427915 [2024-07-31T12:39:15Z TRACE hyper::client::pool] idle interval checking for expired 427910,1 99%
427916 [2024-07-31T12:40:45Z TRACE hyper::client::pool] idle interval checking for expired 427911,1 99%
427917 [2024-07-31T12:42:15Z TRACE hyper::client::pool] idle interval checking for expired 427912,1 99%
427918 [2024-07-31T12:43:45Z TRACE hyper::client::pool] idle interval checking for expired 427913,1 99%
427919 [2024-07-31T12:45:15Z TRACE hyper::client::pool] idle interval checking for expired 427914,1 99%
427920 [2024-07-31T12:46:45Z TRACE hyper::client::pool] idle interval checking for expired 427915,1 99%
427921 [2024-07-31T12:48:15Z TRACE hyper::client::pool] idle interval checking for expired 427916,1 99%
427922 [2024-07-31T12:48:40Z TRACE mio::poll] deregist]ring event source from poller 427917,1 99%
427923 [2024-07-31T12:48:40Z TRACE mio::poll] deregistering event source from poller 427918,1 99%
427924 [2024-07-31T12:48:40Z DEBUG sccache::]ompiler::compiler] [em_data.cpp.o]: Compiled in 1379.075 s, storing in cache 427919,1 99%
427925 [2024-07-31T12:48:43Z DEBUG sccache::]ompiler::compiler] [em_data.cpp.o]: Created cache artifact in 2.915 s 427920,1 99%
427926 [2024-07-31T12:48:43Z DEBUG sccache::server] compile re]ult: cache miss 427921,1 99%
[ ]
427927 [2024-07-31T12:48:43Z DEBUG sccache::server] stats after compile result: ServerStats { compile_requests: 2402, requests_unsupported_compiler: 0, requests_not_compile: 0, requests_not_cacheable: 0, requests_executed: 2402, cache_err [rs: PerLanguageCount { counts: {}, adv_cou]ts: {} }, cache_hits: PerLanguageCount { counts: {"C/C++": 2312}, adv_counts: {"c++ [gcc]": 1527, "c [gcc]": 785} }, cache_misses: PerLanguageCount { counts: {"C/C++": 90}, adv_counts: {r [ [gcc]": 10, "c++ [gcc]": 80} }, cache_tim]outs: 0, cache_read_errors: 0, non_cacheable_compilations: 0, forced_recaches: 0, cache_write_errors: 0, cache_writes: 89, cache_write_duration: 3.656143265s, cache_read_hit_duration: 16r .970775474s, compiler_write_duration: 1677.436806419s, compile_fails: 0, not_cached: {}, dist_compiles: {}, dist_errors: 0 } 427925,1 99%" 27928 [2024-07-31T12:48:43Z TRACE sccache::server] CompileFinished retcode: exit status: 0 427926,1 99%6
427929 [2024-07-31T12:48:43Z TRACE tokio_util::cod]c::framed_impl] flushing framed transport
427930 [2024-07-31T12:48:43Z TRACE tokio_util::codec::framed_impl] writing; remaining=34
427931 [2024-07-31T12:48:43Z DEBUG opendal::services] service=s3 ]peration=write path=8/e/8/8e8eae55b7d23e3375c7947effa08cea7b02f6ac28cf9337cd9f6dbc06ad16ab -> started
427932 [2024-07-31T12:48:43Z TRACE tokio_util::codec]:framed_impl] framed transport flushed 427927,1 99%
427933 [2024-07-31T12:48:43Z DEBUG opendal::services] service=s3 ]peration=write path=8/e/8/8e8eae55b7d23e3375c7947effa08cea7b02f6ac28cf9337cd9f6dbc06ad16ab -> start writing 427928,1 99%
427934 [2024-07-31T12:48:43Z TRACE opendal::services] service=s3 operation=Writer::write path=8/e/8/8e8eae55b7d23e3375c7947effa08cea7b02f6ac28cf9337cd9f6dbc06ad16ab written=77411766B -> input data 77411766B, write 774117427929,1 99%
427935 [2024-07-31T12:48:43Z DEBUG reqsign::aws::v4]]calculated scope: 20240731/us-east-1/s3/aws4_request 427930,1 99%
427936 [2024-07-31T12:48:43Z DEBUG reqsign::aws::v4] calculated string to sign: AWS4-HMAC-SHA256 427931,1 99%
427937 [ 20240731T124843Z ] 427932,1 99%
427938 20240731/us-east-1/s3/aws4_request 427933,1 99%
427939 c9e0f76311376f965945993f8d521416aca0cab60015468411ba6a732693fa01 427934,1 99%