Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support nvcc in sccache-dist #2247

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open

Support nvcc in sccache-dist #2247

wants to merge 13 commits into from
+3,038 −465

Commits on Nov 8, 2024

  1. * Refactor start_compile_task() into an async function 

    * Spawn the tokio compilation task in `check_compiler()` instead of in `start_compile_task()`
* Map the tokio compilation task into a stream, and return that stream as the response body

These changes reduce the amount of memory leaked when building the default linux target with glibc.

Refactoring `start_compile_task()` to be an async function is a necessary change to support re-entrancy for sccache-dist nvcc support.
    @trxcllnt @sylvestre
    trxcllnt authored and sylvestre committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    22a55c4 View commit details
    Browse the repository at this point in the history

  2. Ensure use_preprocessor_cache_mode considers the current value of t… 

    …he `SCCACHE_DIRECT` envvar.
    @trxcllnt @sylvestre
    trxcllnt authored and sylvestre committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    10ba495 View commit details
    Browse the repository at this point in the history

  3. Refactor CompileCommand into a trait 

    * Refactor `CompileCommand` into a trait with an execute function that accepts an `SccacheService<T>` ref and a `CommandCreatorSync` ref
* Rename `CompileCommand` struct to `SingleCompileCommand`
* Update all `generate_compile_command()` implementations to return a boxed `CompileCommand` trait

This allows compilers to return custom implementations of the `CompileCommand` trait that execute more complex logic than simply launching a subprocess.
    @trxcllnt @sylvestre
    trxcllnt authored and sylvestre committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    1aa9d88 View commit details
    Browse the repository at this point in the history

  4. Add an extra_dist_files field to the ParsedArguments struct, so c… 

    …ompilers can add additional files to be included for each distributed compilation
    @trxcllnt @sylvestre
    trxcllnt authored and sylvestre committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    8d9f910 View commit details
    Browse the repository at this point in the history

  5. Add an UnhashedFlag argument type for flags which should not affect… 

    … the computed hash
    @trxcllnt @sylvestre
    trxcllnt authored and sylvestre committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    de276b1 View commit details
    Browse the repository at this point in the history

  6. Update Nvcc compiler to support distributed compilation 

    * Adds top-level `Cicc` and `Ptxas` compiler types
* Updates `Nvcc` compiler implementation to decompose nvcc calls into its constituent subcompiler invocations via the `nvcc --dryrun` flag
* Bumps the `sccache-dist` request timeout from 5 to 10 minutes, because nvcc compilations can take a while
* Updates the CUDA tests, separates into tests for nvcc and clang since now their behavior is different
* Fixes lint
    @trxcllnt @sylvestre
    trxcllnt authored and sylvestre committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    b27a186 View commit details
    Browse the repository at this point in the history

  7. Detect both nvcc and host compiler version so we can safely revert to… 

    … caching outer nvcc invocations
    @trxcllnt @sylvestre
    trxcllnt authored and sylvestre committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    f14f10d View commit details
    Browse the repository at this point in the history

  8. fix exit_status propagation, handle long nvcc language option (--x)

    @trxcllnt @sylvestre
    trxcllnt authored and sylvestre committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    83f1332 View commit details
    Browse the repository at this point in the history

  9. Ensure compatibility with MSVC compatibility on Windows

    @trxcllnt @sylvestre
    trxcllnt authored and sylvestre committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    07bacc1 View commit details
    Browse the repository at this point in the history

  10. Test nvcc and clang-cuda in workflows/ci.yml 

    * Test nvcc and clang-cuda in workflows/ci.yml
* Fix clang-cuda tests
* Ensure /tmp/sccache_*.txt files are included in failed job artifacts on Windows
    @trxcllnt @sylvestre
    trxcllnt authored and sylvestre committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    91e7b4f View commit details
    Browse the repository at this point in the history

  11. Attempt to fix intermittent config_from_env() test failures by ensuri… 

    …ng envvars are reset before unwrapping/asserting
    @trxcllnt @sylvestre
    trxcllnt authored and sylvestre committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    a5704b2 View commit details
    Browse the repository at this point in the history

  12. ensure SCCACHE_NO_CACHE calls aren't tracked as an uncacheable compil… 

    …ations in the stats
    @trxcllnt @sylvestre
    trxcllnt authored and sylvestre committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    4e1646b View commit details
    Browse the repository at this point in the history

  13. Make start_compile_task spawn a tokio task again

    @trxcllnt @sylvestre
    trxcllnt authored and sylvestre committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    99ad742 View commit details
    Browse the repository at this point in the history