Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Split sharded Llama dataset exporting and loading in export scripts #327

Merged
merged 4 commits into from
Oct 25, 2024
Merged

Split sharded Llama dataset exporting and loading in export scripts #327

merged 4 commits into from
Oct 25, 2024

Commits on Oct 25, 2024

  1. Split sharded Llama dataset exporting and loading in export scripts 

    Separate the 2 steps. We need exported irpa files for the IREE module
anyway.
    @sogartar
    sogartar committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    5ccadf2 View commit details
    Browse the repository at this point in the history

  2. Add tensor parallelism size to dataset properties and fix test

    @sogartar
    sogartar committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    e0ab7de View commit details
    Browse the repository at this point in the history

  3. Default to tensor_parallelism_size=1

    @sogartar
    sogartar committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    5365bd4 View commit details
    Browse the repository at this point in the history

  4. Improve readability

    @sogartar
    sogartar committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    1dc7897 View commit details
    Browse the repository at this point in the history