Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Distributed dataloader #14

Merged
merged 19 commits into from
Mar 28, 2024
Merged

Distributed dataloader #14

merged 19 commits into from
Mar 28, 2024

Commits on Mar 18, 2024

  1. add self-distributing dataset for multi gpu training

    @philippguevorguian
    philippguevorguian committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    e9f4095 View commit details
    Browse the repository at this point in the history

  2. set test dataloader num workers to 1

    @philippguevorguian
    philippguevorguian committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    77827f4 View commit details
    Browse the repository at this point in the history

  3. change file state logic for new dataloader

    @philippguevorguian
    philippguevorguian committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    8184f22 View commit details
    Browse the repository at this point in the history

  4. add test status

    @philippguevorguian
    philippguevorguian committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    79ad1cf View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2024

  1. fix small opt configuration

    @philippguevorguian
    philippguevorguian committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    b6d7dd7 View commit details
    Browse the repository at this point in the history

  2. fix generator line skips between file iterations

    @philippguevorguian
    philippguevorguian committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    5f0d2f6 View commit details
    Browse the repository at this point in the history

  3. correct chemlactica imports

    @philippguevorguian
    philippguevorguian committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    7c9cc13 View commit details
    Browse the repository at this point in the history

  4. add test for line by line distributed dataset

    @philippguevorguian
    philippguevorguian committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    82ee745 View commit details
    Browse the repository at this point in the history

  5. add test status

    @philippguevorguian
    philippguevorguian committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    14735f8 View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2024

  1. remove comments

    @philippguevorguian
    philippguevorguian committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    6675334 View commit details
    Browse the repository at this point in the history

  2. add test status

    @philippguevorguian
    philippguevorguian committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    771a801 View commit details
    Browse the repository at this point in the history

  3. merge main changes

    @philippguevorguian
    philippguevorguian committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    8e859e1 View commit details
    Browse the repository at this point in the history

Commits on Mar 26, 2024

  1. merge main changes

    @philippguevorguian
    philippguevorguian committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    398bcf9 View commit details
    Browse the repository at this point in the history

Commits on Mar 27, 2024

  1. merge num eval sft changes

    @philippguevorguian
    philippguevorguian committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    c498fec View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2024

  1. Merge remote-tracking branch 'origin/integrate_gemma' into distribute… 

    …d_dataloader
    @philippguevorguian
    philippguevorguian committed Mar 28, 2024
    Configuration menu
    Copy the full SHA
    e01ebe2 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'integrate_gemma' into distributed_dataloader

    @philippguevorguian
    philippguevorguian committed Mar 28, 2024
    Configuration menu
    Copy the full SHA
    34a469c View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into distributed_dataloader

    @philippguevorguian
    philippguevorguian committed Mar 28, 2024
    Configuration menu
    Copy the full SHA
    e3b3214 View commit details
    Browse the repository at this point in the history

  4. update env file

    @philippguevorguian
    philippguevorguian committed Mar 28, 2024
    Configuration menu
    Copy the full SHA
    e826628 View commit details
    Browse the repository at this point in the history

  5. add test status

    @philippguevorguian
    philippguevorguian committed Mar 28, 2024
    Configuration menu
    Copy the full SHA
    fc34337 View commit details
    Browse the repository at this point in the history