Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[sharktank] Evaluation - Add Perplexity test #233

Merged
merged 38 commits into from
Oct 16, 2024
Merged

[sharktank] Evaluation - Add Perplexity test #233

merged 38 commits into from
Oct 16, 2024

Commits on Sep 27, 2024

  1. Add 'datasets' package to load golden dataset

    @archana-ramalingam
    archana-ramalingam committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    aee0d58 View commit details
    Browse the repository at this point in the history

  2. Isolate padding function in tokenizer

    @archana-ramalingam
    archana-ramalingam committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    1ee8594 View commit details
    Browse the repository at this point in the history

  3. Add utility function to load/run LLMs for evaluation pipeline

    @archana-ramalingam
    archana-ramalingam committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    0103293 View commit details
    Browse the repository at this point in the history

  4. Add perplexity test

    @archana-ramalingam
    archana-ramalingam committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    1dfdbc6 View commit details
    Browse the repository at this point in the history

  5. Cleanup

    @archana-ramalingam
    archana-ramalingam committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    14050f5 View commit details
    Browse the repository at this point in the history

  6. delete file

    @archana-ramalingam
    archana-ramalingam committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    b7c75f3 View commit details
    Browse the repository at this point in the history

  7. Add perplexity test

    @archana-ramalingam
    archana-ramalingam committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    a44a8a2 View commit details
    Browse the repository at this point in the history

  8. Fix dataset loading

    @archana-ramalingam
    archana-ramalingam committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    8034432 View commit details
    Browse the repository at this point in the history

  9. Update page_cache_size

    @archana-ramalingam
    archana-ramalingam committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    a26b17d View commit details
    Browse the repository at this point in the history

  10. add run_perplexity and prompts

    @archana-ramalingam
    archana-ramalingam committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    cd079a7 View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2024

  1. Merge branch 'main' into perplexity-test

    @archana-ramalingam
    archana-ramalingam authored Sep 28, 2024
    Configuration menu
    Copy the full SHA
    df84163 View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. Shift logits and change activation dtype

    @archana-ramalingam
    archana-ramalingam committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    3e0871e View commit details
    Browse the repository at this point in the history

  2. Add Grok model

    @archana-ramalingam
    archana-ramalingam committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    4a74107 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. Remove decode and run prefill on every turn

    @archana-ramalingam
    archana-ramalingam committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    64e812d View commit details
    Browse the repository at this point in the history

  2. Change activation dtype to enable quantized models

    @archana-ramalingam
    archana-ramalingam committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    29e6031 View commit details
    Browse the repository at this point in the history

  3. Add timing wrapper

    @archana-ramalingam
    archana-ramalingam committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    9c168f3 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. Add instructions to run evaluation-perplexity

    @archana-ramalingam
    archana-ramalingam committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    38590bb View commit details
    Browse the repository at this point in the history

  2. Add prompts text file

    @archana-ramalingam
    archana-ramalingam committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    2bf8739 View commit details
    Browse the repository at this point in the history

  3. Add logging + cleanup

    @archana-ramalingam
    archana-ramalingam committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    70f6ba5 View commit details
    Browse the repository at this point in the history

  4. Add CI perplexity test

    @archana-ramalingam
    archana-ramalingam committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    848da59 View commit details
    Browse the repository at this point in the history

  5. Update prompt file path

    @archana-ramalingam
    archana-ramalingam committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    7e49580 View commit details
    Browse the repository at this point in the history

  6. Remove unit tests for nightly

    @archana-ramalingam
    archana-ramalingam committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    ec6968f View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. Add relative path + push attention_mask to device

    @archana-ramalingam
    archana-ramalingam committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    f7667ec View commit details
    Browse the repository at this point in the history

  2. Remove debug changes

    @archana-ramalingam
    archana-ramalingam authored Oct 3, 2024
    Configuration menu
    Copy the full SHA
    7054141 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into perplexity-test

    @archana-ramalingam
    archana-ramalingam authored Oct 3, 2024
    Configuration menu
    Copy the full SHA
    70a7b10 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. Update dtype to F32 for compatibility across torch versions

    @archana-ramalingam
    archana-ramalingam committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    134c77f View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2024

  1. Merge branch 'main' into perplexity-test

    @archana-ramalingam
    archana-ramalingam authored Oct 7, 2024
    Configuration menu
    Copy the full SHA
    27f4e15 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2024

  1. Add decode

    @archana-ramalingam
    archana-ramalingam committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    8a0a081 View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2024

  1. Fix padding logits

    @archana-ramalingam
    archana-ramalingam committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    e47fe4a View commit details
    Browse the repository at this point in the history

  2. Add local model path

    @archana-ramalingam
    archana-ramalingam committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    b15c06d View commit details
    Browse the repository at this point in the history

  3. Add CI test for evaluation

    @archana-ramalingam
    archana-ramalingam committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    6da2b38 View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2024

  1. Add perplexity calculated from prefill logits only

    @archana-ramalingam
    archana-ramalingam committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    0afe63f View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2024

  1. Merge branch 'main' into perplexity-test

    @archana-ramalingam
    archana-ramalingam authored Oct 16, 2024
    Configuration menu
    Copy the full SHA
    e4ccb10 View commit details
    Browse the repository at this point in the history

  2. Add CI tests for perplexity

    @archana-ramalingam
    archana-ramalingam committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    478f1a1 View commit details
    Browse the repository at this point in the history

  3. Clean up

    @archana-ramalingam
    archana-ramalingam committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    96458a8 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'perplexity-test' of https://github.com/nod-ai/sharktank … 

    …into perplexity-test
    @archana-ramalingam
    archana-ramalingam committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    b4e3635 View commit details
    Browse the repository at this point in the history

  5. Clean up

    @archana-ramalingam
    archana-ramalingam committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    4af53c3 View commit details
    Browse the repository at this point in the history

  6. Update argument

    @archana-ramalingam
    archana-ramalingam committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    e2eb98c View commit details
    Browse the repository at this point in the history