Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Less Sampler Allocations #735

Merged
merged 5 commits into from
May 14, 2024
Merged

Less Sampler Allocations #735

merged 5 commits into from
May 14, 2024

Commits on May 12, 2024

  1. - Reduced the number of allocations required for sampling, by allowin… 

    …g passing in a temporary blob of memory to work in. Using this new method in `LLamaContext` and `BaseSamplingPipeline`.

 - Using `Guidance` method in guidance example, instead of low level one working directly on logits.
 - Fixed `Guidance` method passing incorrectly sized span.
    @martindevans
    martindevans committed May 12, 2024
    Configuration menu
    Copy the full SHA
    3e918c7 View commit details
    Browse the repository at this point in the history

  2. Spelling

    @martindevans
    martindevans committed May 12, 2024
    Configuration menu
    Copy the full SHA
    9fbc045 View commit details
    Browse the repository at this point in the history

  3. Renamed _temporarySampling to _samplingBuffer

    @martindevans
    martindevans committed May 12, 2024
    Configuration menu
    Copy the full SHA
    bf33a19 View commit details
    Browse the repository at this point in the history

  4. Hidden LLamaTokenData* and exposed span instead

    @martindevans
    martindevans committed May 12, 2024
    Configuration menu
    Copy the full SHA
    60ddd44 View commit details
    Browse the repository at this point in the history

Commits on May 13, 2024

  1. Changed based on review feedback.

    @martindevans
    martindevans committed May 13, 2024
    Configuration menu
    Copy the full SHA
    b80f043 View commit details
    Browse the repository at this point in the history