-
Notifications
You must be signed in to change notification settings - Fork 531
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introduce Numba-based FSM utilities #272
Merged
brandonwillard
merged 3 commits into
dottxt-ai:main
from
brandonwillard:numba-fsa-implementation
Sep 29, 2023
Merged
Introduce Numba-based FSM utilities #272
brandonwillard
merged 3 commits into
dottxt-ai:main
from
brandonwillard:numba-fsa-implementation
Sep 29, 2023
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
brandonwillard
added
enhancement
optimization
Related to performance optimizations
structured generation
Linked to structured generation
labels
Sep 6, 2023
brandonwillard
force-pushed
the
numba-fsa-implementation
branch
from
September 6, 2023 22:05
23fe929
to
80a2fb1
Compare
brandonwillard
force-pushed
the
numba-fsa-implementation
branch
5 times, most recently
from
September 9, 2023 23:56
27feb9d
to
7ae64b3
Compare
Closed
brandonwillard
force-pushed
the
numba-fsa-implementation
branch
5 times, most recently
from
September 16, 2023 03:40
9f416a5
to
d79967e
Compare
brandonwillard
force-pushed
the
numba-fsa-implementation
branch
4 times, most recently
from
September 16, 2023 20:26
9a186fe
to
c7b3cc8
Compare
brandonwillard
force-pushed
the
numba-fsa-implementation
branch
from
September 17, 2023 21:43
c7b3cc8
to
6739b30
Compare
4 tasks
brandonwillard
force-pushed
the
numba-fsa-implementation
branch
from
September 23, 2023 00:34
6739b30
to
bd032c4
Compare
brandonwillard
force-pushed
the
numba-fsa-implementation
branch
from
September 23, 2023 17:24
bd032c4
to
61cf813
Compare
brandonwillard
force-pushed
the
numba-fsa-implementation
branch
2 times, most recently
from
September 26, 2023 17:10
ab84dc5
to
b4d4b2b
Compare
brandonwillard
force-pushed
the
numba-fsa-implementation
branch
2 times, most recently
from
September 27, 2023 18:57
87c97cd
to
fb37a1c
Compare
brandonwillard
force-pushed
the
numba-fsa-implementation
branch
from
September 27, 2023 19:06
fb37a1c
to
9004440
Compare
will it help to speed up the "self.regex_fsm = regex_pattern.to_fsm().reduce()" in outlines 0.0.8,i found when the set the constrain field long like maxLength=1000, it takes very long in regex_fsm construction |
Did you try with |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
enhancement
optimization
Related to performance optimizations
structured generation
Linked to structured generation
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces Numba JITed FSM utilities with 20x speed-ups over the current pure Python implementations.
It also introduces a more memory efficient "end-to-end" means of producing FSM indices. I avoided implementing it this way originally because it involves multiple iterations through a vocabulary, but, in order to address some memory-related shortcomings of the CFG indexing approaches tested in #178, this might be the better approach for now. It's a clear trade-off between processing and memory—now leaning toward processing—but, with the JIT speed-ups, it's reasonable.
Closes #226 (for now), closes #239, and should help with #192.
Regex
implementation.This was always how it was supposed to work, but our previous prototype didn't implement it. Since we're updating/replacing that prototype, it might be best to add it now.
We need to make sure that caching works exactly as expected (i.e. only once for all the index-building code).
uint
s for the states, instead ofint64
.