Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

epic: dictionary-based word-breakers 🔬 #12142

Draft
wants to merge 38 commits into
base: master
Choose a base branch
from
Draft

epic: dictionary-based word-breakers 🔬 #12142

wants to merge 38 commits into from
+706 −2

Commits on Aug 9, 2024

  1. feat(common/models/wordbreakers): starting on dictionary-based wordbr… 

    …eaking
    @jahorton
    jahorton committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    425a0a0 View commit details
    Browse the repository at this point in the history

  2. feat(common/models/wordbreakers): actual first-pass implementation 

    Only wordbreaks anything AFTER the last space / ZWNJ.  Doesn't bother with anything before it.
    @jahorton
    jahorton committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    7b456c1 View commit details
    Browse the repository at this point in the history

  3. feat(common/models/wordbreakers): pass 2 - should now tokenize full c… 

    …ontext
    @jahorton
    jahorton committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    2da7b7c View commit details
    Browse the repository at this point in the history

  4. feat(common/models/wordbreakers): dict-breaker helper unit tests (BMP)

    @jahorton
    jahorton committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    5568167 View commit details
    Browse the repository at this point in the history

  5. feat(common/models/wordbreakers): dict-breaker helper unit tests (non… 

    …-BMP)
    @jahorton
    jahorton committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    d40a09d View commit details
    Browse the repository at this point in the history

  6. feat(common/models/wordbreakers): dict-breaker unit tests with simple… 

    … mocked-fixture
    @jahorton
    jahorton committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    10f91e0 View commit details
    Browse the repository at this point in the history

  7. fix(common/models/wordbreakers): blocks empty span output on empty co… 

    …ntext
    @jahorton
    jahorton committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    1d07e9f View commit details
    Browse the repository at this point in the history

  8. fix(common/models): update re base branch change

    @jahorton
    jahorton committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    3e82434 View commit details
    Browse the repository at this point in the history

  9. docs(common/models): updates dict-breaker comments

    @jahorton
    jahorton committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    56e4052 View commit details
    Browse the repository at this point in the history

  10. change(common/models/wordbreakers): allows wordbreaker unit tests to … 

    …use Tries
    @jahorton
    jahorton committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    66c156e View commit details
    Browse the repository at this point in the history

  11. feat(common/models/wordbreakers): baby's first khmer wordbreaking test

    @jahorton
    jahorton committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    3181a4c View commit details
    Browse the repository at this point in the history

  12. chore(common/models/wordbreakers): comment tweak

    @jahorton
    jahorton committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    07f6747 View commit details
    Browse the repository at this point in the history

  13. feat(common/models/wordbreakers): rejoins adjacent single-point spans… 

    … from unmatched chars in path
    @jahorton
    jahorton committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    8c5e54a View commit details
    Browse the repository at this point in the history

  14. fix(common/models/wordbreakers): handling of penalty transitions

    @jahorton
    jahorton committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    766992d View commit details
    Browse the repository at this point in the history

  15. change(common/models): use spread operator to split on codepoints 

    Addresses #10973 (comment)
    @jahorton
    jahorton committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    99e1f4b View commit details
    Browse the repository at this point in the history

  16. chore(common/models/wordbreakers): Merge branch 'feat/common/models/w… 

    …ordbreakers/dict-breaker-start' into change/common/models/wordbreakers/unit-test-trie-access
    @jahorton
    jahorton committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    51b62e6 View commit details
    Browse the repository at this point in the history

  17. chore(common/models/wordbreakers): Merge base branch into feat/common… 

    …/models/wordbreakers/fuse-dict-unmatched-chars
    @jahorton
    jahorton committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    295c70b View commit details
    Browse the repository at this point in the history

  18. chore(common/models): drops unit tests for replaced func

    @jahorton
    jahorton committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    8ae6a62 View commit details
    Browse the repository at this point in the history

  19. chore(common/models/wordbreakers): Merge branch 'feat/common/models/w… 

    …ordbreakers/dict-breaker-start' into change/common/models/wordbreakers/unit-test-trie-access
    @jahorton
    jahorton committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    258763b View commit details
    Browse the repository at this point in the history

  20. chore(common/models/wordbreakers): Merge base branch fixes into feat/… 

    …common/models/wordbreakers/fuse-dict-unmatched-chars
    @jahorton
    jahorton committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    8942883 View commit details
    Browse the repository at this point in the history

  21. chore: establish dictionary breakers epic

    @mcdurdin
    mcdurdin committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    001126b View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. Merge pull request #12253 from keymanapp/chore/merge-master-into-dict… 

    …-breaker

chore: merge master into dict-breaker
    @mcdurdin
    mcdurdin authored Aug 22, 2024
    Configuration menu
    Copy the full SHA
    1b3dfda View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. chore(common/models/wordbreakers): Merge branch 'epic/dict-breaker' i… 

    …nto feat/common/models/wordbreakers/dict-breaker-start
    @jahorton
    jahorton committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    fcbffc6 View commit details
    Browse the repository at this point in the history

  2. chore(common/models/wordbreakers): Merge branch 'feat/common/models/w… 

    …ordbreakers/dict-breaker-start' into change/common/models/wordbreakers/unit-test-trie-access
    @jahorton
    jahorton committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    d38ed7c View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2024

  1. chore(common/models): Apply suggestions from code review 

    Co-authored-by: Marc Durdin <[email protected]>
    @jahorton @mcdurdin
    jahorton and mcdurdin authored Aug 26, 2024
    Configuration menu
    Copy the full SHA
    c33c10d View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2024

  1. Merge pull request #12139 from keymanapp/feat/common/models/wordbreak… 

    …ers/dict-breaker-start

feat(common/models/wordbreakers): begin development of dictionary-based wordbreaking algorithm 🔬
    @jahorton
    jahorton authored Aug 27, 2024
    Configuration menu
    Copy the full SHA
    9ca5d0a View commit details
    Browse the repository at this point in the history

  2. Merge pull request #12140 from keymanapp/change/common/models/wordbre… 

    …akers/unit-test-trie-access

change(common/models/wordbreakers): allow wordbreaker tests to access TrieModel implementation 🔬
    @jahorton
    jahorton authored Aug 27, 2024
    Configuration menu
    Copy the full SHA
    b0aaf1e View commit details
    Browse the repository at this point in the history

  3. Merge pull request #12141 from keymanapp/feat/common/models/wordbreak… 

    …ers/fuse-dict-unmatched-chars

feat(common/models/wordbreakers): fuse adjacent unmatched characters when dictionary-breaking 🔬
    @jahorton
    jahorton authored Aug 27, 2024
    Configuration menu
    Copy the full SHA
    2e7e9d7 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2024

  1. chore: Merge branch 'epic/dict-breaker' into chore/merge-master-into-… 

    …dict-breaker
    @jahorton
    jahorton committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    1df99e5 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. chore: move dict.ts into src

    @mcdurdin
    mcdurdin committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    cf43c0e View commit details
    Browse the repository at this point in the history

  2. Merge pull request #12317 from keymanapp/chore/merge-master-into-dict… 

    …-breaker

chore: merge master into dict-breaker 🔬
    @mcdurdin
    mcdurdin authored Sep 10, 2024
    Configuration menu
    Copy the full SHA
    dad7a6c View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2024

  1. chore: Merge branch 'epic/dict-breaker' into chore/merge-master-into-… 

    …dict-breaker
    @mcdurdin
    mcdurdin committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    c2e80b4 View commit details
    Browse the repository at this point in the history

  2. chore: fixup dependency path

    @mcdurdin
    mcdurdin committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    c4991a7 View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2024

  1. Merge pull request #12411 from keymanapp/chore/merge-master-into-dict… 

    …-breaker

chore: merge master into dict-breaker 🔬
    @mcdurdin
    mcdurdin authored Sep 14, 2024
    Configuration menu
    Copy the full SHA
    367d32e View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2024

  1. Merge branch 'epic/dict-breaker' into chore/merge-master-into-dict-br… 

    …eaker
    @mcdurdin
    mcdurdin authored Oct 10, 2024
    Configuration menu
    Copy the full SHA
    39430bf View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2024

  1. Merge pull request #12530 from keymanapp/chore/merge-master-into-dict… 

    …-breaker

chore: merge master into dict-breaker 🔬
    @mcdurdin
    mcdurdin authored Oct 11, 2024
    Configuration menu
    Copy the full SHA
    04cab9d View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2024

  1. Merge branch 'epic/dict-breaker' into chore/merge-master-into-dict-br… 

    …eaker
    @mcdurdin
    mcdurdin authored Oct 25, 2024
    Configuration menu
    Copy the full SHA
    0db0b26 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #12575 from keymanapp/chore/merge-master-into-dict… 

    …-breaker

chore: merge master into dict-breaker 🔬
    @mcdurdin
    mcdurdin authored Oct 25, 2024
    Configuration menu
    Copy the full SHA
    c3f9917 View commit details
    Browse the repository at this point in the history