epic: dictionary-based word-breakers 🔬

[mcdurdin]

No description provided.

[keymanapp-test-bot]

User Test Results

Test specification and instructions

ERROR: user tests have not yet been defined

Test Artifacts

• Android
  • Keyman for Android apk
  • FirstVoices for Android 16.0 apk
  • FirstVoices Keyboards for Android apk
  • KeyboardHarness apk
  • Keyman for Android 16.0 apk
  • KMSample1 apk
  • KMSample2 apk
• Developer
  • Keyman Developer
  • Compiler Regression Tests
  • kmcomp.zip
• iOS
  • Keyman for iOS (simulator image)
  • FirstVoices Keyboards for iOS (simulator image)
  • TestFlight internal PR build version - 18.0.131 (0.12142.12197)
• Keyboards
  • Test Keyboards
• Web
  • KeymanWeb Test Home
• Windows
  • Keyman for Windows
  • FirstVoices Keyboards for Windows
  • Text Editor (32 bit)
  • Text Editor (64 bit)

[jahorton]

Jotting down notes of lingering ideas before this goes on pause for a while:

Once we build a model for the user-dictionary data, we may wish to aim to 'hash' the source (or similar) and cache it. Of course, the data is subject to change as the user adds new contacts, etc, but if our 'hash' can indicate that there has been no change to the source data, there's no need to rebuild the user-dictionary model.

Potential design paths:

• rig up the model compiler within its own WebView or a separate worker; have the host app compile the user-dict model, then pass that in.
  • We don't want the operation to be 'blocking'.
• build a JSON object or array that can be passed into the... compiler's thread... in order to build the user-dict model.
  • "compiler's thread": it could be the predictive-text worker, a separate worker, or even the main thread of a separate WebView. We haven't explicitly made a design-decision here yet.
  • If it's essentially a JSON-encoding of what would be a .tsv file, the JSON parse should be relatively simple and straightforward.
  • Since the model compiler is in TS, compiled down to JS... we do have to solve the issue of data transfer in one manner or other.

[jahorton]

I got to wondering if there are any "relatively simple" ways to avoid spinning up a WebView to run the model-compiler, should we decide to keep the user-dictionary compilation completely separate from the keyboard.

After a bit of searching, I found this: https://github.com/nodejs-mobile - a library for running Node-oriented JS scripts for mobile devices. That said, it'd be a new dependency.

[jahorton]

Other notable thoughts:

We should probably not associate a language code with user-dictionary data. That is, we collate the data once and use that with any language supporting predictive-text.

My original strategy (as of #11994) was to blend the models into a single, "traversable" model.

• This would require that the standard lexical model for each language implements the LexiconTraversal interface, though - which is not strictly required for all custom models.
  • We'd need an alternate strategy to support scenarios where a language-specific custom model lacks this feature.
• Thinking ahead, we'd want a similar strategy to be in place once we start doing 'learning', which would adjust a model's probability data to better suit the user's actual typing patterns.

@mcdurdin previously suggested instead doing multiple correction-searches and picking the best from among their results after applying relative weighting. This would work, though it would also require support for multiple correction searches that does not yet exist.

• They should likely use the same allotment for total execution time... likely requiring some form of load balancing.
• We'd likely need the ability to pause and resume whichever search is currently returning 'more likely' paths at the time.

[mcdurdin]

• we do have to solve the issue of data transfer in one manner or other.

Data transfer into the webview could be via local file: or http: request. This opens up a number of extensibility questions for KeymanWeb itself and how we could make the web-based experience consistent with the Keyman Android/iOS app experience.