JS: Backport³ and more additions & fixes

[Willy-JL]

What's new

• backport (here) of backported missing things (there) of great backport & rework (over yonder) xD
• i separated into individual commits that make sense on their own, might be easier to look there instead of whole diff
• original authors/contributors credited with co-author where relevant
• badusb:
  • numpad keys support
  • keyboard layout support
  • altPrint() and altPrintln() support
  • quit() function to unlock usb profile (original intent was to alternate badusb profile and mass storage profile, we have this in momentum and other cfw as usbdisk module, but outside scope of this pr, can backport this module later on if you're interested)
• serial
  • readAny() to avoid starving read loops with many small reads or hinder usability with few large reads
  • end() to release serial handle allowing new initialization in same script, maybe for gpio usage or different serial port or settings
  • expansion service is automatically disabled to avoid conflict
• storage
  • added simple example script we had previously, ported to new remade storage module
• gui:
  • text_input:
    • fixed null ptr crash when user did not pass max len prop. logic of props is from what i can see, always configure view for minimum usable state, props should configure things on top as optional
    • defaultText and defaultTextClear props
  • byte_input
  • file_picker
    • there is no file picker based on View object, so this is only gui/* module that is not based on view_factory
    • instead it uses usual dialogs app file picker in sync manner
  • viewDispatcher.currentView to allow users to have extra logic in navigation callbacks for example
  • updated gui example script with showcase of all additions above
• globals:
  • toString() works correctly on negative numbers, toString(-42) used to give "0"
  • parseInt()
  • toUpperCase() and toLowerCase()
  • example for string functions
  • __filepath and __dirpath to allow relative path contextualization, for example for load() or storage.open()
• example interactive.js repl script, allows running js in limited capacity without editing files
• many additions to typedefs that were missing or incomplete, still not everything is documented i believe but getting closer

Needs feedback:

• toString(), toUpperCase(), toLowerCase() could all be moved to respective classes i think, for strings there is getprop_builtin_string() in mjs source, can add there, for number there is no such function but may be possible to add another if branch in getprop_builtin() for number type
• __dirpath and __filepath are set when launching script in global scope, but means that when executing scripts with load(), these have values of original script, since scope lookup is recursive (see mjs_find_scope() and its usages). i think better behavior would be load() gives child script its own correct __dirpath and __filepath but there are some issues:
  • load() accepts a custom scope parameter. with custom scope, assignment in child script is confined to custom scope, lookup is recursive to parent scopes. with no custom scope, code is executed in current scope, so everything is shared like code was inlined
  • for custom scope, we can get custom scope object passed by user and inject __filepath and __dirpath into it, easy
  • for no custom scope, i dont see any good solution: one way is saving current __dirpath and __filepath values from scope, changing them, running load(), then resetting to previous values. might work fine in most cases but if child script runs load() of its own and uses callbacks or things like this, child2 might run a function defined in child1 and then its screwed up. this is a rare edge case of course, if no function callback is passed from child1 to child2 then no code from child1 could run while child2, and second instance of mjs_load() would restore child1's __dirpath and __filepath correctly

Verification

• unit tests
• example scripts

Checklist (For Reviewer)

• PR has description of feature/bug or link to Confluence/Jira task
• Description contains actions to verify feature/bugfix
• I've built this code, uploaded it to the device and verified feature/bugfix

[portasynthinca3]

I'm curious whether it's at all possible to integrate the file picker with the rest of the GUI system as an ordinary view and make its API asynchronous. Gonna look into it.

[Willy-JL]

@portasynthinca3 @skotopes with ofw 1.1 close to release, and the significant changes to battery driver, i want to release momentum 008 as close after 1.1 as possible, and this for me includes atleast agreeing on user-facing changes of this PR: we of course already had these things that this PR adds, and dont want users to fix scripts on this release, and then change the same things again in next release.

hasProperty() is gone, __dirname and __filename names are fixed, toUpperCase() and toLowerCase() are moved to string class. this is most of review comments.

although i did not get any feedback on moving toString() to number class, i got it working, so please let me know if you agree with this or not.

behavior of __dirname and __filename in load() can be fixed later on.

for filepicker, i can label in our changelog that its changed but may change again depending on this.

does this sound fine?

[skotopes]

@Willy-JL I hear you. We can postpone release till this series of PRs will be ready to merge.
Also we are open to discussion if we any changes needed on our side. I've explained to @portasynthinca3 where we aiming and she we'll be discussing changes with you guys so we all be able to schedule development

[Willy-JL]

i just wanted some feedback on whether these user-facing changes would be ok, no need to rush this pr or delay release, i did not mean to disrupt plans 😄

but besides this, from what i understand this is basically ready, only missing feedback on whether moving of toString() to number class is ok, and whether there is a way to move file picker to new gui system

lets see what porta thinks :D

[portasynthinca3]

To be honest, did not mean to delay the release either.

[portasynthinca3]

@Willy-JL, here's our statement on the whole JS API / SDK / versioning mess.

Goals

In short

• We want to maintain a base JS API that is consistent across all firmware flavors.
• We want JS app developers to be able to leverage additional functionality found in custom firmware editions.
• We want users not to think about cross-FW compatibility of the scripts that they find online too much.

The user's perspective

• A JS app that works on OFW should also work on a CFW.
• A JS app that works on a CFW and leverages its extended features may or may not work on OFW.

The JS developer's perspective

• A JS app that does not leverage any additional CFW functionality is guaranteed to work across all firmware editions.
• A JS app that requires additional CFW functionality will not work on OFW.
• One can write JS apps that can make use of but do not require additional CFW functionality.

The CFW maintainer's perspective

• CFWs are free to introduce additional JS APIs.
• CFWs are discouraged from breaking the base JS API found in OFW.
• CFWs are encouraged to backport any of their extended features into OFW, assuming they are compatible with OFW's vision.

How do we achieve that?

Like I already said, I propose that we introduce a major-minor semantic versioning system like the one that we already have for native apps. Since we have adopted TypeScript, I propose that we consult the following standard to determine whether a change is a major or a minor one: https://www.semver-ts.org/.

Version compatibility could be checked by the developer using a combination of the following functions:

• checkSdkCompatibility(expectedMajor, expectedMinor): checks if the JS API version is compatible with the one that the script expects (i.e. expectedMajor == actualMajor && expectedMinor >= actualMinor), and if it doesn't, warns the user and asks whether they want to continue executing the script anyways. This is the simplest of the three checks, and it will be automatically inserted in the beginning of the output file unless the developer explicitly asks our JS "toolchain" (SDK) not to do so. By asserting compatibility by default, we guarantee that the user receives a clear warning if their firmware is incompatible with the script instead of cryptic error messages like calling non-callable.
• isSdkCompatible(expectedMajor, expectedMinor): performs the same check, but just returns the result as a boolean.
• sdkCompatibilityStatus(expectedMajor, expectedMinor): performs the same check, but returns a more detailed description ("compatible", "firmwareTooOld", "firmwareTooNew")

To check for additional features found in CFWs, I propose that we introduce the following symbols:

• checkSdkVendor(expectedVendor): checks if the JS API vendor matches the one that the script expects, and if it doesn't, warns the user and asks whether they want to continue executing the script anyways. This is the simplest of the two checks for cases where an app requires a feature found in a CFW. This check will not be automatically inserted by our SDK (unlike checkSdkCompatibility).
• sdkVendor: a string containing the SDK vendor (e.g. "flipperdevices" or "momentum"). A JS app that is able to work without additional features on a best-effort basis may query this global constant to find out whether it can use those features.

Let us know what you think.

[Willy-JL]

This sounds like a great plan! There's only one thing i see as suboptimal:

Many features added by cfw are shared between other cfw pretty quickly, checkSdkVendor() would allow the developer to ask for the feature set of a certain firmware, and a different firmware could detect this and return a positive result even if it's not the firmware that the script asked for, rather if it supports the features present in said firmware. For example, unleashed includes most, but not all, extra js features that there are in momentum. A script requesting unleashed vendor features would therefore work on momentum, which is a superset of their features, but not viceversa.
That's good to have of course, but the alternative of letting the user check based on name might mean they do a simple if (sdkVendor === "unleashed") to use a certain unleashed feature, which even though its present in momentum too, would return false and not be allowed to use such features.
Perhaps something similar to what there is for versioning could work: checkSdkVendor() is a blocking call and will warn the user and possibly quit the script, while isSdkVendorCompatible() could perform the same check and return a boolean.

[portasynthinca3]

That's a valid point. However, I'm afraid that CFWs falsely presenting themselves as other CFWs, although done in good faith, will quickly devolve into a mess similar to what User-Agent is today.

I propose that we declare OFW the baseline for the feature set, and allow CFWs to introduce and share named features.

Suppose that you as the maintainer of Momentum decide to add a regex validator prop to the text input view. You shall, under this social contract, introduce a named feature (e.g. "text-input-regex") that developers may check compatibility with using doesSdkSupport("text-input-regex"). They may also use checkSdkFeatures(["text-input-regex", "something-else"]) to display a warning screen to the user if the API does not support the required features.

Now, to keep firmware image sizes under control, we need a framework for removing these strings from the code. In other words, we need a mechanism for recognizing a feature as a baseline feature after it has been merged into OFW. I propose that we recognize a backported feature as a baseline feature one SDK major version after it has been merged. Consider the following chain of events as an example:

1. Both OFW and Momentum are at JS SDK version 1.0.
2. Momentum introduces a feature called "text-input-regex" and bumps its JS SDK version to 1.1.
3. The feature is backported into OFW, bumping its JS SDK version to 1.1 as well.
4. Both OFW and Momentum continue to report true for doesSdkSupport("text-input-regex") until their JS SDK versions are bumped up to 2.0
5. Starting with JS SDK 2.0, doesSdkSupport("text-input-regex") starts reporting false and checkSdkFeatures(["text-input-regex"]) starts warning the user that a feature is unsupported. I assert that the negative effect from these incorrect results is alleviated thanks to SDK version checks, which will fail if the script expects v1.x but is actually running on 2.x.

This leaves the question of vendoring unanswered. Assuming that the system for user-agent-line-creep prevention that I outlined is in place, it would make no sense to keep the sdkVendor and checkSdkVendor symbols. If a script wants to query this data for purely informational purposes, it would do so using a new require("flipper").firmwareEdition API, again, for purely informative purposes, not to be relied on for querying the existence or absence of features.

[portasynthinca3]

@Willy-JL, @xMasterX and I came to a conclusion on this outside of GitHub. I'll be implementing this shortly.