Custom Preprocessor breaks brightScript/brighterScript parsing

[justinMBullard]

In my project we have a custom preprocessor. It has some of its own custom statements that act similar to enums, but cause parsing errors for both brightscript and brigherscript files. Eventually it is probably best to migrate to brigherScripts enums. However, until that transition can be made, I would like to be able to create a plug-in that would bridge compatibility between our custom preprocessor and brighterscript.

I am working on a compiler plug-in using beforeFileParse events and modifying the source.source. This is working but it also changes the column information. I know it is causing issues when reporting parsing errors. The underline errors in VSCode match up with the substituted code not the code from the file. I need to research to see how this affects rokucommunity/bslint.

I went the route of beforeFileParse because I didn't see any other way to affect the tokens before parsing occurs. Is there another way?

[TwitchBronBron]

Could you provide a few syntax examples of the custom statements?

For things like "I don't recognize that function", we can solve lots of those types of problems with later plugin hooks.

For custom code that is invalid syntax, I don't think there's another way to handle this at the moment. For example, using handlebars syntax like this:

print {{SOME_CONST}}

wouldn't work because the {{ and }} get flagged as syntax errors in the parser. You'd definitely need to remove those before the parser starts working.

But values in strings could definitely be solved after the parser phase because they're valid syntax to start with.

print "{{SOME_CONST}}"

I've toyed with the idea of hooks inside the tokenizer (lexer) and parser so you could do this type of thing, but haven't gotten around to it yet.

Another option is, if your replaced code is SMALLER than the original custom statements, you could pad with space characters to ensure the inserted code is the same size as the original. But that doesn't solve when the replacement text is longer than the original

[justinMBullard]

We have some substitutions that we share between BRS files and XML files. We store them in json files and will do substitutions like {@resource ui.button.padding.left}. This will look up a the ui.json file in the Resource directory and then substitute the value into the BRS file.

I have looked into adding hooks into the tokenizer as well. I have a branch that has added

afterFileLexed?: (file:BrsFile, tokens: Token[])=> void;

           //if the preprocessor generated tokens, use them.
            let tokens = preprocessor.processedTokens.length > 0 ? preprocessor.processedTokens : lexer.tokens;

            this.program.plugins.emit('afterFileLexed', this, tokens);

            this.program.logger.time(LogLevel.debug, ['parser.parse', chalk.green(this.pathAbsolute)], () => {

I have not done any testing on this or work on this as I went the beforeFileParse path. I know the call works, but I haven't tried walking or modifying the tokens in the array. @TwitchBronBron I am more than happy to share what I have done.

[elsassph]

@justinMBullard I guess it's not an option to support "@resource ui.button.padding.left" as syntax for your preprocessor? e.g. using the double-quote instead of the curly bracket to mark a token.

[justinMBullard]

@elsassph The code base is pretty big. Anything that I do I really need to support in a plug-in. It would be very complicated to make changes to the code base and then verify those changes.

[elsassph]

That said, a syntax for "DSLs" would be nice @TwitchBronBron. Whatever is captured between certain characters is kept as-is in a special token, meant to be processed by a plugin post tokenzation or after transpilation. Maybe the mustache double-curly braces for instance.