[WIP] Embedding Jinja2 into more syntaxes

[ROpdebee]

Here's my proof-of-concept take on #7.

Changes

• I've moved the HTML embedding out of the main Jinja syntax. As a result, Jinja.sublime-syntax can now be used as a "catch-all" syntax for anything containing Jinja2 that does not have a corresponding embedding syntax.
• I've added YAMLMacros templates to easily derive new embedding languages, along with a couple of custom macros to do this.
  • The embed-base.yaml-macros contains a base template for any embedding language, with macros to populate the necessary fields (file extensions, base scopes, name).
  • The embed.py file contains custom macros to make this easier.
  • The remaining Jinja *.sublime-syntax.yaml-macros contain a small amount of boilerplate to instantiate a template from the syntax definition of the embedding language. Importantly, the !base_syntax macro loads the embedding syntax (must be installed) and populates a couple of meta-variables used in the macros from that syntax definition.
  • Any Jinja *.sublime-syntax file is autogenerated.
• Instead of just including the main context of the base Jinja syntax, I've opted to use a lookahead to find the beginning of a Jinja2 statement/expression/comment, and only push the base Jinja syntax when this lookahead succeeds. Reasoning is that this allows us to more easily clear the existing scopes, otherwise, if the expression occurs inside of a string in the host language, everything will inherit the string scope, which may lead to issues.

TODO

• Base syntax split.
• Macros for host languages embedding Jinja
• Disabling line statements for host languages using # as the comment.
• Syntax tests.
• More host languages?

Screenshots

HTML

JSON

(The orange highlight on L5 is because of a linter, not because of the syntax)

Shell

Generic

[ROpdebee]

A potential issue came to mind: Line statements in host languages that use # as the comment sign (YAML, Python, Shell, ...) will likely break. A fix could be to omit line statements in the base template if the host language uses those types of comments, since I don't think it's likely that people enable line statements for those languages.

I should also clarify that the dependency on YAMLMacros is purely on the dev side. AFAIK, clients don't need to have this installed, we should just make sure to build each syntax before pushing new versions. Of course, clients still need to have the host syntax installed, as usual.

[UltraInstinct05]

I wasn't expecting PR's this early in this project TBH but whatever :)

As this is a significant PR, I'll need time to go through it. Hopefully, you don't mind :)
I have heard of YAMLMacros but never really used it, so I'll have to understand how it works as well. It is not listed in dependencies.json so is it a python library ?

A suggestion for naming convention for the generated syntax can be <language> (Jinja2).sublime-syntax. E.g. HTML (Jinja2).sublime-syntax, JSON (Jinja2).sublime-syntax etc.

A potential issue came to mind: Line statements in host languages that use # as the comment sign (YAML, Python, Shell, ...) will > likely break. A fix could be to omit line statements in the base template if the host language uses those types of comments, > > since I don't think it's likely that people enable line statements for those languages.

I really doubt if we need line statements at all (I am not sure how many Jinja2 users even use them). It was mentioned in the Jinja2 docs, so I had them in a very limited sense, but yes I can see how this would be problematic.

Anyways thanks for you work on this. It is really appreciated.

[ROpdebee]

I wasn't expecting PR's this early in this project

1. Well, this is quite an important package since Jinja2 support in Sublime is a bit lackluster at the moment, so I wouldn't be surprised if it picks up a lot once the PC PR is merged.
2. I think I mentioned in an issue that I had been working on my own Jinja syntax for my own needs without doing the proper research, so it makes more sense to just contribute whatever I have or had planned here. :)

As this is a significant PR, I'll need time to go through it.

Of course 👍 It is a WIP after all.

I have heard of YAMLMacros but never really used it, so I'll have to understand how it works as well. It is not listed in dependencies.json so is it a python library ?

It pretty much exposes YAML macros and offers a build system to evaluate them, focusing heavily on Sublime syntax definitions. The main idea is to create YAML files containing macros (prefixed by !) to remove boilerplate code. Building them (either through the Sublime build system or through a CLI) will evaluate the macros and replace them with transformed content, and writes the resulting YAML back to disk. Their repo has a couple of nice examples, especially the jsx one.

The main idea of the YAML macros in this PR is that we create a base template for any syntax with embedded Jinja, and create another per-syntax YAML file with the appropriate macros to transform the base template into a usable syntax. Then we build the per-syntax yaml-macros file, and out comes the sublime-syntax file with the necessary stuff set. In a sense, it isn't much different from Jinja itself in this context. I'll add some comments to the templates in a new commit.

I don't think it really is a dependency of the package, if anything, it's a dev dependency. Like I said, the idea is to generate the actual syntax and distribute those. The yaml-macros files technically don't need to be included into the package, so there's not really a dependency.

A suggestion for naming convention for the generated syntax can be <language> (Jinja2).sublime-syntax. E.g. HTML (Jinja2).sublime-syntax, JSON (Jinja2).sublime-syntax etc.

Definitely a possibility.

I really doubt if we need line statements at all (I am not sure how many Jinja2 users even use them). It was mentioned in the Jinja2 docs, so I had them in a very limited sense, but yes I can see how this would be problematic.

If it's mentioned in the docs, it's probably a good idea to include them for completion's sake. I can probably disable them on a per-syntax basis with some macros. I personally have no experience with them either though. I've seen a TODO in the syntax regarding the negative lookbehinds to make sure they're on the start of a line, I'm wondering whether this can be resolved using a start-of-line match (something like ^\s*(##) perhaps?).

[jdoss]

Is this PR/idea dead? I'd like to use this in a YAML based syntax plugin I am working on.

[UltraInstinct05]

No the PR/idea is not dead (Though I apologies for taking a long time to get to this). The PR was made when ST4 was still in beta period and ST3 was being used. In ST3, the only way (AFAIK) to reuse a syntax was to use yaml-macros. But now, we have inheritance as a feature in ST4. I have now gotten around to adding this to PC and restricting it to ST4 (Since ST4 has had a couple of stable releases & I don't want to go down the road for maintaining it for ST3).

I am currently working on a v1.2.0 to clean up a bunch of stuff/bugs in the syntax & make the test suite more robust. Once done, I'll be working on a v2.0.0 to see if we can use Inheritance to combine YAML & Jinja (Since I want to avoid the yaml-macros dependency). If that doesn't work out, we can always use this.

[deathaxe]

Bash is not yet included in #22, but the overall strategy of that PR is what, we are planning to move forward with.