Apply the standard rule mechanism also to text nodes

[martincizek]

There might be strong reasons for creating rules also for text nodes. This simple patch makes it possible. We believe it even makes the current code more consistent, as we can rely on the node name "#text".

Backround:
Our use case is related to text escaping. As the docs admit, it is quite simplistic and adds unnecessary backslashes. Unfortunately any simple GFM escaping also necessarily corrupts data. We aim at nearly-lossless conversions, so we have developed GfmEscape to address context-dependent escaping.

I admit that full escaping complexity might be overkill for Turndown.
But Turndown is still a fantastic framework for user-developed conversions, so let's just provide the desired hook. :)

Thank you!

[domchristie]

Hey @martincizek!
Thanks for all your work and your detailed pull requests 💫

I really like how this cleans up the process function. However I'm not sure it's the right solution yet. As you've figured out, escaping in Turndown is simple, but naive. It works for most cases, however it's often customised. I'd like a offer an improved system for customising escaping, perhaps along the lines of addEscape, removeEscape (custom escaping also discussed in #242 (comment)). Is this something you'd be interested in helping out on?

[domchristie]

😍

[domchristie]

Slight concern here is that it requires a fair bit of knowledge to add to the behaviour i.e. a developer has to remember to:

• pass through unchanged code content
• iterate over the existing escapes and performs the replacements
• trim the value

[martincizek]

Hey @domchristie, thank you for your reply! As I went through all the Turndown's code over time (again, nice job!), I admit another approach would be better - introducing textReplacement (similarly to blankReplacement etc.).

We have done it, please check this commit, we just somehow forgot to make a new PR and deprecate this PR. :) Is this the way to go?

Regarding relation to escaping:

• I believe textReplacement is still a valid extension point even if there were totally customizable escaping
• Even though we can now override the escape() method, it is not called for isCode nodes and the node and options arguments are not passed. All these things are necessary for implementing for comprehensive escaping.

[martincizek]

Regarding help with the escaping subsystem:

• Currently, iterative escaping is used. I.e. all text is escaped according to first replacement, the result is escaped with the second replacement, etc.
• Iterative escaping can only work just in the simplest cases, that's why we created the UnionReplacer project. The README also describes its Turndown-related background and issues with different escaping approaches.

So I believe that Turndown would need something like UnionReplacer to do the custom escaping right, efficiently and user friendly (without need of placeholdering magic).

The mentioned project GfmEscape is actually very thoroughly configured UnionReplacer with comprehensive replacements, so this one is not appropriate to integrate.

But UnionReplacer seems to be a perfect match for configurable escaping (actually it is designed for it :-)). I know Turndown does not use any libs, but this one:

1. was actually designed to be used with tools like Turndown
2. it is something like 100 lines (but extreme effort was put into making it this small)
3. Our performance tests suggested that the only eventual performance footprint is related to increased code amount (those 100 lines). It becomes more efficient than iterative escaping for around 5 replacements depending on the JS engine.

Is integrating UnionReplacer dependency an option when making the configurable escaping? If it is an option, then we can help. :)

If it were done this way, the comprehensive GfmEscape rules can be used as some sort of Turndown plugin then.

[martincizek]

Closing this PR in favour of #339, which provides related hooks in a way more consistent with the rest of Turndown.