Parametric snippets [MEP]

[Witiko]

Rationale

Themes and snippets in the Markdown package for TeX are like modules and functions in conventional programming languages. Users can import snippets from themes and use them to change how YAML and Markdown content is shown on page. For an introduction to themes and snippets, see my article titled Markdown 2.10.0: LaTeX Themes & Snippets, Two Flavors of Comments, and LuaMetaTeX published in TUGboat 42:2. For more information about themes and snippets, see also my work-in-progress article titled Markdown Themes in Practice to appear in TUG 2024 proceedings and in TUGboat.

Unlike functions in conventional programming languages, snippets are zero-arity, i.e. they accept no parameters. The original rationale behind this design decision was that parameters would make snippets more difficult to use for authors and that authors would parametrize snippets indirectly using YAML metadata. While this rationale is still sound, snippets are used not just by authors but also by TeXperts to structure the internal implementation of themes. The latter use case would commonly benefit from the ability to pass parameters to snippets.

Syntax

The body of a snippet may contain control sequences titled \markdownParameter⟨parameter name⟩:

\markdownSetupSnippet
  { greet }
  { code = { Hello,~\markdownParameterWho! } }

When a snippet is called, a key–value that maps each ⟨parameter name⟩ to a value can be provided:

\markdownSetup
  { snippet = greet { who = world } }

The above code expands to snippet greet with all control sequences titled \markdownParameterWho replaced with the text "world".

\markdownSetup
  { code = { Hello,~world! } }

Considered alternatives

I considered using the parameters #1, #2, ..., #9 for the parameters:

\markdownSetupSnippet
  { greet }
  { code = { Hello,~#1! } }
\markdownSetup
  { snippet = greet { world } }

However, since snippets may contain renderer and function definitions with their own parameters, this would require that all parameter tokens (#) in existing snippets are doubled (##) when the snippets become parametric. Besides breaking backwards-compatibility, lists of parameters also seem less self-documenting and more error-prone compared to key–values.

I also considered using any control sequences as parameters, not just those titled \markdownParameter⟨parameter name⟩:

\markdownSetupSnippet
  { greet }
  { code = { Hello,~\who! } }
\markdownSetup
  { snippet = greet { who = world } }

While this makes snippet definitions less verbose, it is less obvious at a glance which control sequences are parameters. Besides the readability concerns, this also makes it impossible for the Markdown package to automatically determine whether all parameters were provided to a snippet call. Furthermore, while the ability to replace any control sequence may seem useful for patching snippets, this ability falls outside the intended scope of this change.

Examples

Lazy imports

Since version 2.22.0 (2023-04-02), the Markdown package has supported the import statement, which imports snippets from themes:

\markdownSetup
  {
    import = {
      jdoe / lists = romanNumerals as roman,
    },
  }

The above code imports the snippet romanNumerals from theme jdoe/lists immediately. However, since version 3.3.0 (2023-12-30), the Markdown package has supported plain TeX themes that can be loaded at any time, not just in the document preamble. This makes it possible to lazily import a snippet from a theme at the time of first use:

\markdownSetupSnippet
  { roman }
  {
    import = jdoe / lists,
    snippet = jdoe / lists / romanNumerals,
  }

The above code works but its intention seems unclear without additional comments. With parametric snippets, we can extract it into a higher-order snippet titled lazy-import:

\markdownSetupSnippet
  { lazy-import }
  {
    \markdownSetupSnippet
      { \markdownParameterAs }
      {
        import = \markdownParameterFrom,
        snippet = \markdownParameterFrom / \markdownParameterSnippet,
      }
  }

Then, we might lazily import the snippet romanNumerals from the theme jdoe/lists as follows:

\markdownSetup
  {
    snippet = lazy-import
      {
        snippet = romanNumerals,
        from = jdoe / lists,
        as = roman,
      }
  }

Local snippet calls

In Section 1.1 of Markdown Themes in Practice, I discuss the following design pattern that uses a snippet (here istqb/common/metadata/provided-by) locally inside a Markdown element (here jekyllDataSequence):

\markdownSetup
  {
    renderers = {
      jekyllDataSequenceBegin = {
        \markdownSetup
          {
            code = \group_begin:,
            renderers = {
              jekyllDataSequenceEnd =
            },
            snippet = istqb / common / metadata / provided-by,
            renderers = {
              jekyllDataSequenceEnd += \group_end:
            },
          }
      },
    }
  }

This design pattern is used in several places and is highly repetitive. Furthermore, the intention of the design pattern seems unclear without additional comments. With parametric snippets, we can extract this design pattern into a higher-order snippet titled local-call:

\markdownSetupSnippet
  { local-call }
  {
    renderers = {
      \markdownParameterElement Begin = {
        \markdownSetup
          {
            code = \group_begin:,
            renderers = {
              \markdownParameterElement End =
            },
            snippet = \markdownParameterSnippet,
            renderers = {
              \markdownParameterElement End += \group_end:
            },
          }
      },
    }
  }

Then, we might call the snippet as follows:

\markdownSetup
  {
    snippet = local-call
      {
        element = jekyllDataSequence,
        snippet = istqb / common / metadata / provided-by,
      }
  }

[Witiko]

I also considered using any control sequences as parameters, not just those titled \markdownParameter⟨parameter name⟩:

\markdownSetupSnippet
  { greet }
  { code = { Hello,~\who! } }
\markdownSetup
  { snippet = greet { who = world } }

While this makes snippet definitions less verbose, it is less obvious at a glance which control sequences are parameters. Besides the readability concerns, this also makes it impossible for the Markdown package to automatically determine whether all parameters were provided to a snippet call. Furthermore, while the ability to replace any control sequence may seem useful for patching snippets, this ability falls outside the intended scope of this change.

Although having the ability to convert a parameterless snippet into a parametric zero-arity snippet without changing the call syntax seems important, the definition syntax for parameterless and parametric snippets can be different. Namely, we might extend the snippet name with a comma-list of all accepted parameters at definition time:

\markdownSetupSnippet
  { greet { who } }
  { code = { Hello,~\who! } }
\markdownSetup
  { snippet = greet { who = world } }

This allows us to safely use shorthands such as \who while still allowing us to do the following like with the previous proposal:

• Parameters that are declared but not provided can be checked at call time.
• Parameters that are provided but undeclared can be checked at call time.

Furthermore, this also has the following benefits compared to the previous proposal:

• The parameters accepted by the snippet are clear at a glance without having to read the snippet body.
• Parameters that are declared but unused in the snippet body can be checked at definition time.

[Witiko]

Furthermore, it seems natural to also allow the definition to declare default values for parameters that would be used when no values are provided for a parameter at call time:

\markdownSetupSnippet
  { greet { who = world } }
  { code = { Hello,~\who! } }
% All three following calls are equivalent.
\markdownSetup
  { snippet = greet { who = world } }
\markdownSetup
  { snippet = greet { } }
\markdownSetup
  { snippet = greet }

To see how this might be useful, consider the example parametric snippet local-call from #445 (comment), rewritten using the new syntax proposed in this subthread:

\markdownSetupSnippet
  { local-call { element, snippet } }
  {
    renderers = {
      \element Begin = {
        \markdownSetup
          {
            code = \group_begin:,
            renderers = { \element End = },
            snippet = \snippet,
            renderers = { \element End += \group_end: },
          }
      },
    }
  }

For example, it may occasionally be useful to surround the code not just in \group_begin: and \group_end: but also other group types such as:

• \color_group_begin: and \color_group_end:
• \group_align_safe_begin: and \group_align_safe_end:
• \cctab_begin:N and \cctab_end:

Adding optional parameters before and after makes it possible to change the group type for callers who need it, whereas existing callers remain unaffected:

\markdownSetupSnippet
  { local-call { element, snippet, before = , after = } }
  {
    renderers = {
      \element Begin = {
        \markdownSetup
          {
            code = {
              \group_begin:
              \before
            },
            renderers = { \element End = },
            snippet = \snippet,
            renderers = {
              \element End += {
                \after
                \group_end:
              }
            },
          }
      },
    }
  }

This makes it possible to extend existing parametric snippets with new parameters in a backwards-compatible way:

% Both calls are legal.
\markdownSetup
  {
    snippet = local-call
      {
        element = jekyllDataSequence,
        snippet = istqb / common / metadata / provided-by,
      }
  }
\markdownSetup
  {
    snippet = local-call
      {
        element = jekyllDataSequence,
        snippet = istqb / common / metadata / provided-by,
        before = \color_group_begin:,
        after = \color_group_end:,
      }
  }

[Witiko]

It also seems useful to support some parameter expansion by allowing users to optionally use expl3-style argument specifiers for parametric snippet declarations (:N and :n) and calls (:V, :v, :x, ...):

\markdownSetupSnippet
  { greet { who:n = world } }
  { code = { Hello,~\who! } }
\tl_new:N
  \g_markdown_who_str
\tl_gset:Nn
  \g_markdown_who_str
  { world }
\markdownSetup
  { snippet = greet { who:V = \g_markdown_who_str } }

If left unspecified, the argument specifiers would default to :n / :N, similarly to the parameters of key–values.

This is related to #132: Rather than extending renderer (prototype) definitions to support expl3-style argument specifiers, users may route the definitions to parametric snippets with named parameters and receive the same benefit.

[Witiko]

Below are some of my concerns from reading this discussion thus far.

The expl3 programming language supports key–values, which is a power-horse for creating complex user interfaces. However, in many cases, a TeX macro would be a simpler and tidier abstraction if TeX macros supported named parameters with default values, similarly to e.g. Python. Reading this discussion seems like an attempt to add support for such improved TeX macros, which is fine but let's perhaps do that first, in a separate library, and then base parametric snippets on that library? This seems more useful to everyone, not just to the users of the Markdown package for TeX. We even have a name for the library already: pig-slow control sequences (pigcs) from latex3/latex3#1542 (comment); more on performance below.

In ticket #467 and in PR #479, we started precompiling our snippets, following @josephwright's talk about LaTeX templates at TUG 2024. This was a transparent optimization for us with no impact on our interfaces, since our snippets are parameterless whereas LaTeX templates are not and must be manually precompiled for different parameter values (so-called instances). However, making snippets parametric will open a new can of worms for us. While we can likely automatically just-in-time compile snippets for calls such as greet { who:V = \g_markdown_who_str }, where we can keep a late binding for the value of \g_markdown_who_str, we would still need some sort of an LRU cache for calls with many different values of :n parameters.

Even with clever optimizations, we will still be many times slower than TeX macros. While readable code seems like an OK place to spend CPU time and the conversion from Markdown to TeX is still the main bottleneck of this package, not the TeX macros that render the conversion output, this is still something to keep our eyes on.

[Witiko]

We haven't yet discussed how parametric snippets would interact with the import statement.

It seems useful to enable partial application on import:

% Import a parametric snippet as a simple snippet.
\markdownSetup
  {
    import = {
      witiko/markdown/examples = {
        greet { who = everyone } as greet-all 
      }
    }
  }
% Call the parametric snippet through a simple snippet.
\markdownSetup
  { snippet = greet-all }

Parameters without default values could be omitted but would need to be supplied when the snippet is called.

[Witiko]

However, making snippets parametric will open a new can of worms for us. While we can likely automatically just-in-time compile snippets for calls such as greet { who:V = \g_markdown_who_str }, where we can keep a late binding for the value of \g_markdown_who_str, we would still need some sort of an LRU cache for calls with many different values of :n parameters.

We may be able to dodge this with a simple rule such as: Precompile parametric snippets for parameters that were used at least 10 times in this compilation run. This should get us 90% there without introducing complex structures like an LRU cache.

It seems useful to enable partial application on import: [...]

Partial application seems as a useful feature for the pigcs package, discussed in #445 (reply in thread). To the degree that pigcs would be a more direct and intuitive interface for the key–values provided by l3keys, partial application would represent the support for inheritance in l3keys.

Even with clever optimizations, we will still be many times slower than TeX macros.

Moreover, on the performance side of things: Even if we precompile snippets, this compilation never goes beyond the boundaries of a single snippet. For example, if snippet witiko/markdown/examples/a calls snippet /b and /b calls /c, we would not include /c in the precompiled /a, since 1) both /b and /c can be potentially redefined and 2) neither /b nor /c are necessarily defined yet when we compile or call /a. However, since redefining snippets is undefined behavior, 1) provides opportunities for future optimizations. 2) is more tricky but we might perform partial compilation and then fully compile when /b and /c become available.

Regardless, as discussed in #445 (reply in thread), the TeX side of things is not currently a bottleneck, so none of these optimizations seem currently necessary or even desirable. This discussion is more about future-proofing the design, so that we can optimize if we need to in the future without breaking our interfaces.