Using prism-js-templates support with Formidable/prism-react-renderer

[JaKXz]

I am working with this implementation of Prism in a React component and I'm wondering how I can leverage prism-js-template support in a jsx codeblock with my and @RunDevelopment's recent contributions to the improved GraphQL support :)

The library does run the before- and after-tokenize hooks, but, I think they have an explicit limitation on how deep down the token tree they will go... that being said, the tokens logged below are directly copied from the function linked in this paragraph, as far as I can tell.

I wanted to write here just in case it's simply solvable with some configuration -- I have passed in my own Prism setup as documented and I am calling require('prismjs/components/prism-js-templates'), but:

this code sample

import React from 'react';
import gql from 'graphql-tag';
import { Query } from 'react-apollo';
import {
  Card,
  ResourceList,
  Stack,
  TextStyle,
  Thumbnail,
} from '@shopify/polaris';
import store from 'store-js';
import { Redirect } from '@shopify/app-bridge/actions';
import { Context } from '@shopify/app-bridge-react';

// GraphQL query to retrieve products by IDs.
// The price field belongs to the variants object because
// variations of a product can have different prices.
const GET_PRODUCTS_BY_ID = gql`
  query getProducts($ids: [ID!]!) {
    nodes(ids: $ids) {
      ... on Product {
        title
        handle
        descriptionHtml
        id
        images(first: 1) {
          edges {
            node {
              originalSrc
              altText
            }
          }
        }
        variants(first: 1) {
          edges {
            node {
              price
              id
            }
          }
        }
      }
    }
  }
`;

is tokenized by the library [even though it seems to call the same set of hooks as Prism itself] to:

[
    {
        "type": "keyword",
        "content": "import",
        "length": 6
    },
    " React ",
    {
        "type": "keyword",
        "content": "from",
        "length": 4
    },
    " ",
    {
        "type": "string",
        "content": "'react'",
        "length": 7
    },
    {
        "type": "punctuation",
        "content": ";",
        "length": 1
    },
    "\n",
    {
        "type": "keyword",
        "content": "import",
        "length": 6
    },
    " gql ",
    {
        "type": "keyword",
        "content": "from",
        "length": 4
    },
    " ",
    {
        "type": "string",
        "content": "'graphql-tag'",
        "length": 13
    },
    {
        "type": "punctuation",
        "content": ";",
        "length": 1
    },
    "\n",
    {
        "type": "keyword",
        "content": "import",
        "length": 6
    },
    " ",
    {
        "type": "punctuation",
        "content": "{",
        "length": 1
    },
    " Query ",
    {
        "type": "punctuation",
        "content": "}",
        "length": 1
    },
    " ",
    {
        "type": "keyword",
        "content": "from",
        "length": 4
    },
    " ",
    {
        "type": "string",
        "content": "'react-apollo'",
        "length": 14
    },
    {
        "type": "punctuation",
        "content": ";",
        "length": 1
    },
    "\n",
    {
        "type": "keyword",
        "content": "import",
        "length": 6
    },
    " ",
    {
        "type": "punctuation",
        "content": "{",
        "length": 1
    },
    "\n  Card",
    {
        "type": "punctuation",
        "content": ",",
        "length": 1
    },
    "\n  ResourceList",
    {
        "type": "punctuation",
        "content": ",",
        "length": 1
    },
    "\n  Stack",
    {
        "type": "punctuation",
        "content": ",",
        "length": 1
    },
    "\n  TextStyle",
    {
        "type": "punctuation",
        "content": ",",
        "length": 1
    },
    "\n  Thumbnail",
    {
        "type": "punctuation",
        "content": ",",
        "length": 1
    },
    "\n",
    {
        "type": "punctuation",
        "content": "}",
        "length": 1
    },
    " ",
    {
        "type": "keyword",
        "content": "from",
        "length": 4
    },
    " ",
    {
        "type": "string",
        "content": "'@shopify/polaris'",
        "length": 18
    },
    {
        "type": "punctuation",
        "content": ";",
        "length": 1
    },
    "\n",
    {
        "type": "keyword",
        "content": "import",
        "length": 6
    },
    " store ",
    {
        "type": "keyword",
        "content": "from",
        "length": 4
    },
    " ",
    {
        "type": "string",
        "content": "'store-js'",
        "length": 10
    },
    {
        "type": "punctuation",
        "content": ";",
        "length": 1
    },
    "\n",
    {
        "type": "keyword",
        "content": "import",
        "length": 6
    },
    " ",
    {
        "type": "punctuation",
        "content": "{",
        "length": 1
    },
    " Redirect ",
    {
        "type": "punctuation",
        "content": "}",
        "length": 1
    },
    " ",
    {
        "type": "keyword",
        "content": "from",
        "length": 4
    },
    " ",
    {
        "type": "string",
        "content": "'@shopify/app-bridge/actions'",
        "length": 29
    },
    {
        "type": "punctuation",
        "content": ";",
        "length": 1
    },
    "\n",
    {
        "type": "keyword",
        "content": "import",
        "length": 6
    },
    " ",
    {
        "type": "punctuation",
        "content": "{",
        "length": 1
    },
    " Context ",
    {
        "type": "punctuation",
        "content": "}",
        "length": 1
    },
    " ",
    {
        "type": "keyword",
        "content": "from",
        "length": 4
    },
    " ",
    {
        "type": "string",
        "content": "'@shopify/app-bridge-react'",
        "length": 27
    },
    {
        "type": "punctuation",
        "content": ";",
        "length": 1
    },
    "\n\n",
    {
        "type": "comment",
        "content": "// GraphQL query to retrieve products by IDs.",
        "length": 45
    },
    "\n",
    {
        "type": "comment",
        "content": "// The price field belongs to the variants object because",
        "length": 57
    },
    "\n",
    {
        "type": "comment",
        "content": "// variations of a product can have different prices.",
        "length": 53
    },
    "\n",
    {
        "type": "keyword",
        "content": "const",
        "length": 5
    },
    " ",
    {
        "type": "constant",
        "content": "GET_PRODUCTS_BY_ID",
        "length": 18
    },
    " ",
    {
        "type": "operator",
        "content": "=",
        "length": 1
    },
    " gql",
    {
        "type": "template-string",
        "content": [
            {
                "type": "template-punctuation",
                "content": "`",
                "alias": "string",
                "length": 1
            },
            {
                "type": "string",
                "content": "\n  query getProducts($ids: [ID!]!) {\n    nodes(ids: $ids) {\n      ... on Product {\n        title\n        handle\n        descriptionHtml\n        id\n        images(first: 1) {\n          edges {\n            node {\n              originalSrc\n              altText\n            }\n          }\n        }\n        variants(first: 1) {\n          edges {\n            node {\n              price\n              id\n            }\n          }\n        }\n      }\n    }\n  }\n",
                "length": 452
            },
            {
                "type": "template-punctuation",
                "content": "`",
                "alias": "string",
                "length": 1
            }
        ],
        "length": 454
    },
    {
        "type": "punctuation",
        "content": ";",
        "length": 1
    }
]

which is different from what is produced in the tokenizer tests when you put this in tests/languages/jsx!+graphql+js-templates (which behaves as expected of course).

The correct tokenization properly parses the graphql template string:

[
	["keyword", "import"],
	" React ",
	["keyword", "from"],
	["string", "'react'"],
	["punctuation", ";"],

	["keyword", "import"],
	" gql ",
	["keyword", "from"],
	["string", "'graphql-tag'"],
	["punctuation", ";"],

	["keyword", "import"],
	["punctuation", "{"],
	" Query ",
	["punctuation", "}"],
	["keyword", "from"],
	["string", "'react-apollo'"],
	["punctuation", ";"],

	["keyword", "import"],
	["punctuation", "{"],

	"\r\n  Card",
	["punctuation", ","],

	"\r\n  ResourceList",
	["punctuation", ","],

	"\r\n  Stack",
	["punctuation", ","],

	"\r\n  TextStyle",
	["punctuation", ","],

	"\r\n  Thumbnail",
	["punctuation", ","],

	["punctuation", "}"],
	["keyword", "from"],
	["string", "'@shopify/polaris'"],
	["punctuation", ";"],

	["keyword", "import"],
	" store ",
	["keyword", "from"],
	["string", "'store-js'"],
	["punctuation", ";"],

	["keyword", "import"],
	["punctuation", "{"],
	" Redirect ",
	["punctuation", "}"],
	["keyword", "from"],
	["string", "'@shopify/app-bridge/actions'"],
	["punctuation", ";"],

	["keyword", "import"],
	["punctuation", "{"],
	" Context ",
	["punctuation", "}"],
	["keyword", "from"],
	["string", "'@shopify/app-bridge-react'"],
	["punctuation", ";"],

	["comment", "// GraphQL query to retrieve products by IDs."],

	["comment", "// The price field belongs to the variants object because"],

	["comment", "// variations of a product can have different prices."],

	["keyword", "const"],
	["constant", "GET_PRODUCTS_BY_ID"],
	["operator", "="],
	" gql",
	["template-string", [
		["template-punctuation", "`"],
		["graphql", [
			["keyword", "query"],
			["definition-query", "getProducts"],
			["punctuation", "("],
			["variable", "$ids"],
			["punctuation", ":"],
			["punctuation", "["],
			["scalar", "ID"],
			["operator", "!"],
			["punctuation", "]"],
			["operator", "!"],
			["punctuation", ")"],
			["punctuation", "{"],

			["property-query", "nodes"],
			["punctuation", "("],
			["attr-name", "ids"],
			["punctuation", ":"],
			["variable", "$ids"],
			["punctuation", ")"],
			["punctuation", "{"],

			["operator", "..."],
			["definition-mutation", "on"],
			["class-name", "Product"],
			["punctuation", "{"],

			["property", "title"],

			["property", "handle"],

			["property", "descriptionHtml"],

			["property", "id"],

			["property-query", "images"],
			["punctuation", "("],
			["attr-name", "first"],
			["punctuation", ":"],
			["number", "1"],
			["punctuation", ")"],
			["punctuation", "{"],

			["object", "edges"],
			["punctuation", "{"],

			["object", "node"],
			["punctuation", "{"],

			["property", "originalSrc"],

			["property", "altText"],

			["punctuation", "}"],

			["punctuation", "}"],

			["punctuation", "}"],

			["property-query", "variants"],
			["punctuation", "("],
			["attr-name", "first"],
			["punctuation", ":"],
			["number", "1"],
			["punctuation", ")"],
			["punctuation", "{"],

			["object", "edges"],
			["punctuation", "{"],

			["object", "node"],
			["punctuation", "{"],

			["property", "price"],

			["property", "id"],

			["punctuation", "}"],

			["punctuation", "}"],

			["punctuation", "}"],

			["punctuation", "}"],

			["punctuation", "}"],

			["punctuation", "}"]
		]],
		["template-punctuation", "`"]
	]],
	["punctuation", ";"]
]

[RunDevelopment]

The problem seems to be that JS templates isn't loaded correctly. Could you provide a minimal example that reproduces the bug?

[JaKXz]

Working on a Codesandbox as we speak :)

[JaKXz]

@RunDevelopment here ya go: https://codesandbox.io/s/prism-react-renderer-js-templates-support-fkp9p

There are some slight differences between how this is setup vs how I have it actually working in my project but this is the only way I could get the codesandbox to play nice with everything going on... what I have working in my project is what is commented out. It won't show the new tokens we added because it won't let me load from master on GitHub unfortunately.

[RunDevelopment]

Ok, this took me a while. The problem has two layers to it.

Problem 1: JSX

JSX extends JS. For JS Templates to affect JSX, JS Templates has to be loaded before JSX. Since JSX is included by default, JSX has to be reloaded after JS Templates.

Fix 1: Add a require("prismjs/components/prism-jsx"); after require("prismjs/components/prism-js-templates");

This will reload JSX.

This problem isn't your fault. The FormidableLabs doc is simply incorrect. Languages have dependencies, so simply importing files won't always work.

Problem 2: Hooks

But even after the JSX fix, the GraphQL code still isn't highlighted. The template string was correctly tokenized as a GraphQL template but for some reason, the embedded GraphQL code isn't highlighted.

This is because the after-tokenize hook isn't run.

I have no idea why this hook isn't run, it should be but it isn't. You can see this for yourself using: Prism.hooks.add('after-tokenize', (env) => { console.log("I was executed!"); }).

[JaKXz]

my previous comment fixed by Fix 1

With some digging I have found something interesting... based on the codesandbox above

By adding this line to the loadAdditionalLangs file:

window.Prism.languages.jsx['template-string'] = window.Prism.languages.javascript['template-string'];

I'm seeing the tokenization come out [from prism-react-renderer] like this:

<span class="token template-string embedded-code graphql">  query getProducts($ids: [ID!]!) {</span>

[JaKXz]

@RunDevelopment I am noticing a strange issue (if you try this whole setup out on your machine) with PrismJS/prism#master installed -- I made some updates so you can put in a different code sample, and found if you change to the language to liquid with a valid liquid code sample you don't get the new and improved highlighting. This seems to "work" on the codesandbox but I'm not sure why, and the only difference I can see is that we're only able to load the published v1.2.3 from npm in the sandbox.