Internal mermaid clickable links

[dionjwa]

Fixes #28

I use mermaid diagrams with clickable link to other notion documents:

You can click on "An internal page" and it takes you to another notion document.

This PR adds some config and processing to get those internal mermaid URLs working with docu-saurus.

I tried to make a CodeTransformer in the same pattern as the other transformers, but it didn't have the right context for getting the links path to the notion documents, so I just added a link processor.

The result is the same mermaid diagrams but the links correctly link to docusaurus pages:

You can then click the first node and you go to another page.

There is an extra config flag, I was not able to figure out how to align the different docusaurus root documents you can specify (e.g. ./docs, ./blog), with the docu-notion->saurus step, so some experimentation may be needed to align the paths and the slugs.

---

This change is 

[hatton]

@dionjwa thanks for this. Sorry for the slow response, we'll be digging out from our December hibernation for a while yet. Your need here puts a spotlight on our need for a plugin capability. My own product's even rarer needs -- embedding our ebook types -- also should be moved out to a plugin. I'm hopeful that I might be able to work on that next week.

[dionjwa]

I think that plugins need:

• every bit of context available
• a canonical way to compute URLs for a given document
• possibly a preferred ordering if URLs cannot be computed immediately (this is purely speculative, I don't know enough about the internals)

It would definitely be possible move the mermaid URL processing into a plugin then and would avoid having to regex everything.

[hatton]

Good point about ordering. I'm thinking that we can convert most of the existing transforms into plugins that are just enabled by default. If it exists ./docunotion.config.js will be read to get settings including plugins, and so I suppose that will also need a way to define an ordering.
Other requirements that occur to me:

• A chance to transform the results of the JSON provided by Notion before it gets to the markdown converter
• A chance to transform the markdown after it comes out of the markdown converter
• Perhaps Access to both the official and unofficial Notion APIs, since we're running into cases where the official one just doesn't give you the info you need.

[dionjwa]

Those make a lot of sense. My feeling is with those options, plugin writers would have all they needed produce the final docu-saurus markdown 👍🏻

[hatton]

Just an update... I'm about halfway through restructuring everything, including all the built-in stuff, as plugins. Part of this change is to make it easy to write unit tests on plugins that don't have to hit actual Notion API.

[dionjwa]

Sounds good, and I appreciate the update! I am in no rush because I am using my own local repo for this right now, and am happy to adapt this to the new plugin system once you're done.

[hatton]

OK @dionjwa, the plugin stuff is now merged. Can you give it a try? See https://github.com/sillsdev/docu-notion/blob/main/src/plugins/README.md.

For your custom mermaid slug, instead of a command-line argument, you'd put something in the new docu-notion.config.ts. I'm afraid I didn't try or demonstrate a plugin that takes arguments, maybe we'll need to do more there.

[dionjwa]

Hi @hatton I'm having trouble getting this this working.

"docu-notion": "^0.11.0"

docu-notion.config.ts:

import { IDocuNotionConfig, IPlugin, IDocuNotionContext, Log } from "docu-notion";
import { join} from "path";

// The mermaid interactive click syntax:
// https://mermaid.js.org/syntax/flowchart.html#interaction
// NB this processing is just for internal link navigation
// const linkRegExp = /\s*click\s+([A-za-z][A-za-z0-9_-]*)\s+"(https:\/\/www\.notion\.so\/\S*)"/g;
const linkNotionRegExp = /^(https:\/\/www\.notion\.so\/)(.*)/g;

// This is an example of a plugin that needs customization by the end user.
// It uses a closure to supply the plugin with the customization parameter.
const pluginName = "mermaidLinks";
function docunotionMermaidLinks(args:{slugPrefix:string}): IPlugin {
  const {slugPrefix} = args;
  return {
    name: pluginName,

    // corrections to links after they are converted to markdown
    linkModifier: {
      match: linkNotionRegExp, // does this plugin apply to this link?
      convert: (
        context: IDocuNotionContext,
        markdownLink: string
      ) => {
        Log.verbose(`[plugin ${pluginName}] markdownLink ${markdownLink}`);
        console.log("mermaidLinks ", markdownLink)
        return "fakelink";
      }
    },
  };
}

const config: IDocuNotionConfig = {
  plugins: [docunotionMermaidLinks({slugPrefix:"customParameter"})],
};
export default config;

NB: I am setting the links to some fake link to check that this plugin is working.

Command:

node_modules/docu-notion/$(cat node_modules/docu-notion/package.json | jq -r .bin) --notion-token {{NOTION_TOKEN}} --root-page {{NOTION_DOCUMENT_ROOT}} --markdown-output-path $(pwd)/{{DOCU_NOTION_OUTPUT_PATH}} --log-level verbose

I see the config loaded and the plugin detected:

Active plugins: [standardEscapeHtmlBlockModifier, standardHeadingTransformer, standardColumnTransformer, standardColumnListTransformer, DownloadImagesToRepo, standardCalloutTransformer, standardTableTransformer, standardNumberedListTransformer, standard internal link conversion, standard external link conversion, imgur, gif, youtube, vimeo, mermaidLinks]

But I don't see any processing by the plugin. console.logs don't work, and Log.verbose indicates that the plugin is not getting called. But the regex matches the links supplied.

The mermaid diagram in my notion page looks like:

graph LR
  A[An internal page] -->|Get money| B(Go shopping)
  click A "https://www.notion.so/metapages/Introduction-to-docu-notion-779f83504bd94642a9b87b2afc810a97"

Loading

The page link in the mermaid diagram is to a valid docu-notion page, as its processed and shows up in docusaurus:

So my problems are:

• am I using the linkModifier field of IPlugin correctly?
• is logging working correctly? Am I using it wrong?
• If my goal is to update links inside mermaid markdown block, is this the recommended way to process these links?

[dionjwa]

Also: the linkModifier function has the argument context: IDocuNotionContext which is required here, but the other means of modification e.g. notionBlockModifications does not have this context passed in. This feels like a conflict or something I don't understand: shouldn't all means of modification all have the IDocuNotionContext?

[dionjwa]

Also: I run this process multiple times, for docs and for blog and need to adjust the link outputs for both. For each call, I am adjusting an argument, but this is not very convenient for strict config based approaches. I think this can be solved by passing in env vars? I'm not sure this is the best solution. Sorry, my problem is complicated :-)

[dionjwa]

More:

When I make a customnotionBlockModifications

Synced code blocks are not populated, but one synced block is visible, the other is not. IE one version of the synced block is correctly shown in the final docs, but the other synced block is not shown. Both NotionBlock objects do not have the synced content. Maybe this is where I call out to the notion API myself?

[plugin mermaidLinks] block.type synced_block
block {"object":"block","id":"b701a7c4-fccb-42ea-a81b-4687b711810c","parent":{"type":"page_id","page_id":"63f6f281-a336-49cc-9dfa-bb86d26ad829"},"created_time":"2022-12-27T19:34:00.000Z","last_edited_time":"2022-12-27T19:34:00.000Z","created_by":{"object":"user","id":"38433169-6845-4c23-a787-f24081088d0c"},"last_edited_by":{"object":"user","id":"38433169-6845-4c23-a787-f24081088d0c"},"has_children":true,"archived":false,"type":"synced_block","synced_block":{"synced_from":null}}

[dionjwa]

I think the lack of data for synced blocks is what is blocking me. Correct links in mermaid diagrams are important, but the real value comes when I can create them and sync them across different places, which I cannot seem to get working here.

I think if I had access to the configured notion client I could make these API calls myself

[dionjwa]

Feedback about the plugin system:

• all plugin methods should:
  • be async, so await (fetching data) can be done
  • be provided with the IDocuNotionContext
    • and the IDocuNotionContext should have the notion client be public and acccessible

There appears to be a bug with notion-to-md where the synced_block outputs double. I could work around this by converting all synced blocks to their content before processing, but this doesn't work because notionBlockModifications is not async not does it have context.

[hatton]

The plugin isn't being called because the regular expression in your code above was only matching on URLs, rather than the complete markdown link. This works for matching:

const linkNotionRegExp = /\[([^\]]+)\]\((https:\/\/www\.notion\.so\S+)\)/;

[hatton]

Well actually, two reasons. The other is that docu-notion isn't applying changes inside of code blocks, which normally would be what we want, right? The normal point of a code block is to say "take this literally, don't interpret it".

Maybe we could reasonably say that link conversion should still happen within mermaid code blocks?

[hatton]

I'm working on seeing if this would be an appropriate approach:

  const mermaidLinks: IPlugin = {
    name: "mermaidLinks",
    regexMarkdownModifications: [
      {
        regex: /```mermaid\n.*"(https:\/\/www\.notion\.so\S+)"/,
        getReplacement: async (context: IDocuNotionContext, s: string) => {
          return await context.convertNotionLinkToLocalDocusaurusLink(s);
        },
      },
    ],
  };

That is,

• use a plugin that operates on the final markdown
• supply such plugins with an IDocuNotionContext
• add a method to the context that knows how to take a notion link and give you back a docusuarus link, including using the slug

[hatton]

OK, @dionjwa please see the branch linkConversionFromPlugin and the sample mermaid plugin there.

If it is enough, then I can get that branch merged so that you can use this approach in a real plugin.

[dionjwa]

If I check out that branch, and run npm run test it shows a whole lot of errors in src/transform.ts. It also breaks my local setup. Is the branch up-to-date?

[dionjwa]

@hatton I got the mermaid links plugin working on the latest alpha package.

I've put together all the plugins I needed so that I got the results I needed, I hope they are helpful.

Nice work!

https://www.npmjs.com/package/@metapages/docu-notion-plugins