Plain-text TOC: Can the Table of Contents be made to ignore HTML tags within headings?

[artistofmind]

Hello,

I tend to individually format headings, e.g. with line breaks and big and small tags, so I can control the number of words per line in long titles; and sometimes I render some words in a different font (small caps, swash italics), as you might see in a magazine for example. (Of course I specify a default style with CSS, but these tweaks are by nature discretionary.) All of this works just fine; however, it’s duplicated in the automatically generated Table of Contents, where it’s unwanted.

Is there a way to have the TOC simply ignore inline formatting, and just generate the list of section titles in plain text?

Currently the only workarounds I’ve found are (1) manually editing documents post conversion to cleanup the TOC, or (2) hiding actual header elements via CSS, then adding redundant text in DIVs styled the same way right after each one. Obviously, neither approach is optimal, so I just thought I’d ask here if there’s a more direct fix.

The automatic generation of a hyperlinked TOC is a swell feature, and a real time saver; however, there’s no need for it to reproduce any formatting that happens to exist within headers. I think that’s taking things a step too far; my only complaint, then, is that this feature works too well! 😅

Cheers

[jgm]

Well in general, we certainly wouldn't want to remove all formatting. For example, if a section heading is `The plot of the Iliad', we'd like 'Iliad' to be italicized in the TOC too. I think that currently we do remove links and footnotes.

[artistofmind]

Of course italics aren’t as big a deal as differently sized text, let alone line breaks. 😊 But I sometimes use them purely stylistically; and while I may want a particular heading to receive that style, I don’t need it reproduced in the TOC. Personally, I wouldn’t even want it in the case you describe: while titles are typically italicized in body text, it’s rare to italicize them in other titles. 🤷 So that’s an uncommon use case. (That said, I see no problem with making an exception for the CITE tag, so when that one is specifically used it’s still reflected there. Perhaps users could manually specify such exceptions.)

But what I have in mind isn’t a change to the default handling of this feature — which I appreciate works fine for most people — rather, I was simply wondering if there is, or could be implemented, a command (under e.g. pandoc.htmlOptString in VS Code’s settings.json) to force plain text in the TOC, similar to the way you can currently tell Pandoc to output ASCII entities instead of Unicode, for instance. So it would be strictly optional.

Bottom line: If you prefer the current behavior, you don’t have to put that command in your settings. But crazy people like me would be free to format headings to our hearts’ content, and keep the TOC tidy despite it — win/win. The beauty of configurability, no?

[Moonbase59]

@artistofmind If you’re working from Markdown, you might want to play around with my little gh-toc Markdown ToC generation tool, which removes HTML from headings (but leaves content). Just because I also needed that feature…

I typically use it before Pandoc conversion, since it was originally meant to make ToCs for GitHub.

In general I agree: Would be nice to have something like "strip some" (as we have) and "strip all". But of course the next question would be: Remove contents between HTML tags or not?

[bpj]

Note that it’s perfectly possible to create a Markdown or GFM ToC with Pandoc:

## First heading

First text.

## Second heading

Second text.

pandoc -sw gfm --toc readme.md

- [First heading](#first-heading)
- [Second heading](#second-heading)

## First heading

First text.

## Second heading

Second text.

pandoc -sw markdown --toc readme.md

-   [First heading](#first-heading){#toc-first-heading}
-   [Second heading](#second-heading){#toc-second-heading}

## First heading

First text.

## Second heading

Second text.

I do it often. Just one caveat: you will want to keep your source, say docs/readme.md and your actual README.md distinct so that you don’t need to delete the old ToC manually every time you rebuild it!

[tarleb]

Here's an approach that depends on implementation details, so there's no guarantee that it's going to keep working in the future. But with most templates, setting the table-of-contents metavalue in a Lua filter would work:

function Pandoc (doc)
  doc.meta['table-of-contents'] = pandoc.structure.table_of_contents(doc):walk{
    -- Unlink and remote notes
    Link = function (link) return link.content end,
    Note = function (note) return pandoc.utils.blocks_to_inlines(note.content) end,
  }
  return doc
end

Use the above filter by saving it to a file, and then passing it to pandoc via the -L/--lua-filter command line option.

pandoc -L my-filter.lua --toc ...

[wlupton]

Interesting. I've done that too, but I didn't realize that it relied on an implementation detail. Could the table-of-contents metadata variable become a documented feature?

[jgm]

Only certain formats handle the toc by inserting content into this variable, and it will only work for those formats:

% rg table-of-contents data/templates 
data/templates/default.markua
14:$table-of-contents$

data/templates/default.dzslides
180:$table-of-contents$

data/templates/default.commonmark
14:$table-of-contents$

data/templates/default.html4
65:$table-of-contents$

data/templates/default.plain
14:$table-of-contents$

data/templates/default.html5
65:$table-of-contents$

data/templates/default.s5
74:$table-of-contents$

data/templates/default.slidy
67:$table-of-contents$

data/templates/default.slideous
83:$table-of-contents$

data/templates/default.markdown
14:$table-of-contents$

data/templates/default.revealjs
72:$table-of-contents$

data/templates/default.chunkedhtml
95:$table-of-contents$

data/templates/default.rtf
21:$table-of-contents$

[wlupton]

Makes sense (thanks), but all the same might it not be worth mentioning it in the manual? I'd be happy to create an issue/PR.

[jgm]

I'd rather not clutter the manual with this. People who want to exploit this kind of implementation detail can inspect the templates themselves.

[bpj]

@jgm @wlupton Generating a TOC with GFM/Markdown is in the GitHub wiki.

I don't think there is any link to the wiki in the manual, but perhaps there should be. IMO the wiki is generally underused, because it is hard to find. FAQs should certainly be answered there. Often it is just a matter of copying an answer in discussions/issues and pasting it somewhere in the wiki.

[artistofmind]

Thanks, everyone, for sharing all these different ideas! It’s heartening to know I’m not the only user who’s found the current functionality less than ideally suited to my needs, despite its working so well. 😊

Do I understand correctly that most of these approaches involve disabling the TOC in Pandoc, and generating one with another extension, instead? . . . That’s something I hadn’t considered doing, and I confess I’m a bit overwhelmed at the prospect. But I’ll give it due consideration.

Has anyone tried more than one of the above methods? Which one is the easiest to implement, gives you the most control, is the most compatible, or for some other reason should be tried first? I guess I’m looking for the “least radical” solution.

Finally, is it at all likely that this functionality will eventually be added to Pandoc? Should I just create an issue for it, instead of asking questions here?

[wlupton]

@artistofmind, I can offer this lua function (see below for usage examples):

-- helper to generate a table of contents or navigation bar from the supplied
-- elements (which will be either headers or sections); returns (by default)
-- (1) a bulleted list, and (2) the index of the next element to be processed
local function table_of_contents(elems, opts)
    opts = opts or {}
    local start = opts.start or 1
    local curlev = opts.curlev or 1
    local maxlev = opts.maxlev or TOC_DEPTH
    local ignunl = opts.ignunl or false
    local joiner = opts.joiner or pandoc.BulletList

    local items = pandoc.List()

    local index = start
    while index <= #elems do
        local elem = elems[index]
        local level = elem.level
        local content = elem.content
        local identifier = elem.attr.identifier
        local classes = elem.attr.classes

        if level > maxlev then
            -- ignore
        elseif not ignunl and classes:includes('unlisted') then
            -- ignore
        elseif level < curlev then
            goto RETURN
        elseif level == curlev then
            local item = pandoc.Plain(
                identifier and pandoc.Link(content, '#' .. identifier) or
                    content)
            items:insert(pandoc.List({item}))
        else
            local item = nil
            item, index = table_of_contents(
                    elems, {start=index, curlev=level, maxlev=maxlev,
                              ignunl=ignunl, joiner=joiner})
            index = index - 1
            if #items == 0 then
                items:insert(pandoc.List())
            end
            items[#items]:insert(item)
        end

        index = index + 1
    end

How I use it (the sidebar and navigation ToCs are used by a custom template):

    <all_headers and sections are assumed defined; they're both lists of headers,
     and sections are typically level-1 headers>

    -- the pandoc-generated ToC omits most headers within sections, so we
    -- can't rely on it; also, only generate ToCs when there are more than
    -- two sections (the first is always the cover page)
    local navigation = meta['navigation']
    if #sections > 2 then
        meta['table-of-contents'] = table_of_contents(all_headers)
        meta['sidebar'] = meta['table-of-contents']
        if not navigation then
            meta['navigation'] = table_of_contents(
                sections, {ignunl=true, joiner=joiner})
        end
    end

FYI this is the joiner function that's used above:

    -- the default joiner is suitable for a pandoc.BulletList; this one is used
    -- for navigation bars
    local joiner = function(blocks_list)
        local items = pandoc.List()
        for _, blocks in ipairs(blocks_list) do
            for _, block in ipairs(blocks) do
                items:insert(block)
            end
        end
        return pandoc.Para(
            pandoc.utils.blocks_to_inlines(items, pandoc.Inlines(' | ')))
    end

Please let me know if I forgot anything or if you have any questions.

[artistofmind]

I figure I should share with you all the simple solution I came up with, or rather stumbled upon.

I noticed HTML headers don’t generate an entry in the TOC, only Markdown ones do. (So Pandoc adds ###, but ignores <h3>.) I now use both, and place the Markdown one inside a div with “display: hidden.” I then have the ability to style each version of the title independently. 🙂

So, using an example from earlier in this thread, I can still italicize Iliad in the TOC if I wish, and just avoid any line breaks there. Meanwhile, I can stylize the in-text chapter title to my heart’s content, without affecting the TOC in any way. Perhaps even add a subtitle there, while displaying only the main title in the TOC. 🤷

As an added benefit, VS Code’s internally generated index also sees the Markdown and ignores the HTML. Since the former is generally simpler, this makes it easier to navigate in the editor. 😉

Finally, since it’s linking to hidden text, the anchor point can be independently placed. So I can put a chapter number in between the two versions of the title, and the TOC link effectively takes you to that number, displaying the title under it. (Before, it would go right to the title, and you’d have to scroll up to see the number.)

Just a little bit more work, for a great deal more control. 👌