Regression in \verbatim & \autodoc:codeblock handling of blank lines

[alerque]

As reported by @chrisvest on mastodon the SILE manual has an issue with whitespace handling in code samples. We've lost all blank line to paragraph transformations in code samples. These days in the manual that's with \autodoc:codeblock, but the mechanism should be essentially the same as \verbatium. I've tried to get an MWE to bisect but I can't find an MWE that works/doesn't work in a reliable way. In fact at the moment even in versions where the manual appears to be correct my MWE's do not work.

Paging back through release manuals, we apparently completely broke all the code sample rendering in v0.14.10. That would make sense if you just looked at the fact that we used all now formatting macros for the whole manual starting in that release, but even restoring the v0.14.9 code for rendering code blocks or building a quick check with \verbatium still fails even in v0.14.9. And indeed all the way back to v0.11.0 at least.

Here is the v0.14.9 manual:

One release later in v0.14.10 the parbreak is gone:

And here it is today in v0.15.5:

The source is still to this day like I would expect:

\begin[type=autodoc:codeblock]{raw}
Paragraph one.

Paragraph two.
This is not paragraph three.

This is paragraph three.
\end{raw}

Any ideas @Omikhleia?

[Omikhleia]

Any ideas @Omikhleia?

Hmmm. It if ever worked and got broken at one point, it might even have occurred early (I remember having added explicit spaces in my own verbatim examples some long time ago...).

I also mentioned the issue here in a longer issue: #2006 (comment) = "The way SILE implements obeying to lines in such environments is not perfect (it doesn't honor blank lines, which are skipped)." -- And mentioned opening a bug issue but apparently I wholly forgot ^^

Without looking to the code right now, it seems I was onto something (empty lines are skipped somewhere). Digging in the tests, we do have a test "Empty lines are skipped" for #479 = PR #1766 "Conservative approach, skipping empty lines and lines containing only discardable nodes (incl. glues)."

But the latter (which went into 0.14.9) did fix some other tricky/worse issue (content disappearing in some cases)... So perhaps something is missing all along before even that fix, for the specific case of verbatim...

[Omikhleia]

Hmm... (again)... So #479 (0.14.9) might have caused a side effect here maybe.
But digging further, we already had some weird thing on blank lines in verbatim environment: https://github.com/Omikhleia/omikhleia-sile-packages/blob/c3aa22153efcd98321e559cf9a791952e7909d06/packages/experimental/codehighlighter.lua#L40 = That was in 0.12.5 and visibly I had troubles there too.

[Omikhleia]

Hint: If we can have a MWE, we should track this debug: SU.debug("typesetter", "Skipping a line containing only discardable nodes") -- maybe the seenNonDiscardable condition should not apply as-is in cases where we want to honor empty lines (= typically verbatim / raw)

[Omikhleia]

But then we'll maybe wonder "how can we know?" and notice all we have is a non-semantic typesetter.parseppattern setting... It does ring another bell on what are paragraphs (see #2111 § "Thoughts on paragraphing")...

[Omikhleia]

( Unrelated note FWIW: the other issue reported by chrisvest on Mastodon regarding the split footnote is likely a bug too. Not the fact that the footnote gets split, -- but the fact that on this very page there's visibly far enough space for it to fit without needing the split if frames are broken as they are here -- or (rather) that much more more text could have been pushed to the main frame then, and the split would have been more legit.... And my deep feeling is that #1828 or a friend of it might be at play, with "gaps" sometimes occurring, although I nearly never observed them in my own books, but on one occasion which forced me to rephrase some content...)

[alerque]

The change in #1766 for #479 was first released in v0.14.9. I tried throwing an MWE against v0.14.8 and it behaved the same way, so I am still confused.