Formatting of long text files is slow

[mthuurne]

On a fast PC, a log file of 20k lines takes about 3.5 seconds to render at the server side and 10 seconds to be rendered by the web browser (tried both Chrome and Firefox). While Chrome reports this time as "Content Download", further tests show it is actually the layout that takes up this time, not loading the page source.

The server-side rendering can probably be optimized. Currently we process the text with Pygments, but we could use specialized code instead. Also xmlgen hasn't received a lot of performance tuning yet. But unless we're doing something dramatically inefficient there, it will still be slow after optimization. So we should consider to either stream the page output (as opposed to rendering it all in one go) or to perform the text highlighting client-side.

The time spent in the browser seems to be linear to the number of tags: when contained in a single preformatted tag, the page is rendered in a fraction of a second, but as soon as I replace the newline characters by <br> tags, performance takes a big hit. Without style sheets, the browser takes ~3s to load a page with 20k lines of text instead of ~10s, so CSS is a factor, but the issue likely cannot be fixed with just CSS tweaks.

You can view the text while the browser is doing its work, so the user doesn't have to wait for the browser to become idle. But having the document grow while scrolling through it is not a great user experience.

As a workaround, I'm going to change the code to serve long files as raw text in an <iframe>.