How can I embed a complete TTF font without subsetting?

[ndbroadbent]

I found some old threads on the Google Group:

• Are font subsets embedded by default? - https://groups.google.com/g/prawn-ruby/c/MsOrVAMt9mQ/m/3sUuTCIaDtkJ
• Disabling font subsetting so that the full font gets embedded - https://groups.google.com/g/prawn-ruby/c/_N4EFJC-CV0

Also a related discussion:

• Embedded fonts in PDF? #1241

I would like to re-open the discussion about embedding fonts without subsetting. I need to embed the full TTF font for some custom AcroForm support that I'm working on, so that text fields support other character sets (e.g. cyrillic, hebrew, chinese). The editable form fields need to support custom user input after the document has rendered, since they might type in any other characters.

I also looked at the TTFunk and prawn code to see if I could figure out how to disable embedding, but it's quite difficult to understand how it works. I made a little bit of progress in the #embed method in lib/prawn/font/ttf.rb:

Instead of font_content = @subsets[subset].encode, I called font_content = @subsets[subset].original.contents.read. And instead of font = TTFunk::File.new(font_content) I tried font = @subsets[subset].original.

I struggled to figure out the rest of that method (cmap, widths, ranges, etc.) and broke everything:

    RuntimeError:
       Error running pdfbox command to generate page images: output: , error: Oct 13, 2023 2:55:40 PM org.apache.pdfbox.pdmodel.font.PDTrueTypeFont <init>
       WARNING: Could not read embedded TTF for font HelveticaWorld-Regular
       java.io.EOFException
       	at org.apache.fontbox.ttf.MemoryTTFDataStream.readSignedShort(MemoryTTFDataStream.java:138)
       	at org.apache.fontbox.ttf.TTFDataStream.read32Fixed(TTFDataStream.java:48)
       	at org.apache.fontbox.ttf.TTFParser.parse(TTFParser.java:136)
       	at org.apache.fontbox.ttf.TTFParser.parse(TTFParser.java:110)
       	at org.apache.pdfbox.pdmodel.font.PDTrueTypeFont.<init>(PDTrueTypeFont.java:199)
       	at org.apache.pdfbox.pdmodel.font.PDFontFactory.createFont(PDFontFactory.java:89)
       	at org.apache.pdfbox.pdmodel.PDResources.getFont(PDResources.java:146)
       	at org.apache.pdfbox.pdmodel.interactive.form.AppearanceGeneratorHelper.validateAndEnsureAcroFormResources(AppearanceGeneratorHelper.java:157)
       	at org.apache.pdfbox.pdmodel.interactive.form.AppearanceGeneratorHelper.<init>(AppearanceGeneratorHelper.java:113)
       	at org.apache.pdfbox.pdmodel.interactive.form.PDTextField.constructAppearances(PDTextField.java:263)
       	at org.apache.pdfbox.pdmodel.interactive.form.PDAcroForm.refreshAppearances(PDAcroForm.java:331)
       	at org.apache.pdfbox.pdmodel.fixup.processor.AcroFormGenerateAppearancesProcessor.process(AcroFormGenerateAppearancesProcessor.java:54)
       	at org.apache.pdfbox.pdmodel.fixup.AcroFormDefaultFixup.apply(AcroFormDefaultFixup.java:56)
       	at org.apache.pdfbox.pdmodel.PDDocumentCatalog.getAcroForm(PDDocumentCatalog.java:132)
       	at org.apache.pdfbox.pdmodel.PDDocumentCatalog.getAcroForm(PDDocumentCatalog.java:113)
       	at org.apache.pdfbox.tools.PDFToImage.main(PDFToImage.java:213)
       	at org.apache.pdfbox.tools.PDFBox.main(PDFBox.java:89)

       Oct 13, 2023 2:55:40 PM org.apache.pdfbox.pdmodel.font.PDTrueTypeFont <init>
       WARNING: Using fallback font 'Helvetica' for 'HelveticaWorld-Regular'
       Oct 13, 2023 2:55:40 PM org.apache.fontbox.ttf.TTFParser parse
       WARNING: Skip table '' which goes past the file size; offset: 65902, size: 24248322, font size: 656444
       Oct 13, 2023 2:55:40 PM org.apache.fontbox.ttf.TTFParser parse
       WARNING: Skip table 's�' which goes past the file size; offset: 43188227, size: 43254887, font size: 656444
       Oct 13, 2023 2:55:40 PM org.apache.fontbox.ttf.TTFParser parse
       WARNING: Skip table 'h' which goes past the file size; offset: 74908791, size: 66680, font size: 656444
       Oct 13, 2023 2:55:40 PM org.apache.fontbox.ttf.TTFParser parse
       WARNING: Skip table 'x' which goes past the file size; offset: 66682, size: 75300867, font size: 656444
       Oct 13, 2023 2:55:40 PM org.apache.fontbox.ttf.TTFParser parse
       WARNING: Skip table '~�' which goes past the file size; offset: 75563011, size: 75630168, font size: 656444
       Oct 13, 2023 2:55:40 PM org.apache.fontbox.ttf.TTFParser parse
       WARNING: Skip table 'Y' which goes past the file size; offset: 108332777, size: 132842, font size: 656444
       Oct 13, 2023 2:55:40 PM org.apache.fontbox.ttf.TTFParser parse
       WARNING: Skip table '
        ' which goes past the file size; offset: 1634885986, size: 2122617, font size: 656444
       Oct 13, 2023 2:55:40 PM org.apache.fontbox.ttf.TTFParser parse
       WARNING: Skip table 'rlH' which goes past the file size; offset: 6449253, size: 1651638382, font size: 656444

This message from Gregory Brown is very interesting:

I wish I could offer more in the way of support here, but when I wrote
TTFunk, it didn't have subsetting at all. Jamis Buck added all of the
subsetting stuff later, but even that was five years ago.

Maybe I can go back to a much earlier comment and see what the code looked like before subsetting was added.

Any tips would be greatly appreciated! And I think it would be very helpful if there was an option to disable subsetting. Both for this AcroForm use case, and also for the other people who mentioned issues with combining lots of PDFs together.

---

P.S. I asked ChatGPT, but I think it might have pointed me in the wrong direction and made up some false information.

Me: How can I embed a font without subsetting using the Prawn ruby library? I need this for custom AcroForm support.

ChatGPT: To embed a font in its entirety without subsetting using the Prawn Ruby library, you can use the subset option and set it to false in the font_families hash:

Prawn::Document.generate('example.pdf') do |pdf|
  pdf.font_families.update(
    "YourFont" => {
      normal: { file: "path/to/YourFont.ttf", font: "YourFontName", subset: false }
    }
  )

  pdf.font "YourFont"
  pdf.text "Hello, world!"
end

Replace "path/to/YourFont.ttf" with the actual path to your TTF or OTF font file, and "YourFontName" with the PostScript name of the font, if necessary.

This should embed the font without subsetting, making all characters available for AcroForms.

Is this something that Prawn used to support?

[ndbroadbent]

Another thing to note is that we were getting this error from Prawn when we called document.formatted_text_box to generate our appearance stream:

Your document includes text that's not compatible with the Windows-1252 character set.
If you need full UTF-8 support, use TTF fonts instead of PDF's built-in fonts.

However, we were previously just rendering the text directly into the stream, and it was actually rendering fine in Chrome and Preview. I guess most PDF readers on modern operating systems can now render UTF-8 text (e.g. cyrillic characters), even if we use one of the default 14 PDF fonts and don't properly embed a TTF font.

So an easier workaround might be to just patch Prawn's #normalize_encoding method to ignore this error and just return the original text. I guess it won't be a valid PDF file but it seems like they render fine in most PDF readers anyway.

EDIT: Nevermind, that approach isn't working. We're getting errors from PDFBox (converting PDFs to images for tests), and it's showing garbled text in Preview:

[pointlessone]

On subsetting. At the moment there's no way to disable it.

ChatGPT completely made that up but I kinda like the API. You are on the right track but that's not all you need. Font embedding is kinda involved in PDF. You also need to produce a correct font descriptor, not just dump the font file. I've been away from Prawn for a while so can't give exact pointer but it's most likely somewhere in pdf-core.

On the message. I think it's misleading in your case. It probably means that your font (and fallback fonts if any) have no glyphs defined for some of the characters in your text. It specifically mentions Windows-1252 because that's the extent of what is supposed to be provided by the default fonts.

[ndbroadbent]

Hi @pointlessone, thanks for your reply! It looks like I will have to dig in a bit further and see if I can produce a correct font descriptor for the whole font file. I couldn't figure out how it tracks which characters are used in order to build the subset, so that will take some more research.

I was also just wondering if you might be available for any contract work? It would be very helpful if we were able to pay for some of your time and have a call, or perhaps sponsor some feature development and bug fixes. Or if not, do you know any other Prawn contributors who might be interested? I would also be interested in sponsoring the project in general, perhaps using a service such as https://opencollective.com.

[pointlessone]

• In Prawn a font is actually an object that tracks glyph usage. Every time text is rendered this object makes note of the glyphs requested (see TTFunk::SubsetCollection#use). Later this object is used to produce a subset font (see Prawn::Fonts::TTF#embed for an example).
• It's very generous of you. At the moment I'm not available. I also have no knowledge of anyone else's availability. I certainly will not oppose contributions from others based on whether that work was sponsored or not. However, I make no promise accepting any and all contributions based on their sponsorship status either. I will try reviewing them as quickly as possible.