Map Apple's ftxvalidator checks for which we still lack corresponding Font Bakery checks

[thundernixon]

This issue was originally called "macOSX installation issue snags", but in the meantime we deprecated the ftxvalidator wrapper-check, so now we have a different scope for this task. Please, read the issue thread for more details.

-- @felipesanches, Oct 17, 2021

---

Observed behaviour

I'm setting up a fresh computer that is macOS Mojave, 10.14.4, and having some issues.

The macOS installation instructions are a bit hard to follow, because there are little things that don't work well, and I don't know how to check whether it worked in the end.

In zshell (zsh), it seemed to fail:

 cd ~/Downloads/ ;
hdiutil attach osxfonttools.dmg ;
mkdir -p /tmp/ftx ;
cd /tmp/ftx ;
cp "/Volumes/OS X Font Tools/OS X Font Tools.pkg" . ;
xar -xf "OS X Font Tools.pkg" ;
cd fontTools.pkg/ ;
cat Payload | gunzip -dc | cpio -i ;
sudo mv ftx* /usr/local/bin/ ;
hdiutil: attach failed - No such file or directory
cp: /Volumes/OS X Font Tools/OS X Font Tools.pkg: No such file or directory
Error opening xar archive: OS X Font Tools.pkg
cd: no such file or directory: fontTools.pkg/
cat: Payload: No such file or directory
gunzip: (stdin): unexpected end of file
0 blocks
zsh: no matches found: ftx*

I then tried to install from Apple Developer. Font Tools for Xcode 9 seemed to work fine. However, OS X Font Tools gives this warning:

This package is incompatible with this version of macOS and may fail to install.

In bash, the terminal instructions seemed to work better, but I still don't know how to test whether it actually installed. The ftx command just returns ftx: command not found.

Expected behaviour

I should be able to follow the instructions exactly, without scary-sounding warnings, then have an immediate way to confirm that they worked. For onboarding less-technical designers, this is even more important.

Resources and exact process needed to replicate

To replicate, one could set up a new user account on a mac, then follow the FB installation instructions.

---

Small Update: I haven't installed the OS X Font Tools package yet, and I think that I still have the tools I need for FontBakery. However, I will double-check that no FontBakery checks are skipped because of this decision, and report back if I do eventually install that.

[thundernixon]

Oh, and should I "Install Anyway" in that installation package, or cancel?

What checks use this specific software?

[felipesanches]

Oh, and should I "Install Anyway" in that installation package, or cancel?

What checks use this specific software?

"Apple provides various font utilities, and ftxvalidator is especially useful as it runs the same checks that are run for users when they install a font using Font Book. "

That's all I know. I would love to learn exactly what those checks are, but the tool is proprietary. If we knew what it does, I would gladly reimplement it in pure python and "cut the crap", resulting in a smoother user experience. But as far as I can tell we don't know yet what it does.

[thundernixon]

I see; there's a specific ftxvalidator check.

And this is specifically in OS X Font Tools, not in Font Tools for Xcode 9?

[felipesanches]

com.google.fonts/check/ftxvalidator runs the command-line ftxvalidator tool on MacOS and that's all. It is a wrapper just the the one we have around MSFontVal as well.

[felipesanches]

And this is specifically in OS X Font Tools, not in Font Tools for Xcode 9?

I am not sure. If I recall correctly, I ran these install instructions on my Mac once and it worked, so I never looked at it again.

[felipesanches]

I agree with you that this is ugly and scary and we should have something much better in place.

[felipesanches]

This contains an example of the ftxvalidator output:
http://www.fontgeek.net/blog/?p=168

But it is not an extensive example. It does not show much.

[felipesanches]

This one goes into a bit more detail:
https://blog.idrsolutions.com/2013/09/using-os-xs-font-validator/

[thundernixon]

@felipesanches thanks for the info! Those are helpful to read.

I suppose that the installation doc could say:

To test whether the OSX tools have successfully installed, run ftxinstalledfonts -f in your command line. It should provide a list of currently installed fonts.

[vv-monsalve]

Running gftools on many QA processes I've received repeatedly this message:
⚠ WARN ftxvalidator is not available.

After a first online search (of which I don't have the rout to sources now) there were mentioned as probable causes: OS or Apple Font Tools old versions. I'm working on Mojave 10.14.5, so I dismissed it as a reason. Then, I installed new versions of Font Tools (before finding this here). Now I even have Font Tools Xcode 11 Beta 4 but I still get the same message.

I've also run the ftxinstalledfonts -f to check and yes it appears as not found:
-bash: ftxinstalledfonts: command not found

What am I missing?

[sponcey]

I also am receiving the following error after successfully installing the Font Tools Xcode 11 Beta 4 package:

-bash: ftxvalidator: command not found

I checked my /usr/local/bin folder, and ftxvalidator is not there :( I am running on OS 10.14.6, with the latest version of Xcode 10.3.

[chrissimpkins]

@sponcey Spencer, shoot me an email. I have a workaround until this is fixed.

[kontur]

Since there is nowhere any real documentation on how to install this really, this is what worked for me (with the caveat that the actual binary does not seem to work, though, as per ##2508):

-removed install instructions for old version-

[chrissimpkins]

You likely need an updated framework and ftxvalidator executable. Try these instructions with the current Font Tools for XCode 11 release: #2691

[kontur]

Aha! Thanks @chrissimpkins

More errors though with the XCode 11 binaries:

dyld: lazy symbol binding failed: Symbol not found: __ZN3ftx8FTStringC1EPKcz
  Referenced from: /Users/johannes/bin/ftxvalidator
  Expected in: /Library/Frameworks/FontToolbox.framework/Versions/A/FontToolbox

dyld: Symbol not found: __ZN3ftx8FTStringC1EPKcz
  Referenced from: /Users/johannes/bin/ftxvalidator
  Expected in: /Library/Frameworks/FontToolbox.framework/Versions/A/FontToolbox

Abort trap: 6

Any clue what that is about?

[davelab6]

Given no one could easily install the apple font linter, and apple didn't keep it updated to new macOS releases, I'm ok with #3479

However, as Felipe suggests above, there's a bunch of checks described in the verbose check results that we can map to current fb checks - or file issues to describe and implement the dogmas - such that a "fruit profile" could be made.

[twardoch]

Apple does keep it up-to-date. The most recent version is Font Tools for Xcode 13, released on 20 September 2021. The new ftxvalidator runs fine on my macOS 11.5.1:

ftxvalidator build 208, FontToolbox.framework build 353
Usage:  ftxvalidator [-hlrRSTvV] [-t testList] [-o outputfile] fontfile
Options:
	-e (--exclude) An optional list of tests to exclude
	-h (--help) Output this message
	-K (--check-build-number) Check that the tool is at least the given version
		`ftxvalidator-K 200` returns -1 if the build number is not 200 or higher
	-l (--list) List available tests
	-o (--output) An optional output file for test results; stdout is used if this parameter is not specified
	-r (--full-report) Generate a full report
	-R (--return-value-results) Use return value to indicate overall results
	-S (--summary-report) Generate a summary report
	-t (--test-list) The tests to execute.  Valid inputs are:
		"all", execute all tests,
		"core", execute the core test set,
		"default", execute the default test set,
		or a list of font test types, separated by commas,
		or a list of test identifiers, separated by commas`
	-T (--table-report) Generate a tabular report
	-v (--verbose) Verbose operation, additional data is output to stderr
	-V (--version) Output the tool's version to stdout
	-W (--suppress-warnings) Suppress warning messages

Return values:
	0 No results
	1 Execution errors
	2 No execution errors, validator fatal errors found
	3 No execution errors, validator major errors found
	4 No execution errors, validator minor errors found
	5 No execution errors, spec violations found
The return value, if greater than 1, is the lowest value possible.
Return values greater than 1 are only used if --return-value-results is set.

[felipesanches]

there's a bunch of checks described in the verbose check results that we can map to current fb checks - or file issues to describe and implement the dogmas

@twardoch, would you be able to provide such input so that we could figure out the full set of font problems that ftxvalidator supposed checks for?

Once we have this info, I'd probably create a spreadsheet to map which font problems Font Bakery still does not check for.

[HinTak]

FWIW, apple font tool for xcode 15 beta was released two months ago in June 2023.

[HinTak]

I haven't tried it for a while, but past versions of Apple's font tools (xcode 11 was what I tried last I think) run with limited functionality on linux with patched darling : https://github.com/HinTak/ftx-darling

[HinTak]

Fontools for xcode 14 was released last September. I checked the past release dates, it seems that Apple is doing annual Fontools releases in September/October, with a beta in June. So Apple Fontools 15 is likely to be released in the next few weeks, given that there was a 15 beta in June two months ago.

[HinTak]

As I wrote 6 weeks ago, apple font tools for xcode 15 was released on 18th Sept 2023, less than 3 weeks after I wrote. It was preceded 6 days before that by a release candidate on 12th Sept 2023.

[davelab6]

@vv-monsalve showed me this:

That seems like something I really wish fontbakery prevented :)

@simoncozens please can you queue up an effort to make the macOS font validator run with fontbakey when run on macOS, and find out if github runners can be macOS or if we need another runner service to sort this out at scale.

This is related perhaps to #4801 as I'd like to see all relevant checks from other vendor profiles, especially Adobe and Microsoft, be imported in the Google Fonts profile, since users expect the fonts they download from GF to work in those company's apps.

[HinTak]

Github runners are available for both the x86_64 macos and arm macos. The arm macos also contains a x86_64 emulation (rosetta it is called). We build the skia-python wheels bundled with skia with it.

[simoncozens]

Yep, step one here is to set up a GH action to install ftxvalidator. I'm a bit loathe to de-deprecate the ftxvalidator wrapper check; I'd rather (a) make it easy to run ftxvalidator in a CI, and (b) reverse-engineer all the checks and add them to fontbakery.

[HinTak]

Installing apple footstools in GH action is pretty easy technically, not sure about the legal aspect (does it count as redistribution etc). Of course me doing it for fun for myself is a little different from Google offering apple tools to others, legally, too.

[HinTak]

FWIW, I also started a discussion with Apple folks for porting Apple's libValidation (the backend of ftxvalidator) and possibly open-sourcing it via FontVal-RX (https://github.com/FontVal-extras/FontVal-RX) a while ago.

[HinTak]

The quickest way for me personally to set up ftxvalidator in a mac runner for GH action is simply to sync and upload my local darling playground on Linux to one of github's mac runners. Not the neatest, and probably shouldn't be done by a corporation.

[HinTak]

Apple font tools had a 15.1 release in December 2023, and a 16 beta two months ago. So likely there will be a release 16 in the autumn. I have created a new repo for this sort of information:
https://github.com/FontVal-extras/Apple_Font_Tools_Info

There is a _playground repo under the FontVal-extras org for running the apple font tools under github's CI runner tool. That's private . I'll hope to get the 16 beta running under it and see what 16 beta's ftxvalidator says now.

Most new FontVal related things are being moved under that org, except fscaler (the new RX backend), for obvious reasons.

[HinTak]

Both fonttools 15.1 and the 2-month-old 16 beta reports its version as ftxvalidator build 208, FontToolbox.framework build 354 , which is one up from what @twardoch said about build 353 in font tool 13. This info comes from the private _playground repo. I'll just update the readme on the info repo for now.

[simoncozens]

Yep, step one here is to set up a GH action to install ftxvalidator. I'm a bit loathe to de-deprecate the ftxvalidator wrapper check

Especially because in this instance, the problem is a bug in Apple's font parsing. :-)

Setting up ftxvalidator in a CI job is cumbersome because the download source to it is behind a developer paywall. So far I've written a GitHub action which requires you to provide a cookie from your browser, but I don't know how permanent those session ID cookies are, so it's brittle at best.

[HinTak]

It is not a paywall, just requiring a (free) login, as well as seem to check browser / client. I have a simpler github action CI - just uploading what I downloaded. Here is all the versions I have:
https://github.com/FontVal-extras/Apple_Font_Tools_Info - they all run on current mac os, except the 13-year-old 2011 a.k.a . 4 beta 1. Is that a bug in 16 beta? If that's the case, report it so that 16 get fixed in the autumn. If it is earlier - well, that's just older versions...

[simoncozens]

I've already reported the bug (FB14695457).

It turns out the session cookies are not permanent at all (worked yesterday, does not work today), so we have to go through some nasty login automation. Which requires 2FA. Urgh. I'm not sure we can do this. "Uploading what I downloaded" is redistribution of licensed software, which as Google I'm pretty sure we can't do.

[HinTak]

Yes, it requires a free registration/ login, and even then, download is only allowed from only some browser/ client combo's (I can login and look around but not download from a different device, side-by-side). Already said a few comments ago, that me as a private person playing with it in a private repo (https://github.com/FontVal-extras/Apple_Font_Tools_playground) then posting what I learned about their self-reported versions in a public web page (https://github.com/FontVal-extras/Apple_Font_Tools_Info) is fine, but you probably cannot re-distribute the apple package or binary as a corporate - sponsored project, nor offer to run it as a service as any kind of 3rd party. Hence I kept the _playground repo private.