Fix regression when using both libraries #16845

Job	Run time
Arch Linux	19m 16s
CUDA (on Arch)	7m 20s
Fedora	18m 51s
Arch / PyPy	20m 5s
Gentoo	20m 57s
OpenSUSE	15m 36s
Ubuntu Bionic	14m 9s
Ubuntu Rolling (gcc, g++)	18m 20s
Ubuntu Rolling (--unity=on -Ddefault_library=static, --no-unittests, gcc, g++)	13m 3s
Ubuntu Rolling (-Ddefault_library=both, --no-unittests, gcc, g++)	13m 23s
Ubuntu Rolling (clang, clang++)	20m 57s
Ubuntu Rolling (--unity=on, --no-unittests, clang, clang++)	15m 20s
Ubuntu Rolling (--cross ubuntu-armhf.json --cross linux-mingw-w64-64bit.json, 0, gcc, g++)	13m 19s
Ubuntu Rolling (--cross ubuntu-armhf.json --cross linux-mingw-w64-64bit.json, --unity=on, gcc, g++)	13m 38s
	3h 44m 14s

Provide feedback