Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

env2mfile: Add unit test coverage for dpkg -> Meson, and fix some obvious errors #13722

Merged
merged 7 commits into from
Oct 2, 2024

Conversation

smcv
Copy link
Contributor

@smcv smcv commented Sep 27, 2024

  • env2mfile: Convert MachineInfo into a dataclass

    This will make it easier to unit-test functions that work with a
    MachineInfo, by constructing the expected object in a single call.

  • env2mfile: Split detect_cross_debianlike()

    Separating the part that runs dpkg-architecture from the part that
    interprets its results will make it easier to unit-test the latter.

  • unittests: Test env2mfile's dpkg_architecture_to_machine_info

    This test parses several possible outputs of dpkg-architecture and
    asserts that they produce the expected MachineInfo.

    To avoid depending on suitable cross-tools being installed, use
    unittest.mock to override locate_path with a version that pretends that
    all of the tools we're interested in are in /usr/bin.
    Similarly, use mock environment variables to exercise what happens
    when we have those set.

    The test data used here exercises most variations:

    • big- and little-endianness
    • GNU CPU (x86_64) differing from dpkg CPU (amd64)
    • Linux, kFreeBSD and Hurd
    • special-cased architectures: x86, arm, mips64el, ppc64el

    expected_compilers() intentionally doesn't assume that every compiler
    is gcc (even though they all are, right now), because env2mfile: Generalize handling of architecture-prefixed tools on Debian derivatives #13721 proposes
    adding valac which does not take a gcc suffix.

  • env2mfile: Add a function for mostly-identity mappings with special cases

    This makes the frequent pattern of things like "CPU families are the
    same as GNU CPUs, with a few known exceptions" less verbose.

  • env2mfile: Map Debian GNU/Hurd to system() -> gnu

    As per https://mesonbuild.com/Reference-tables.html, and matching what
    happens when running Meson for a native build on Debian GNU/Hurd.

  • env2mfile: Don't hard-code Debian as always being Linux

    All official Debian release architectures use the Linux kernel, but
    unofficial ports like hurd-i386 and kfreebsd-amd64 use the Hurd and
    FreeBSD kernel, respectively.

    Map Linux to 'linux' and kFreeBSD ports to 'freebsd' as per the
    reference tables in Meson's documentation. For now, use the Debian
    system name such as 'hurd' for anything else (see Cannot detect kernel during native build on Debian GNU/Hurd #13740 for the
    question of whether Hurd should identify its kernel differently).

  • env2mfile: Base cpu on DEB_HOST_GNU_CPU unless DEB_HOST_ARCH is special

    DEB_HOST_ARCH encodes both the CPU family and the OS, so using it to
    get the CPU type gives the wrong answer for non-Linux ports.

    However, DEB_HOST_GNU_CPU gives less detailed information about the
    CPU: it's arm for all 32-bit ARM CPUs, and doesn't distinguish between
    the differing baselines of armel (ARMv5 softfloat) and armhf
    (ARMv7 hardfloat).

    When cross-compiling for x86_64 Linux, this changes the cpu() from
    amd64 to x86_64, which is consistent with the answer we get during
    native builds on that architecture.

    When cross-compiling for ppc64el, this changes the cpu() from
    ppc64el to ppc64, which is a reasonable change but is still not
    consistent with what we see during native builds (which is ppc64le):
    see Inconsistent host_machine.cpu() for Debian ppc64el cross vs. native #13741 for that.

    Resolves: Inconsistent host_machine.cpu() for Debian amd64 cross vs. native #13742


This is not a complete solution for #13740 and #13741, but would be a good starting point for fixing those afterwards.

@smcv smcv changed the title env2mfile: Don't hard-code Debian as always being Linux env2mfile: Add unit test coverage for dpkg -> Meson, and fix some obvious errors Oct 2, 2024
mesonbuild/scripts/env2mfile.py Fixed Show resolved Hide resolved
unittests/internaltests.py Fixed Show resolved Hide resolved
@smcv smcv marked this pull request as ready for review October 2, 2024 13:53
@smcv smcv requested a review from jpakkane as a code owner October 2, 2024 13:53
mesonbuild/scripts/env2mfile.py Outdated Show resolved Hide resolved
Copy link
Member

@dcbaker dcbaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the very thorough testing of this, I've got a few comments below.

mesonbuild/scripts/env2mfile.py Outdated Show resolved Hide resolved
mesonbuild/scripts/env2mfile.py Outdated Show resolved Hide resolved
unittests/internaltests.py Outdated Show resolved Hide resolved
unittests/internaltests.py Outdated Show resolved Hide resolved
unittests/internaltests.py Outdated Show resolved Hide resolved
This will make it easier to unit-test functions that work with a
MachineInfo, by constructing the expected object in a single call.

Signed-off-by: Simon McVittie <[email protected]>
Separating the part that runs dpkg-architecture from the part that
interprets its results will make it easier to unit-test the latter.

Signed-off-by: Simon McVittie <[email protected]>
This test parses several possible outputs of dpkg-architecture and
asserts that they produce the expected MachineInfo.

To avoid depending on suitable cross-tools being installed, use
unittest.mock to override locate_path with a version that pretends that
all of the tools we're interested in are in /usr/bin.
Similarly, use mock environment variables to exercise what happens
when we have those set.

The test data used here exercises most variations:

* big- and little-endianness
* GNU CPU (x86_64) differing from dpkg CPU (amd64)
* Linux, kFreeBSD and Hurd
* special-cased architectures: x86, arm, mips64el, ppc64el

expected_compilers() intentionally doesn't assume that every compiler
is gcc (even though they all are, right now), because mesonbuild#13721 proposes
adding valac which does not take a gcc suffix.

Signed-off-by: Simon McVittie <[email protected]>
…ases

This makes the frequent pattern of things like "CPU families are the
same as GNU CPUs, with a few known exceptions" less verbose.

Signed-off-by: Simon McVittie <[email protected]>
As per <https://mesonbuild.com/Reference-tables.html>, and matching what
happens when running Meson for a native build on Debian GNU/Hurd.

Signed-off-by: Simon McVittie <[email protected]>
All official Debian release architectures use the Linux kernel, but
unofficial ports like hurd-i386 and kfreebsd-amd64 use the Hurd and
FreeBSD kernel, respectively.

Map Linux to 'linux' and kFreeBSD ports to 'freebsd' as per the
reference tables in Meson's documentation. For now, use the Debian
system name such as 'hurd' for anything else (see mesonbuild#13740 for the
question of whether Hurd should identify its kernel differently).

Signed-off-by: Simon McVittie <[email protected]>
`DEB_HOST_ARCH` encodes both the CPU family and the OS, so using it to
get the CPU type gives the wrong answer for non-Linux ports.

However, `DEB_HOST_GNU_CPU` gives less detailed information about the
CPU: it's `arm` for all 32-bit ARM CPUs, and doesn't distinguish between
the differing baselines of `armel` (ARMv5 softfloat) and `armhf`
(ARMv7 hardfloat).

When cross-compiling for x86_64 Linux, this changes the `cpu()` from
`amd64` to `x86_64`, which is consistent with the answer we get during
native builds on that architecture.

When cross-compiling for `ppc64el`, this changes the `cpu()` from
`ppc64el` to `ppc64`, which is a reasonable change but is still not
consistent with what we see during native builds (which is `ppc64le`):
see mesonbuild#13741 for that.

Resolves: mesonbuild#13742
Signed-off-by: Simon McVittie <[email protected]>
Copy link
Member

@dcbaker dcbaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks good to me.

@jpakkane jpakkane merged commit 9d0de83 into mesonbuild:master Oct 2, 2024
33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Inconsistent host_machine.cpu() for Debian amd64 cross vs. native
3 participants