Distrib #370

No code change, just add comments about things being official instead of assumptions. The Gen5 specs is pretty-much finalized, things won't change anymore. Signed-off-by: Brice Goglin <[email protected]>

Check that Linux can add NUMA nodes to x86 CPU information. And check that Linux can annotate x86 AMD topoext NUMA nodes. Signed-off-by: Brice Goglin <[email protected]>

If the workspace clone ever ran on another branch (e.g. in my zbgoglin jobs), git branch returns multiple lines, which causes the 2nd branch name to be ran as a command-line after the only expected line "job-0-tarball.sh <firstbranch>" Use git rev-parse --abbrev-ref HEAD instead. Signed-off-by: Brice Goglin <[email protected]>

This tells the code not to ever merge that group with structurally-identical parent or children. This is useful for Groups implementing new "types" that cannot be backported to stable releases. New types won't be merged by default, but Groups would. Requested by Intel for Die objects. This doesn't break the ABI because the attribute structure has always been calloc'ed, which means this attribute was "0", which matches the default "merge group" behavior. Signed-off-by: Brice Goglin <[email protected]>

Update CPUID.1f x86 test case not to merge Die groups anymore. Hence there's no need to ignore Caches anymore. Signed-off-by: Brice Goglin <[email protected]>

…ile/module types Make them groups. Signed-off-by: Brice Goglin <[email protected]>

…rectories Signed-off-by: Brice Goglin <[email protected]>

I managed to convince Intel that adding another foo_siblings between core_siblings and thread_siblings would break userspace and situation could be even worse if they ever add another intermediate level in the future. So they are finally renaming to filenames whose semantics doesn't depend on intermediate levels: core_cpus and package_cpus. Signed-off-by: Brice Goglin <[email protected]>

Linux 5.3 will have new "die_cpus" and "die_id" sysfs files for upcoming architectures with multiple dies per packages. When the die cpuset is different from the package, add a "Die" group. Don't add it when there's a single Die per package because most CPUs don't want to show a useless additional Die level. We don't want to set the Die level to keep_structure because it would get automerged in L3 caches on CLX, and lstopo displays everything by default anyway. Set the "dont_merge" group flag if HWLOC_DONT_MERGE_DIE_GROUPS is set in the environment, just like in the x86 backend. Signed-off-by: Brice Goglin <[email protected]>

Old kernels exposed two packages on E5v3 in Cluster-on-Die mode because the package core_siblings was wrong. We detected that case when two packages had the same physical_package_id. This was fixed in Linux 3.18, backported in RHEL7. Other important distros use a more recent kernel now. Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

Just like for cores and packages. Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

Fixes commit c1c34a6 Signed-off-by: Brice Goglin <[email protected]>

Otherwise the matrix would be wrong. Further fixes commit c1c34a6 Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

…vailable Hence, we don't have to run both on Linux/x86 anymore, and we don't have to manually tarball the CPUID files. Refs open-mpi#186 Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

Use realpath so that we can change the current directory without breaking the destination relative directory. Signed-off-by: Brice Goglin <[email protected]>

Reported by Intel from the output of klocwork. Signed-off-by: Brice Goglin <[email protected]>

open <6.0.0 has a security vulnerability Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

… environment Enables easier testing of the verbose errors. Signed-off-by: Brice Goglin <[email protected]>

CID #1448189 Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

When we return a distances struct to the user, actually allocate a larger container that points to the internal distances. Might be used later to find the corresponding internal distances, for instance if we ever need to refresh the public distances, or to retrieve name, infos, etc. Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

Not exported to users yet. Useful for better describing distances structure, especially for upcoming matrices for objects with different types. Only "NUMALatency" is currently used, and that's not very useful since the distances structure type and kinds provide the same info. But it will be more useful for identifying specific matrices such as the upcoming NVLink one. Signed-off-by: Brice Goglin <[email protected]>

…es structure. We don't add it to the public distances structure because it would break the ABI. Signed-off-by: Brice Goglin <[email protected]>

Useful for retrieving specific matrices such as "NUMALatency" or "NVLink". Signed-off-by: Brice Goglin <[email protected]>

… distance structure Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

…h the HWLOC_COMPONENTS variable Signed-off-by: Brice Goglin <[email protected]>

…C_COMPONENTS envvar Signed-off-by: Brice Goglin <[email protected]>

It's very incomplete and not interesting to the vast majority of users. And it will be even more outdated soon. Signed-off-by: Brice Goglin <[email protected]>

…ple levels Signed-off-by: Brice Goglin <[email protected]>

…rent types This may be used for upcoming NVLink distances between GPUs and CPUs, or roofline-like distances between PUs/Caches/Memories. Signed-off-by: Brice Goglin <[email protected]>

Use a new "distances2hetero" tag instead of adding hetero-specific attributes to the existing "distances2" tag. The latter would fail to load entirely in pre-2.1 releases, while the former just ignores the remaining of the XML (not an issue since the new tag is always after everything supported in pre-2.1). Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

…pth() anymore Make it work even for Groups etc. Signed-off-by: Brice Goglin <[email protected]>

Includes CID#1449980 Signed-off-by: Brice Goglin <[email protected]>

CID#1445008 Signed-off-by: Brice Goglin <[email protected]>

weight() returns a signed int, so assert is better if we stay signed. CID#1445010 Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

It was bumped again in to 0x20200 in fa636a1, 0x20100 is enough (and matches the main hwloc version) Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

Those were disabled in commit fa0e5d4 in 2009 because mingw32 didn't like parameters with spaces (likely in synthetic strings). That doesn't seem to be an issue on our CI anymore so let's reenable this. Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

For testing compatibility between releases etc. Signed-off-by: Brice Goglin <[email protected]>

To make sure we don't break lstopo/shmem by mistake. The core shmem is already tested in tests/hwloc/shmem. Only enabled on Linux for now, could work everywhere in theory since we detect expected failures on Windows or when the target shmem VM range is busy. Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

Since commit 7d9f952, root_fd is only set when fsroot_path is changed. Otherwise, it's -1. Don't close root_fd when -1 so that we don't propagate errno = EBADF if backend disabling occurs in a load() error path. Signed-off-by: Brice Goglin <[email protected]>

The type will soon be a OR'ed set of phases, we cannot return a const string anymore, and it's not very useful anyway since component types are constants. Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

Replace the single component type with a OR'ed mask of "phases". For each phase, the discover() callback of each relevant backend is invoked with the phase as an argument. Phases may be excluded individually, so that a backend is either entirely or partially disabled. For now, we still don't have any component with multiple phases, they are still GLOBAL, CPU or MISC for now. Signed-off-by: Brice Goglin <[email protected]>

Global (xml and synthetic) CPU (used for OS) +Memory (unused yet, will be used to make sure Memory is added after the CPU hierarchy is ready), +PCI (used for pci and linuxio) +IO (cuda, opencl, etc) Misc (only for Misc devices now, unused yet) +Annotate (unused yet, will be used for adding info to existing objects) +Tweak (for component that want to modify the topology right before returning from load) For now, we still have a single phase for component, but we'll change that soon. Signed-off-by: Brice Goglin <[email protected]>

…lback Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

Either in HWLOC_COMPONENTS=-foo:bar or in hwloc_topology_set_component(topo, "foo:bar", BLACKLIST) Signed-off-by: Brice Goglin <[email protected]>

Split PCI/IO/MISC/ANNOTATE phases Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

Now that the linuxio PCI phase is separated from others, we can use dynamic phase excluding to select between linuxio and pci PCI discovery. Signed-off-by: Brice Goglin <[email protected]>

So that we don't need to pciaccess device struct anymore. And abstract-out to a separate function. Signed-off-by: Brice Goglin <[email protected]>

So that pciaccess names can be added to linuxio pci devices. Signed-off-by: Brice Goglin <[email protected]>

…d fsroot Modified fsroot cannot use the pciaccess backend for getting PCI devices, but it may still annotate sysfs-got devices with vendor/device names later. Hence changing fsroot shouldn't disable pci entirely anymore. However linux tests must re-disable it, otherwise the output would change depending on whether pciaccess is available, and with the version of the pci.ids database. Signed-off-by: Brice Goglin <[email protected]>

If the envvar HWLOC_ANNOTATE_GLOBAL_COMPONENTS=1, global components don't exclude the ANNOTATE phase anymore. Hence, the pci component may add PCI vendor/device strings to objects if the XML generation was performed without pciaccess. Note that it doesn't check whether those names are already available, so they may be duplicated if enabled blindly. Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

It restricts the topology to a single PU and single NUMA. Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

…isions Signed-off-by: Brice Goglin <[email protected]>

"machine:0" is still supported but mostly useless since the custom assembly backend was removed in 2.0. Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

We never checked the parent/child relationship because the number of object in (unrelated) consecutive levels (in depth order) is very often different. However this is really needed in practice. Fixes open-mpi#363 Thanks to Mark Allen for reporting the issue. He had a P9 machine restricted to only 6 cores. And there were also 6 groups for CPU-less GPU memory nodes. Those levels are really unrelated, but the core algorithm decided to place them nearby in the list of depths. In git master, this issue would only occur if we disable the auto-ignoring of these GPU nodes. Signed-off-by: Brice Goglin <[email protected]>

By the way, document the removal of gp_index attributes from XML to avoid spurious failures when the core inserts objects in a different order from some reason. Signed-off-by: Brice Goglin <[email protected]>

length isn't needed, and the missing "%s" caused a warning. Signed-off-by: Brice Goglin <[email protected]>

…ty heuristics 1) Some SLIT tables are buggy for NVDIMMs nodes (asymmetric and wrong). 2) Current Linux kernels (at least up to 5.3) may create conflicting node target/initiators when proximity domains and OS node indexes are different (e.g. when PXM 0 and 1 in CPU0 are nodes 0 and 2 in Linux on dual-CLX platforms in SNC mode). This envvar will allow working around such hardware/software bugs. By default, the envvar is 7, which means SLIT is gathered and (1) and (2) are enabled. Removing bit 0 disables SLIT entirely. Removing bit 1 disables (1). Removing bit 2 disables (2). Signed-off-by: Brice Goglin <[email protected]>

Only disables distance gathering, nothing else is supported there. Signed-off-by: Brice Goglin <[email protected]>

… sysfs directories dev_id is somehow deprecated for exposing the port number. Mostly matters for IB devices. Thanks to Thomas Gruber for the report. Signed-off-by: Brice Goglin <[email protected]>

Some topology may have cores at level N-2 and PU at level N because a unrelated level was placed at N-1 (for instance groups of GPU CPU-less NUMA node on the side). There's no easy way to force PU parents at N-1 and everything lese before. Fix lstopo to handle that case instead of assuming PU parents are always N-1. Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

This partially reverts commit 5d63e48. Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

Commit c9df031 fixed the PU separator but change the I/O separator to 0 too. Fix that case. Signed-off-by: Brice Goglin <[email protected]>

…objects during build Thanks to Eloi Gaudry for the patch. Signed-off-by: Brice Goglin <[email protected]>

Instead of having all of them in the main solution file. Thanks to Eloi Gaudry for the patch. Signed-off-by: Brice Goglin <[email protected]>

Defined with recent VS. Signed-off-by: Brice Goglin <[email protected]>

Thanks to Eloi Gaudry for the patch. We force retarget to an old vs110 for ci.inria.fr. Signed-off-by: Brice Goglin <[email protected]>

Thanks to Eloi Gaudry for the patch. Signed-off-by: Brice Goglin <[email protected]>

Move idea of hwloc-ps to a github issue. Update some comments, add details for command-line build. Thanks to Eloi Gaudry for the suggestion. Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

Will run in the extended nightly tests. Runs only on master on the main repo by default. Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

…-processors Closes open-mpi#368 Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

They are renamed to PREFIX_hwloc_FOO instead of PREFIX_HWLOC_FOO We could fix it but it doesn't matter much (people aren't supposed to use those renamed names anyway) and it could break existing hacks (if anybody actually depends on such renamed name). Thanks to Samuel K. Gutierrez for the report. Signed-off-by: Brice Goglin <[email protected]>

Don't AND(normal, topology_allowed) in the normal (v2) case to avoid hiding internal allowed set bugs. Signed-off-by: Brice Goglin <[email protected]>

In some (old?) corner cases, Linux cpusets may return offline PUs in the allowed sets of cpusets/cgroups. Signed-off-by: Brice Goglin <[email protected]>

…ectory fsroot and cpuid are implemented in tools using environment variables (those debug cases are not in the API since v2). Those backends forced by environment variable override the normal topology thissystem flag that may be set with set_flags() in the API and with --flags or --thissystem in cli tools. One must use the HWLOC_THISSYSTEM envvar to force the this system flag. Implement this automatically in the tools (common helpers). Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: Brice Goglin <[email protected]>

Signed-off-by: ndenoyelle <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Distrib #370

Distrib #370

Commits on Oct 22, 2019

Distrib #370

Are you sure you want to change the base?

Distrib #370

Commits on Oct 22, 2019