Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft: Update core assignment algorithm in benchexec/resources.py #892

Draft
wants to merge 100 commits into
base: main
Choose a base branch
from

Commits on Jan 13, 2023

  1. Assigmnment algorithm redesign

    Charlie committed Jan 13, 2023
    Configuration menu
    Copy the full SHA
    3bee91d View commit details
    Browse the repository at this point in the history

Commits on Jan 16, 2023

  1. Added spreading of runs to assigment algorithm

    Charlie committed Jan 16, 2023
    Configuration menu
    Copy the full SHA
    327d882 View commit details
    Browse the repository at this point in the history

Commits on Feb 2, 2023

  1. Configuration menu
    Copy the full SHA
    30fe0db View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2023

  1. Reformatting

    CGall42 committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    6ee6ef3 View commit details
    Browse the repository at this point in the history
  2. Fixed copyright header

    CGall42 committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    bb281b0 View commit details
    Browse the repository at this point in the history
  3. Comments updated

    CGall42 committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    f540424 View commit details
    Browse the repository at this point in the history
  4. Added root hierarchy level

    This method adds a "root" hierarchy level, if the system topology doesnt have one. Necessary for iterating through the whole topology.
    CGall42 committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    59c6e7d View commit details
    Browse the repository at this point in the history
  5. Fix distribution algorithm

    now chooses the right starting core for the next thread
    CGall42 committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    52fec71 View commit details
    Browse the repository at this point in the history
  6. Various improvements

    CGall42 committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    e2c2a2f View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    fcf7e34 View commit details
    Browse the repository at this point in the history
  8. fix imports

    CGall42 committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    ce795b6 View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2023

  1. Configuration menu
    Copy the full SHA
    7f06e78 View commit details
    Browse the repository at this point in the history
  2. Refactoring

    CGall42 committed Feb 8, 2023
    Configuration menu
    Copy the full SHA
    869554e View commit details
    Browse the repository at this point in the history
  3. Refactoring

    CGall42 committed Feb 8, 2023
    Configuration menu
    Copy the full SHA
    212e25d View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    fab1fca View commit details
    Browse the repository at this point in the history

Commits on Feb 17, 2023

  1. Comments cleanup

    CGall42 committed Feb 17, 2023
    Configuration menu
    Copy the full SHA
    8be832e View commit details
    Browse the repository at this point in the history

Commits on Feb 20, 2023

  1. Configuration menu
    Copy the full SHA
    c69d56d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    45e5446 View commit details
    Browse the repository at this point in the history
  3. fixed variable names

    CGall42 committed Feb 20, 2023
    Configuration menu
    Copy the full SHA
    7257975 View commit details
    Browse the repository at this point in the history
  4. added filter for slow cores

    CGall42 committed Feb 20, 2023
    Configuration menu
    Copy the full SHA
    7b06ee8 View commit details
    Browse the repository at this point in the history
  5. comments edited

    CGall42 committed Feb 20, 2023
    Configuration menu
    Copy the full SHA
    8d07ad8 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    9a0a940 View commit details
    Browse the repository at this point in the history

Commits on Feb 21, 2023

  1. Configuration menu
    Copy the full SHA
    db390c2 View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2023

  1. Configuration menu
    Copy the full SHA
    e2c1e9c View commit details
    Browse the repository at this point in the history

Commits on Apr 18, 2023

  1. Refactoring

    CGall42 committed Apr 18, 2023
    Configuration menu
    Copy the full SHA
    8cce256 View commit details
    Browse the repository at this point in the history

Commits on Apr 20, 2023

  1. Configuration menu
    Copy the full SHA
    7eacff1 View commit details
    Browse the repository at this point in the history

Commits on Apr 21, 2023

  1. Style fixes

    CGall42 committed Apr 21, 2023
    Configuration menu
    Copy the full SHA
    ecc82cc View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2023

  1. Configuration menu
    Copy the full SHA
    943aaeb View commit details
    Browse the repository at this point in the history
  2. Edited sorting function

    CGall42 committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    5af1a05 View commit details
    Browse the repository at this point in the history

Commits on May 2, 2023

  1. fixed new sorting function

    CGall42 committed May 2, 2023
    Configuration menu
    Copy the full SHA
    81e1cb8 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    12d98e6 View commit details
    Browse the repository at this point in the history

Commits on May 4, 2023

  1. Style fixes

    CGall42 committed May 4, 2023
    Configuration menu
    Copy the full SHA
    de87686 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ec4d37f View commit details
    Browse the repository at this point in the history

Commits on May 9, 2023

  1. Configuration menu
    Copy the full SHA
    a7128fd View commit details
    Browse the repository at this point in the history

Commits on May 16, 2023

  1. Documentation added

    CGall42 committed May 16, 2023
    Configuration menu
    Copy the full SHA
    1eee161 View commit details
    Browse the repository at this point in the history

Commits on May 26, 2023

  1. Configuration menu
    Copy the full SHA
    0678ac6 View commit details
    Browse the repository at this point in the history

Commits on May 30, 2023

  1. Refactoring get_sub_unit_dict

    CGall42 committed May 30, 2023
    Configuration menu
    Copy the full SHA
    91eb8e1 View commit details
    Browse the repository at this point in the history
  2. Fix

    CGall42 committed May 30, 2023
    Configuration menu
    Copy the full SHA
    9133f1a View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    cb63077 View commit details
    Browse the repository at this point in the history
  4. More whitespace cleanup

    CGall42 committed May 30, 2023
    Configuration menu
    Copy the full SHA
    199092e View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    4dccc5c View commit details
    Browse the repository at this point in the history

Commits on May 31, 2023

  1. Configuration menu
    Copy the full SHA
    d4e9cd4 View commit details
    Browse the repository at this point in the history
  2. Type notations deleted

    CGall42 committed May 31, 2023
    3 Configuration menu
    Copy the full SHA
    c90ba59 View commit details
    Browse the repository at this point in the history

Commits on Jun 1, 2023

  1. Added type hints

    CGall42 committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    7fa011d View commit details
    Browse the repository at this point in the history
  2. Type Hint & Formating Fixes

    CGall42 committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    b0a9077 View commit details
    Browse the repository at this point in the history
  3. Added debug information

    CGall42 committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    2848667 View commit details
    Browse the repository at this point in the history
  4. More typing types

    CGall42 committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    fcbddda View commit details
    Browse the repository at this point in the history

Commits on Jun 2, 2023

  1. Configuration menu
    Copy the full SHA
    fbf8f5e View commit details
    Browse the repository at this point in the history
  2. Constructor type hint fix

    CGall42 committed Jun 2, 2023
    Configuration menu
    Copy the full SHA
    25f7831 View commit details
    Browse the repository at this point in the history

Commits on Jun 15, 2023

  1. Configuration menu
    Copy the full SHA
    4b5c084 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    fc342a7 View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2023

  1. Configuration menu
    Copy the full SHA
    b9046d9 View commit details
    Browse the repository at this point in the history
  2. Refactoring get_closest_nodes

    CGall42 committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    8dd1b26 View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2023

  1. Configuration menu
    Copy the full SHA
    0689df5 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2023

  1. Configuration menu
    Copy the full SHA
    54c8071 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b125edc View commit details
    Browse the repository at this point in the history
  3. Docstrings added

    CGall42 committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    60a33c1 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    02bf24e View commit details
    Browse the repository at this point in the history
  5. Comment edit

    CGall42 committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    6cd17ea View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2023

  1. Style Fixes

    CGall42 committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    a136fec View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2023

  1. Edge Case Loop Fix

    CGall42 authored Jul 6, 2023
    Configuration menu
    Copy the full SHA
    72106ce View commit details
    Browse the repository at this point in the history

Commits on Dec 13, 2023

  1. Configuration menu
    Copy the full SHA
    4ae8420 View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2024

  1. Configuration menu
    Copy the full SHA
    4cf26e0 View commit details
    Browse the repository at this point in the history
  2. Remove type checks, Python encourages duck typing

    Also turn value checks into asserts,
    as this is an internal method.
    
    This fixes complaints from Ruff.
    PhilippWendler committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    2f3f964 View commit details
    Browse the repository at this point in the history

Commits on Feb 14, 2024

  1. Add some assertions about structure and content of hierarchy_levels

    This helps us to be more convinced that the data structures
    created by the unit tests are the same as in real use.
    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    53702f9 View commit details
    Browse the repository at this point in the history
  2. Refactoring: Don't pass siblings_of_core around

    This is redundant because hierarchy_levels[0] is the same
    and there is only effort to keep the two in sync.
    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    d045bc5 View commit details
    Browse the repository at this point in the history
  3. Simplify creation of hierarchy levels

    - First create the final list of hierarchy levels, then create the
      VirtualCore objects.
      Previously one level was potentially created afterwards and needed
      adjustments to the VirtualCore objects.
    
    - Always insert root level and rely on the filter that we use anyway
      to remove it if necessary. This is the same strategy that we use for
      the siblings_of_core level and thus easier to follow.
    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    a00ede6 View commit details
    Browse the repository at this point in the history
  4. Refactoring: Move creation of VirtualCores into get_cpu_cores_per_run

    All the information in these instances is also present in
    hierarchy_levels, it is just a more convenient way to access it.
    Creating it locally makes it easier to write tests.
    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    1e5c166 View commit details
    Browse the repository at this point in the history
  5. Restore tests for core allocation from main branch

    Because the tests were changed significantly together with the
    implementation of the core allocation in this branch,
    it is hard to see whether the tests still test the same things.
    But we want regression tests that ensure that the new allocation
    behaves in the same way.
    So we restore the old tests and keep the new tests in a separate file
    until we are sure that there are no regressions,
    and then we can unify the test suites.
    
    For now the old tests fail due to API changes,
    this will be adapted next.
    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    b9e1f5d View commit details
    Browse the repository at this point in the history
  6. Perform technical adjustments to old test suite for core allocation

    This makes most of the tests succeed, but some still fail.
    Potentially some further adjustments are needed,
    but there is also one regression
    (https://github.com/sosy-lab/benchexec/pull/892/files#r1229136970).
    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    ce10795 View commit details
    Browse the repository at this point in the history
  7. Remove a test that is no longer relevant

    The new get_cpu_distribution method has no information about
    partial physical cores anymore,
    this is checked outside of it.
    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    ef13d2a View commit details
    Browse the repository at this point in the history
  8. Temporarily disable some tests that need to be investigated or fixed

    It is better to have CI green to be able to notice further regressions.
    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    deae4ac View commit details
    Browse the repository at this point in the history
  9. Fix crash for machines with a single NUMA node in get_closest_nodes

    Also add unit tests for this function.
    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    fd58f75 View commit details
    Browse the repository at this point in the history
  10. remove irrelevant code

    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    f657e9a View commit details
    Browse the repository at this point in the history
  11. Simplify get_closest_nodes

    Also do not use "raise Exception", use assert to encode coding
    assumptions.
    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    8b55c58 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    d0a558b View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    64aa7e5 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    bd1f11a View commit details
    Browse the repository at this point in the history
  15. Simplifications

    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    46a8d42 View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    490d20a View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    5e299c3 View commit details
    Browse the repository at this point in the history
  18. Use standard string format for logging, not f-strings

    Only the standard format is lazy.
    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    3a46391 View commit details
    Browse the repository at this point in the history
  19. remove unused code

    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    316a64e View commit details
    Browse the repository at this point in the history
  20. Refactor get_generic_mapping function

    - Function name starting with "read" to indicate it reads from kernel.
    - Parameters in better order.
    - Identifier naming according to Python standard.
    - Actually use generic identifiers in a generic function
      and not names that are specific to one use case.
    - Also replace all trivial callers with a single function.
    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    b952ece View commit details
    Browse the repository at this point in the history
  21. Improve frequency_filter function

    - Crucial constants should be present only once, documented,
      and defined in a central place.
    - Reading from the system and logic should be separate
      such that the latter is testable.
    - For reading from the system we can use an existing helper method.
    - Add tests.
    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    70ab415 View commit details
    Browse the repository at this point in the history
  22. Avoid uses of defaultdict in resources.py

    Uses of plain dicts may catch errors in callers earlier.
    Furthermore, some of the functions even returned a defaultdict
    in some cases and a plain dict in other cases.
    The return type should be consistent.
    
    With dict.setdefault() the use of a plain dict
    is almost as convenient as a defaultdict.
    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    665bf2c View commit details
    Browse the repository at this point in the history
  23. Refactoring: Remove allCpus parameter from check_distribution_feasibi…

    …lity
    
    It is not really necessary.
    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    81f2b35 View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    289b780 View commit details
    Browse the repository at this point in the history
  25. Fix broken check for missing HT sibling cores

    We always want the user to allow us to use entire physical cores.
    This check was broken, because forbidden sibling cores
    were already removed from the data structure before the check.
    Furthermore, cores forbidden via cgroups and via the
    --allowedCores parameter were treated somehow differently,
    but the effect should be exactly the same.
    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    52afd65 View commit details
    Browse the repository at this point in the history
  26. Restructure reading of siblings information

    So far we read the information about the hyperthreading hierarchy level
    differently from the other levels.
    This made the code more difficult to understand,
    and the way how the ids in the hierarchy_levels[0] dict were chosen
    differed from the other levels.
    But we can also read this information in the same way as for the other
    levels, so let's do this.
    
    We still also need to use the previous way of reading all siblings from
    a given list of cores, but we can also simplify that
    and the separation of concerns still provides
    an understandability benefit.
    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    9b5ad74 View commit details
    Browse the repository at this point in the history
  27. Support arbitrary many cache hierarchy levels for core allocation

    The allocation algorithm already supports an arbitrary number of levels,
    so we can future proof the allocation
    and read all information about cache levels
    that the kernel provides.
    
    We can also use the assumption that caches are named the same
    across all cores, and read the cache names only once
    instead of separately for every core.
    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    4a64142 View commit details
    Browse the repository at this point in the history
  28. Refactor get_sub_unit_dict

    This method actually has nothing to do with "sub" units (children),
    it just takes a set of cores and a level and groups the cores
    as appropriate for the level.
    So the names should reflect that.
    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    a949fda View commit details
    Browse the repository at this point in the history
  29. small fixes for CI

    PhilippWendler committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    12fcb40 View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2024

  1. Configuration menu
    Copy the full SHA
    2404d62 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2bc4d20 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. Configuration menu
    Copy the full SHA
    9636ee0 View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2024

  1. added more flexible way to to generate layer data for tests

    To allow easier generation of new tests (where we ideally can automatically generate tests for a large number
    of (also weird) CPU configurations), it's desireable to be able to specify arbitrary layers more or less directly, without
    having to create new test classes.
    
    The final goal is to be able to generate machine configurations given a single argument (or two arguments: layer configuration as a list
    and total core count), so we can utilize pytest to write easily maintainable test cases.
    schroeding committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    2cc163b View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. remove redundant sorting and removal of duplicate layers

    As the new method already keeps the layers in the correct order and doesn't create duplicate layers,
    we can remove those parts from the code
    schroeding committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    d0d01cd View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. replace layer generation code

    After showing that the new layer generation code is equivalent to the existing one, we can
    remove the old code with the hardcoded layers and use the new code.
    schroeding committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    cab9c90 View commit details
    Browse the repository at this point in the history

Commits on Oct 6, 2024

  1. fix unnecessary import

    schroeding committed Oct 6, 2024
    Configuration menu
    Copy the full SHA
    4073a2b View commit details
    Browse the repository at this point in the history