Adding Combination Operators `Zip`, `CombineLatest`, `WithLatestFrom` #330

mdemoret-nv · 2023-06-03T00:12:44Z

Description

This adds a few new operators for combining multiple streams into one. For example, the code below illustrates using the Zip operator

auto source1 = std::make_shared<node::TestSource<int>>();
auto source2 = std::make_shared<node::TestSource<float>>();

auto zip = std::make_shared<node::Zip<int, float>>();

auto sink = std::make_shared<node::TestSink<std::tuple<int, float>>>();

mrc::make_edge(*source1, *zip->get_sink<0>());
mrc::make_edge(*source2, *zip->get_sink<1>());
mrc::make_edge(*zip, *sink);

source1->run();
source2->run();

sink->run();

EXPECT_EQ(sink->get_values(),
            (std::vector<std::tuple<int, float>>{
                std::tuple<int, float>{0, 0},
                std::tuple<int, float>{1, 1},
                std::tuple<int, float>{2, 2},
            }));

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

This PR contains the changes from running the `ci/release/update-version.sh` script for a new release Authors: - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: nv-morpheus#357

Forward-merge branch-23.07 to branch-23.11

* Adopt patched pybind11 * Update versions of boost & libhwloc to work-around clang CI issue * A test for this bug exists in Morpheus, but is difficult to reproduce in MRC. A MRC based test coming soon. fixes nv-morpheus#262 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#364

This PR replaces the `copy_prs` functionality from the `ops-bot` with the new dedicated `copy-pr-bot` GitHub application. Thorough documentation for the new `copy-pr-bot` application can be viewed below. - https://docs.gha-runners.nvidia.com/apps/copy-pr-bot/ **Important**: `copy-pr-bot` enforces signed commits. If an organization member opens a PR that contains unsigned commits, it will be deemed untrusted and therefore require an `/ok to test` comment. See the GitHub docs [here](https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification) for information on how to set up commit signing. Any time a PR is deemed untrusted, it will receive a comment that looks like this: rapidsai/ci-imgs#63 (comment). Every subsequent commit on an untrusted PR will require an additional `/ok to test` comment. Any existing PRs that have unsigned commits after this change is merged will require an `/ok to test` comment for each subsequent commit _or_ the PR can be rebased to include signed commits as mentioned in the docs below: https://docs.gha-runners.nvidia.com/cpr/contributors. This information is all included on the documentation page linked above. _I've skipped CI on this PR since it's not a change that is tested._ [skip ci]

…-morpheus#377) Cherry picks some changes from nv-morpheus#361 to allow other PRs to run CI since that PR is marked as `DO_NOT_MERGE` Authors: - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) - Christopher Harris (https://github.com/cwharris) URL: nv-morpheus#377

…ironment (nv-morpheus#375) Adds a new `dependencies.yaml` which is used in RAPIDS devcontainer utils to generate conda env yamls, `project.toml`'s, and `setup.py`s on-the-fly using https://github.com/rapidsai/dependency-file-generator. Eventually we can generate/verify our existing environment files this way, but for now this gets us up and running in a RAPIDS devcontainer. Also changes `prometheus-cpp::core` to a `PRIVATE`ly linked library, as it's not longer necessary to be `PUBLIC`, and otherwise breaks a combined MRC/Morpheus build. Also adds a `MRC_PYTHON_INPLACE_BUILD` cmake option, which enables pybind11 modules to be copied back to the source tree. MRC/Morpheus unified build environment is a WIP, but available here: https://github.com/cwharris/cyber-dev. Contributes to nv-morpheus/Morpheus#704 Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#375

Replace usage of deprecated `make_node_full` with `make_node` in the Quick Start Guide. Also fix a typo. Contributes to nv-morpheus/Morpheus#782. Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#376

fixes nv-morpheus#379 ? Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Christopher Harris (https://github.com/cwharris) - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#382

…-cap-add=sys_nice (nv-morpheus#383) REME.md update Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Christopher Harris (https://github.com/cwharris) - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#383

PR updates `rapidsai/ci` references to `rapidsai/ci-conda`. Authors: - Jake Awe (https://github.com/AyodeAwe) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: nv-morpheus#396

@mdemoret-nv

* Based on fixes from @mdemoret-nv : https://github.com/mdemoret-nv/MRC/tree/mdd_control-plane-promises & nv-morpheus#381 * Adopts updated versions of boost, clang & IWYU * Only run IWYU on files changed in PR * Use clang for checks stage * Allocate progress engine promises on the heap fixes nv-morpheus#379 Authors: - David Gardner (https://github.com/dagardner-nv) - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) - Christopher Harris (https://github.com/cwharris) URL: nv-morpheus#391

…orpheus#387) Adds support for unlimited pipe arguments. Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#387

* Docker image uses `rapidsai/ci-conda` as the base per rapidsai/ops#2806 * Adds the ability to CI locally via `./ci/scripts/run_ci_local.sh build` & `./ci/scripts/run_ci_local.sh test` * Fix type-o in benchmark stage causing benchmarks to be executed twice & preventing results from being published. * Add `.tmp` and `.clangd` to `.gitignore` * Document need to set `DOCKER_BUILDKIT=1` for docker builds. fixes nv-morpheus#393 fixes nv-morpheus#397 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Christopher Harris (https://github.com/cwharris) URL: nv-morpheus#394

* PR replicates issue nv-morpheus#362, and will trigger a pybind11 internal error using an un-patched version of pybind11 * Only run IWYU on files changed in PR Note: * This bug requires the code in question to be run in a thread created by C++. fixes nv-morpheus#362 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#365

cwharris

Approved, but I'm not familiar enough with most of the MRC aspects yet to give a thorough review.

cwharris · 2023-09-28T19:08:23Z

cpp/mrc/tests/test_edges.cpp

@@ -824,6 +1012,182 @@ TEST_F(TestEdges, CombineLatest)
    source2->run();

    sink->run();
+
+    EXPECT_EQ(sink->get_values(),


This test doesn't account for all 5 pairs/values that could be yielded in a real scenario where the elements are interleaved, it only accounts for the scenario where source1 is run entirely before source2, meaning we don't get anything except the 2's for source1. Is it worth testing more thoroughly than this?

cwharris · 2023-09-28T19:13:10Z

cpp/mrc/tests/test_edges.cpp

+    source1->run();
+
+    // Should throw when pushing last value
+    EXPECT_THROW(source2->run(), exceptions::MrcRuntimeError);


question: Is this going to cause us to throw when we try to shutdown the pipeline? Is that something we want?

cwharris · 2023-09-28T19:14:27Z

cpp/mrc/tests/test_edges.cpp

+
+    sink->run();
+
+    EXPECT_EQ(sink->get_values(),


question: This differs from the previous test where we throw an exception. Why is that? Is it because we treat sink 0 and sink 1 differently in terms of the number of elements they are allowed to accept?

cwharris · 2023-09-28T19:29:11Z

cpp/mrc/tests/test_edges.cpp

+    // Push 2 from each
+    source2->push(2);
+    source1->push(2);
+    source3->push(2);
+
+    // Push 2 from each
+    source2->push(2);
+    source1->push(2);
+    source3->push(2);
+
+    // Push the rest
+    source3->run();
+    source1->run();
+    source2->run();
+
+    sink->run();
+
+    EXPECT_EQ(sink->get_values(),
+              (std::vector<std::tuple<int, float, std::string>>{
+                  std::tuple<int, float, std::string>{0, 1, "a"},
+                  std::tuple<int, float, std::string>{1, 1, "a"},
+                  std::tuple<int, float, std::string>{2, 3, "b"},
+                  std::tuple<int, float, std::string>{3, 3, "b"},
+                  std::tuple<int, float, std::string>{4, 3, "e"},
+              }));


remark: the only thing I'm fuzzy on is whether 0, 1, a should be in the output, since 0 _ _ was emitted before we had both _ 1 _ and _ _ a.
question: are we suppose to remember and emit for all values of the source observable, or just the ones that occur after we've received at least one from the others?

cwharris · 2023-09-28T20:13:06Z

cpp/mrc/include/mrc/node/operators/with_latest_from.hpp

+    template <size_t N>
+    channel::Status set_upstream_value(NthTypeOf<N, TypesT...> value)
+    {
+        std::unique_lock<decltype(m_mutex)> lock(m_mutex);


Suggested change

std::unique_lock<decltype(m_mutex)> lock(m_mutex);

auto lock = std::unique_lock(m_mutex);

cwharris · 2023-09-28T20:14:12Z

cpp/mrc/include/mrc/node/operators/with_latest_from.hpp

+
+    void edge_complete()
+    {
+        std::unique_lock<decltype(m_mutex)> lock(m_mutex);


Suggested change

std::unique_lock<decltype(m_mutex)> lock(m_mutex);

auto lock = std::unique_lock(m_mutex);

…rpheus#402) * Prevents `check_active_connection` from mistakenly returning true for a holder where `init_owned_edge` has been called but neither the `init_connected_edge method` or the `add_connector` method have not been called. Relates to issue nv-morpheus#360 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#402

Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#467

…s#464) Adds maximum simultaneous tasks support to `TaskContainer` by only starting new tasks if the number of currently executing tasks is less than the maximum number of simultaneous tasks and starting new tasks as older tasks shut down. this eliminates the need for external semaphores or ticketing systems. Authors: - Christopher Harris (https://github.com/cwharris) - David Gardner (https://github.com/dagardner-nv) - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#464

…ranch-*` and `main` to succeed

Morpheus documentation won't build with doxygen 1.10.0, but we need MRC and Morpheus doxygen versions to match so cross-repository environments can be solved correctly. This PR downgrades doxygen to the version used by Morpheus. Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#469

…path methods (nv-morpheus#465) * All three methods performed nearly identical operations, but were defined in anonymous namespaces and not exposed in a header preventing sharing. * Consolidate all three into a new `split_string_to_vector` in `string_utils.hpp` closes nv-morpheus#454 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#465

…ge-24.03 Forward-merge branch-24.03 into branch-24.06

* Adds `AutoRegSinkAdapter` and `AutoRegEgressPort` to `AsyncSink` * Adds `AutoRegSourceAdapter` and `AutoRegIngressPort` to `AsyncSource` * Giving `AsyncSink`, `AsyncSource` and `AsyncioRunnable` the same implicit converter functionality that `PythonNode` has * Fixes unrelated CI issue for IWYU, since IWYU can only run on compilation units, we can't execute it when a change only includes header files. Closes nv-morpheus/Morpheus#1626 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#473

Forward-merge branch-24.03 into branch-24.06

…ng-operators

codecov · 2024-05-01T00:10:38Z

Codecov Report

Attention: Patch coverage is 86.56716% with 18 lines in your changes are missing coverage. Please review.

❗ No coverage uploaded for pull request base (branch-23.07@5f62b46). Click here to learn what that means.
Report is 7 commits behind head on branch-23.07.

❗ Current head 7cbfe8e differs from pull request most recent head 5c52d5c. Consider uploading reports for the commit 5c52d5c to get more accurate results

Additional details and impacted files

@@               Coverage Diff               @@
##             branch-23.07     #330   +/-   ##
===============================================
  Coverage                ?   73.26%           
===============================================
  Files                   ?      386           
  Lines                   ?    13533           
  Branches                ?     1024           
===============================================
  Hits                    ?     9915           
  Misses                  ?     3618           
  Partials                ?        0

Flag	Coverage Δ
cpp	`69.25% <86.56%> (?)`
py	`42.06% <100.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
cpp/mrc/include/mrc/edge/edge_channel.hpp	`85.00% <100.00%> (ø)`
.../mrc/include/mrc/node/operators/combine_latest.hpp	`100.00% <100.00%> (ø)`
cpp/mrc/include/mrc/node/sink_channel_owner.hpp	`100.00% <100.00%> (ø)`
cpp/mrc/include/mrc/node/source_channel_owner.hpp	`90.90% <100.00%> (ø)`
cpp/mrc/include/mrc/utils/tuple_utils.hpp	`100.00% <100.00%> (ø)`
cpp/mrc/include/mrc/node/operators/zip.hpp	`98.21% <98.21%> (ø)`
...rc/include/mrc/node/operators/with_latest_from.hpp	`95.65% <95.65%> (ø)`
cpp/mrc/include/mrc/channel/status.hpp	`0.00% <0.00%> (ø)`

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5f62b46...5c52d5c. Read the comment docs.

Contributes to rapidsai/dependency-file-generator#89. `--file_key` has been deprecated in `rapids-dependency-file-generator` for a few months, and I'm looking to fully remove it. This PR proposes: * switching this project's uses from `--file_key` to `--file-key` * updating the version of `rapids-dependency-file-generator` used in `pre-commit` to its latest release * switching from `mamba env create --force` to `mamba env create --yes` Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Christopher Harris (https://github.com/cwharris) URL: nv-morpheus#482

Forward-merge branch-24.06 into branch-24.10

…morpheus#485) * Invoke `morpheus_utils_initialize_install_prefix` function after calling `project` * Related to nv-morpheus/Morpheus#1776 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#485

…morpheus#493) * When a Python generator source yields a value, and the subscriber is no longer subscribed, stop the source. * Fix out of date docstring comment. This is a partial fix for nv-morpheus/Morpheus#1838 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: nv-morpheus#493

…-morpheus#496) * Allows a Python generator source to check if the subscriber is still subscribed. * Define a class `SubscriberFuncWrapper` for Python sources rather than just a lambda. The reason is that python objects captured by the lambda need to be destroyed while the gil is held, which causes a problem if the lambda is destroyed unexpectedly. * Update `conftest.py` to set the loglevel to `DEBUG` if the `GLOG_v` environment variable is defined. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#496

…heus#497) * Since CPU-only mode will become a supported feature we want to avoid unnecessary warnings. Relates to nv-morpheus/Morpheus#1851 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#497

…iber` (nv-morpheus#499) * Remove the `make_source_subscriber` method in favor of inspecting the Python function signature. * Since the `make_source_subscriber` method was never part of a release I think this can still be considered a non-breaking change. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#499

Closes nv-morpheus#478 Requires nv-morpheus/utilities#75 Authors: - Christopher Harris (https://github.com/cwharris) - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) - David Gardner (https://github.com/dagardner-nv) URL: nv-morpheus#494

…ng-operators

copy-pr-bot · 2024-10-06T21:38:26Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

mdemoret-nv added 2 commits June 2, 2023 13:21

Fixing CombineLatest and adding Zip operator

5bc9b35

Adding WithLatestFrom

293c481

mdemoret-nv added non-breaking Non-breaking change feature request New feature or request labels Jun 3, 2023

mdemoret-nv requested review from a team as code owners June 3, 2023 00:12

IWYU fixes

32672c9

jarmak-nv assigned mdemoret-nv Jun 5, 2023

drobison00 approved these changes Jun 5, 2023

View reviewed changes

mdemoret-nv and others added 19 commits June 21, 2023 12:07

Adding copyright header

34d4767

Merge branch 'branch-23.07' into mdd_combining-operators

cfbafd3

Merge branch 'branch-23.07' into mdd_combining-operators

7cbfe8e

In progress adding children to Object classes

0178730

Creating branch for v23.11

e6a320f

Merge pull request nv-morpheus#358 from nv-morpheus/branch-23.07

9b47924

Forward-merge branch-23.07 to branch-23.11

Revert boost upgrade, and update clang to v16 (nv-morpheus#382)

5747320

fixes nv-morpheus#379 ? Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Christopher Harris (https://github.com/cwharris) - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#382

update rapidsai/ci to rapidsai/ci-conda (nv-morpheus#396)

18ffe6e

PR updates `rapidsai/ci` references to `rapidsai/ci-conda`. Authors: - Jake Awe (https://github.com/AyodeAwe) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: nv-morpheus#396

Update ObservableProxy::pipe to support any number of operators (nv-m…

5a4a1a4

…orpheus#387) Adds support for unlimited pipe arguments. Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#387

cwharris approved these changes Sep 28, 2023

View reviewed changes

dagardner-nv and others added 13 commits April 5, 2024 09:40

Updating versions for v24.06.00

5a6ae3c

Update minimum requirements (nv-morpheus#467)

029ac49

Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#467

Disabling the root level RAPIDS "checks" step in CI to allow CI on `b…

99a7add

…ranch-*` and `main` to succeed

Updating CHANGELOG

edad3c3

Cleaning up style for release

e1d0e9d

Merge branch-24.03 into branch-24.06

2308e23

Merge pull request nv-morpheus#471 from dagardner-nv/branch-24.06-mer…

41afd96

…ge-24.03 Forward-merge branch-24.03 into branch-24.06

Updating CHANGELOG

84bfffc

Merge pull request nv-morpheus#474 from nv-morpheus/branch-24.03

7b599a3

Forward-merge branch-24.03 into branch-24.06

mdemoret-nv requested a review from a team as a code owner May 1, 2024 00:02

Merge remote-tracking branch 'upstream/branch-24.06' into mdd_combini…

5c52d5c

…ng-operators

jameslamb and others added 12 commits June 17, 2024 16:16

Creating branch for v24.10

f754c78

Updating versions for v24.10.00

246ac97

Updating CHANGELOG

cc89f5f

Merge pull request nv-morpheus#484 from nv-morpheus/branch-24.06

8377f6a

Forward-merge branch-24.06 into branch-24.10

Merge remote-tracking branch 'upstream/branch-24.10' into mdd_combini…

7629d78

…ng-operators

Fixing merge

61fa294

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding Combination Operators `Zip`, `CombineLatest`, `WithLatestFrom` #330

Adding Combination Operators `Zip`, `CombineLatest`, `WithLatestFrom` #330

mdemoret-nv commented Jun 3, 2023

cwharris left a comment

cwharris Sep 28, 2023

cwharris Sep 28, 2023

cwharris Sep 28, 2023

cwharris Sep 28, 2023

cwharris Sep 28, 2023

cwharris Sep 28, 2023

codecov bot commented May 1, 2024

copy-pr-bot bot commented Oct 6, 2024

	std::unique_lock<decltype(m_mutex)> lock(m_mutex);
	auto lock = std::unique_lock(m_mutex);

Adding Combination Operators Zip, CombineLatest, WithLatestFrom #330

Are you sure you want to change the base?

Adding Combination Operators Zip, CombineLatest, WithLatestFrom #330

Conversation

mdemoret-nv commented Jun 3, 2023

Description

Checklist

cwharris left a comment

Choose a reason for hiding this comment

cwharris Sep 28, 2023

Choose a reason for hiding this comment

cwharris Sep 28, 2023

Choose a reason for hiding this comment

cwharris Sep 28, 2023

Choose a reason for hiding this comment

cwharris Sep 28, 2023

Choose a reason for hiding this comment

cwharris Sep 28, 2023

Choose a reason for hiding this comment

cwharris Sep 28, 2023

Choose a reason for hiding this comment

codecov bot commented May 1, 2024

Codecov Report

copy-pr-bot bot commented Oct 6, 2024

Adding Combination Operators `Zip`, `CombineLatest`, `WithLatestFrom` #330

Adding Combination Operators `Zip`, `CombineLatest`, `WithLatestFrom` #330