This repository contains the C++ implementation of the shared memory mechanism described in the paper Smart Pointers and Shared Memory Synchronisation for Efficient Inter-process Communication in ROS on an Autonomous Vehicle, along with detailed build and deployment instructions.
The implementation is based on a fork of ros_comm kinetic 1.12.14. along with a suite of benchmarks that were used to generate the LOT benchmark results.
The code can be built either directly on the host machine or within a Docker container via the provided Dockerfile. The former assumes the installation of ROS 1.12.14, Boost 1.58.0 and python2 on the host machine. The latter only requires Docker installation on the host machine. See the Docker website for details of setting up Docker on the host machine.
The benchmarks - either the entire suite or individual tests - can be executed following the procedure described below. Currently, four protocols are supported: TCP, UDP, LOT and TZC.
When executed from within the Docker containers, the benchmarks results are generated under ${HOME}/lot folder. Otherwise, environment variable lot_HOME dictates the results folder.
Note that we clone the repository into $HOME/ros_comm/src/ros_comm in this step. We assume this directory in subsequent steps. If you prefer to use a different working directory, the subsequent steps will need to be appropriately modified.
git clone [email protected]:fiveai/ros_comm.git $HOME/ros_comm/src/ros_comm
cd $HOME/ros_comm/src/ros_comm
git checkout lot
The following steps build the code directly on the host machine.
cd $HOME/ros_comm
source /opt/ros/kinetic/setup.sh # this assumes ROS kinetic 1.12.14 is already installed
catkin_make_isolated
--source ./src/ \
--build $HOME/ros_comm/build-release \
--devel $HOME/ros_comm/devel-release \
--install-space $HOME/ros_comm/install-release \
--install \
--cmake-args -DCMAKE_BUILD_TYPE=Release
cd $HOME/ros_comm
source ./devel-release/setup.sh
roslaunch --screen -v benchmark launch.xml \
use_case:=5p1s_same_host sub_stats_file_path:=$HOME \
transport:=shm pub_queue_size:=200 pub_image_count:=200
cd $HOME/ros_comm/clients/benchmark/docker
export DOCKER_BUILDKIT=1
docker build --ssh default --tag lot .
Launch 6 containers based on the above image, and attach to one of them:
cd $HOME/ros_comm/clients/benchmark/docker
docker-compose up -d
docker attach node1
From within the container, benchmarks can be executed using commands of the form:
python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py --tcp=no --shm=yes --use_case=5p1s_separate_docker
Once you're finished, exit the docker container and shutdown the others
docker-compose stop
You should find the results in $HOME/lot
The following section provides examples of benchmark commands.
- Execute the benchmark suite for 1p5s using TZC protocol, in separate Docker containers, enforcing the subscribers start up order, allocating 16GB of shared memory to be used by the publishers, and telling the subscriber to wait 15 seconds before starting publishing messages,
TCP_NODELAYenabled.
python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py \
--tcp=no --shm=no --tzc=yes --udp=no \
--use_case=1p5s_separate_docker --no_pool=yes --pool=no \
--extra_params sub_enable_synch_startup:=true \
pub_extra_delay_ms:=15000 \
shm_size_mega_bytes:=3000
- Execute the benchmarks suite for 5p1s using TZC protocol, in separate Docker containers, with each publisher waiting for the subscriber to establish connection and with the subscriber start up delayed by 15secs,
TCP_NODELAYenabled.
python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py \
--tcp=no --shm=no --tzc=yes --udp=no \
--use_case=5p1s_separate_docker --no_pool=yes --pool=no \
--extra_params pub_wait_for_subscribers:=true \
sub_extra_delay_ms:=15000
- Execute the benchmarks suite for 5p1s using TZC protocol, in separate Docker containers, with each publisher waiting for the subscriber to establish connection and with the publishers start up order enforced,
TCP_NODELAYenabled, overriding the default results path.
python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py \
--tcp=no --shm=no --tzc=yes --udp=no \
--use_case=5p1s_separate_docker --no_pool=yes --pool=no \
--extra_params pub_wait_for_subscribers:=true \
pub_enable_synch_startup:=true \
sub_stats_file_path:=/path/to/results
- Execute the benchmarks suite for 5p1s using LOT protocol, in separate Docker containers, with each publisher waiting for the subscriber to establish connection and with the publishers start up order enforced, and image pools disabled.
python2 /ros_comm/src/ros_comm/clients/benchmark/execute.py \
--tcp=no --shm=yes --tzc=no --udp=no \
--use_case=5p1s_separate_docker --no_pool=yes --pool=no \
--extra_params pub_wait_for_subscribers:=true \
pub_enable_synch_startup:=true
- Execute one single test using TCP protocol, in same docker container, with image pool disabled.
roslaunch --screen -v benchmark launch.xml \
pub_image_count:=200 \
use_case:=1p5s_same_docker \
transport:=tcp \
image_width_pixels:=512 \
image_height_pixels:=512 \
pub_pool_size:=0
LOT is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
This work is based on a fork of ros_comm kinetic 1.12.14.
We apply the following fixes to ros_comm, in addition to our own code:
- #1115 roslaunch - pass through command-line args to the xmlloader when using the API.
- replace
boost::condition_variablevariable withstd::condition_variablein callback_queue.cpp.
Boost.Process 1_65_0 has been imported into the source tree at ./clients/roscpp/include/boost_1.65.0 to support unit tests. We note that it is released under the Boost Software License.
For convenience of running the benchmarks, we have also included code from TZC at ./clients/roscpp/include/tzc. TZC was developed by a group researchers affiliated to Tsinghua University, China and University of Maryland, USA and is described more fully in their paper TZC: Efficient Inter-Process Communication for Robotics Middleware with Partial Serialization. We note it is released it under a BSD license.
We thank the authors for making their code available.