From 08251581d70c16e387a7620c29e834fbf80dd4f6 Mon Sep 17 00:00:00 2001 From: Mayank Mittal Date: Mon, 12 Feb 2024 15:34:22 +0100 Subject: [PATCH 1/3] adds known issue on stepping --- docs/source/refs/issues.rst | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/docs/source/refs/issues.rst b/docs/source/refs/issues.rst index 8fdce8797d..a6acc8482a 100644 --- a/docs/source/refs/issues.rst +++ b/docs/source/refs/issues.rst @@ -1,6 +1,35 @@ Known issues ============ +Stale values after resetting the environment +-------------------------------------------- + +When resetting the environment, some of the data fields of assets and sensors are not updated. +These include the poses of links in a kinematic chain, the camera images, the contact sensor readings, +and the lidar point clouds. This is a known issue and which has to do with the way the PhysX and +rendering engines work in Omniverse. + +Many physics engines do a simulation step as a two-level call: ``forward()`` and ``simulate()``, +where the kinematic and dynamic states are updated, respectively. Unfortunately, PhysX has only a +single ``step()`` call where the two operations are combined. Due to computations through GPU +kernels, it is not so straightforward for them to split these operations. Thus, at the moment, +it is not possible to set the root and/or joint states and do a forward call to update the +kinematic states of links. This affects both initialization as well as episodic resets. + +Similarly for RTX rendering related sensors, the sensor data is not updated immediately after +setting the state of the sensor. The rendering engine update is bundled with the simulator's +``step()`` call which only gets called when the simulation is stepped forward. This means that +the sensor data is not updated immediately after a reset. + +While the above is erroneous, there is currently no direct workaround for it. From our experience in +using IsaacGym, the reset values are crucial. However, it does not affect the agent learning +critically enough. + +We have made a feature request to the respective Omniverse teams to have complete control +over stepping different parts of the simulation app. However, at this point, there is no set +timeline for this feature request. + + Blank initial frames from the camera ------------------------------------ From 71a0cf2b21f8939fc6de2d82491e1b52e3ec3991 Mon Sep 17 00:00:00 2001 From: Mayank Mittal Date: Mon, 12 Feb 2024 15:50:35 +0100 Subject: [PATCH 2/3] adds clearer explaination --- docs/source/refs/issues.rst | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/source/refs/issues.rst b/docs/source/refs/issues.rst index a6acc8482a..cd46e8f0da 100644 --- a/docs/source/refs/issues.rst +++ b/docs/source/refs/issues.rst @@ -1,6 +1,10 @@ Known issues ============ +.. attention:: + + Please also refer to the `Omniverse Isaac Sim documentation`_ for known issues and workarounds. + Stale values after resetting the environment -------------------------------------------- @@ -19,11 +23,12 @@ kinematic states of links. This affects both initialization as well as episodic Similarly for RTX rendering related sensors, the sensor data is not updated immediately after setting the state of the sensor. The rendering engine update is bundled with the simulator's ``step()`` call which only gets called when the simulation is stepped forward. This means that -the sensor data is not updated immediately after a reset. +the sensor data is not updated immediately after a reset and it will hold outdated values. While the above is erroneous, there is currently no direct workaround for it. From our experience in -using IsaacGym, the reset values are crucial. However, it does not affect the agent learning -critically enough. +using IsaacGym, the reset values affect the agent learning critically depending on how frequently +the environment terminates. Eventually if the agent is learning successfully, this number drops +and does not affect the performance that critically. We have made a feature request to the respective Omniverse teams to have complete control over stepping different parts of the simulation app. However, at this point, there is no set @@ -63,3 +68,4 @@ This is then replicated across other references of the same asset since physics are stored in the instanceable asset's USD file and not in its stage reference's USD file. .. _instanceable assets: https://docs.omniverse.nvidia.com/app_isaacsim/app_isaacsim/tutorial_gym_instanceable_assets.html +.. _Omniverse Isaac Sim documentation: https://docs.omniverse.nvidia.com/isaacsim/latest/known_issues.html From 4bd4e2c60b394bf2f6ba40e056e949da652d051c Mon Sep 17 00:00:00 2001 From: Mayank Mittal Date: Mon, 12 Feb 2024 16:12:49 +0100 Subject: [PATCH 3/3] adds review comments --- docs/source/refs/issues.rst | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/source/refs/issues.rst b/docs/source/refs/issues.rst index cd46e8f0da..1ae53289c0 100644 --- a/docs/source/refs/issues.rst +++ b/docs/source/refs/issues.rst @@ -10,7 +10,7 @@ Stale values after resetting the environment When resetting the environment, some of the data fields of assets and sensors are not updated. These include the poses of links in a kinematic chain, the camera images, the contact sensor readings, -and the lidar point clouds. This is a known issue and which has to do with the way the PhysX and +and the lidar point clouds. This is a known issue which has to do with the way the PhysX and rendering engines work in Omniverse. Many physics engines do a simulation step as a two-level call: ``forward()`` and ``simulate()``, @@ -20,10 +20,11 @@ kernels, it is not so straightforward for them to split these operations. Thus, it is not possible to set the root and/or joint states and do a forward call to update the kinematic states of links. This affects both initialization as well as episodic resets. -Similarly for RTX rendering related sensors, the sensor data is not updated immediately after -setting the state of the sensor. The rendering engine update is bundled with the simulator's -``step()`` call which only gets called when the simulation is stepped forward. This means that -the sensor data is not updated immediately after a reset and it will hold outdated values. +Similarly for RTX rendering related sensors (such as cameras), the sensor data is not updated +immediately after setting the state of the sensor. The rendering engine update is bundled with +the simulator's ``step()`` call which only gets called when the simulation is stepped forward. +This means that the sensor data is not updated immediately after a reset and it will hold +outdated values. While the above is erroneous, there is currently no direct workaround for it. From our experience in using IsaacGym, the reset values affect the agent learning critically depending on how frequently