2024-10-28 ODD working group weekly meeting (211)

[Igata-ctrl]

Attendee

• Daniel Shih (TIER IV/Reference Design WG/National Taiwan University)
• David Walmroth (PIX Moving)
• @mitsudome-r Ryohsuke Mitsudome (AWF TSC/TIER IV)
• @Igata-ctrl Hiroshi IGATA (TIER IV)

Agenda/Minutes

• On September 22, 2024, 298/340 "public road bus" and "pull out" scenarios succeeded on Autoware.universe main branch.

  • 111/142 public road bus scenarios were successful. Identical to the result of recent weeks.

    • UC-23-004-0001-With2Obstacles-ShalunRS : Ego looks like ignoring the traffic from its left.
    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops though "isBLIND:true" is set.
    • In addition to the above, 30 scenarios (including parameter variations) failed due to system errors (Simulation Health Check, Time Out etc.).

  • 187/198 pull out scenarios were successful. Two scenarios became successful after explicitly setting Ego's max speed of 4.2m/s.

    • UC-v2-F-15-00003_001_case01 : Failed for (D1=8, D2=4) case. Ego was not able to back up due to "obstacle stop".
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : Failed -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).
    • UC-v2-F-15-00103_001_case01_cmn_general : "Obstacle stop" prevents EGO from proceeding. -> The scenario intention is to use the space outside of the lane, but current Autoware.universe main is not capable of it.

• On September 22, 2024, 62/96 Dense Urban scenarios were successful on Autoware.universe scenario-dev branch.

  • Test-SafetyPool-map1 : Has been successful for a month, but failed this time. (A simple lane change did not happen.)
  • Test-dense-urban_ITRI(2|3|4) : EGO cannot use the space in th opposite lane when it passes/overtakes NPC/Pedestrians.
  • Test-roundabout1_Shalun : Ego cannot avoid collision with the NPC in the round about. -> It looks like NPC is not detected (as the red bounding box does not appear in Rviz screen).
  • Test_MRM_Fault_Injection1 : MRM not conducted upon the fault injection.
  • UC-NTR-001-0003_case1 : Vn=5.55 (slowest) case failed for the excess deceleration (<-2.499m/s2).
  • UC-NTR-001-0005 : All 3 cases failed for "Ego moved before Npc takeover finished.
  • UC-NTR-001-0008 : Failed for Vslow= 3.472 case as Ego decelerated to < 80% of Vslow. -> Became successful after the map version-up and fixing Npc start timing, etc.)
  • UC-NTR-001-0009 : Failed for as Ego decelerated to < 80% of Vslow. (After explicitly setting Ego max speed to 4.2m/s, it then causes a crash against Npc.)
  • UC-NTR-002-0001_case(1|2), UC-NTR-002-0003_case(1|2) : EGO cannot overtake NPC when Npc is fast (=2.77m/s) and failed with the excess deceleration.
  • UC-NTR-002-0004 : Failed with the excess deceleration.
  • UC-NTR-003-0002 : Failed as Ego stops twice and one of the failure conditions was met.
  • UC-NTR-004-0001 : EGO did not avoid the NPCs coming in the wrong direction. ("max_negative_acc" was not modified.)
  • UC-VRU-001-0001_case1 : Failed with unnecessary stop, or "act_check_ego_stopped_in_overtaking".
  • UC-VRU-001-0001_case2 : Failed with the excess deceleration.
  • UC-VRU-002-0001_case(1|2) : Failed with the excess deceleration.
  • UC-VRU-002-0002_case(1|2) : Failed with the excess deceleration.
  • UC-VRU-002-0003_case(1|2) : Failed with the excess deceleration.
  • UC-VRU-002-0005 : Failed with the excess deceleration.

• On September 29, 2024, 294/340 "public road bus" and "pull out" scenarios succeeded on Autoware.universe main branch.

  • 111/142 public road bus scenarios were successful. Identical to the result of recent weeks. (Strictly speaking this number was from a retry on September 30 with "Retry=1" option, though the number of success cases increased by two without any retries.)

    • UC-23-004-0001-With2Obstacles-ShalunRS : Ego looks like ignoring the traffic from its left.
    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops though "isBLIND:true" is set.
    • In addition to the above, 30 scenarios (including parameter variations) failed due to system errors (Simulation Health Check, Time Out etc.).

  • 185/198 pull out scenarios were successful. Two scenarios became successful after explicitly setting Ego's max speed of 4.2m/s.

    • UC-v2-F-15-00002_001_case03 : 1/6 case failed as Ego stopped unnecessarily. -> Succeeded after explicitly setting Ego max speed = 4.2m/s.
    • UC-v2-F-15-00003_001_case01 : 3/4 cases failed as Ego was not able to back up as intended".
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : Failed -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).
    • UC-v2-F-15-00103_001_case01_cmn_general : "Obstacle stop" prevents EGO from proceeding. -> The scenario intention is to use the space outside of the lane, but current Autoware.universe main is not capable of it. -> Succeeded this week! (The scenario has not been modified.)

• On September 29, 2024, Dense Urban scenarios failed with build error. @mitsudome-r and @brkay54 are kindly retrying them.

  • The result from a retry after the additional modification on October 1 was 79/96 success.

• On October 6, 2024, 299/340 "public road bus" and "pull out" scenarios succeeded on Autoware.universe main branch.

  • 111/142 public road bus scenarios were successful. Identical to the result of recent weeks.

    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops against Ego vehicle though "isBLIND:true" is set.
    • In addition to the above, UC-23-004-0001-With2Obstacles-ShalunRS and 30 other scenarios (including parameter variations) failed due to system errors (Simulation Health Check, Time Out etc.).

  • 188/198 pull out scenarios were successful. 10 scenarios failed.

    • UC-v2-F-15-00003_001_case01 : 1/4 case failed as Ego was not able to back up as intended. (It failed even after "act_ego nostop" condition was released.)
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : Failed -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).

• On October 6, 2024, 79/99 Dense Urban scenarios were successful on Autoware.universe scenario-dev branch. 20 scenarios failed.

  • Test-SafetyPool-map1 : Has been successful for a month, but failed this time. (A simple lane change did not happen.)
  • Test-roundabout1_Shalun : Ego cannot avoid collision with the NPC in the round about. -> It looks like NPC is not detected (as the red bounding box does not appear in Rviz screen).
    • Above two scenarios succeed in the test on October 3 with the build in May 2024.
  • Test-dense-urban_ITRI(2|3|4) : EGO cannot use the space in the opposite lane when it passes/overtakes NPC/Pedestrians.
  • Test_MRM_Fault_Injection1 : MRM not conducted upon the fault injection.
  • UC-NTR-001-0003_case2 : 1/3 case failed for the excess deceleration < -2.499.
  • UC-NTR-001-0005 : All 3 cases failed for "Ego moved before Npc takeover finished.
  • UC-NTR-002-0003_case2 : 1/3 case failed with the excess deceleration.
  • UC-NTR-004-0001 : EGO did not avoid the NPCs coming in the wrong direction.
  • UC-VRU-002-0001_case(1|2) : Failed with the excess deceleration.
  • UC-VRU-002-0005 : Failed with the excess deceleration. (Succeeded in the test on October 3.)

• On October 13, 2024, 297/340 "public road bus" and "pull out" scenarios succeeded on Autoware.universe main branch.

  • 111/142 public road bus scenarios were successful. Identical to the result of recent weeks.

    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops against Ego vehicle though "isBLIND:true" is set.
    • In addition to the above, UC-23-004-0001-With2Obstacles-ShalunRS and 30 other scenarios (including parameter variations) failed due to system errors (Simulation Health Check, Time Out etc.).

  • 186/198 pull out scenarios were successful. 12 scenarios failed.

    • UC-v2-F-15-00002_001_case01 : 1/6 case failed as Ego stopped unnecessarily. -> Succeeded after explicitly setting Ego max speed = 4.2m/s.
    • UC-v2-F-15-00003_001_case01 : 1/4 case failed as Ego was not able to back up as intended. (It failed even after "act_ego nostop" condition was released.)
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : Failed -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).
    • UC-v2-F-15-00007_001_case04_cmn_general : 1/36 case failed as Ego stopped unnecessarily. -> Succeeded after setting Ego max speed = 4.2m/s by directly writing it in numbers not using the parameter $Ve. (TBC next week)

• On October 13, 2024, 76/99 Dense Urban scenarios were successful on Autoware.universe scenario-dev branch. 230 scenarios failed. From this week three modified scenarios for reference design working group have been added and they all succeeded.

  • Test-SafetyPool-map1 : Has been successful for a month, but failed this time. (A simple lane change did not happen.)
  • Test-roundabout1_Shalun : Ego cannot avoid collision with the NPC in the round about. -> It looks like NPC is not detected (as the red bounding box does not appear in Rviz screen).
    • Above two scenarios succeed in the test on October 3 with the build in May 2024.
  • Test-dense-urban_ITRI(2|3|4) : EGO cannot use the space in the opposite lane when it passes/overtakes NPC/Pedestrians.
  • Test_MRM_Fault_Injection1 : MRM not conducted upon the fault injection.
  • UC-NTR-001-0003_case2 : 1/3 case failed for the excess deceleration < -2.499.
  • UC-NTR-001-0005 : All 3 cases failed for "Ego moved before Npc takeover finished.
  • UC-NTR-002-0003_case2 : 2/3 cases failed. One with the excess deceleration, the other with the time out (no result).
  • UC-NTR-001-0008 : 1/3 case failed for Ego deceleration overshoot (<0.8*Vslow).
  • UC-NTR-004-0001 : EGO did not avoid the NPCs coming in the wrong direction.
  • UC-NTR-005-0001 : 1/3 case failed for Ego deceleration overshoot (<0.8*Vslow).
  • UC-VRU-002-0001_case(1|2) : Failed with the excess deceleration.
  • UC-VRU-002-0005 : Failed with the excess deceleration. (Succeeded in the test on October 3.)

• On October 20, 2024, 297/340 "public road bus" and "pull out" scenarios succeeded on Autoware.universe main branch.

  • 111/142 public road bus scenarios were successful. Identical to the results of recent weeks.

    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops against Ego vehicle though "isBLIND:true" is set.
    • In addition to the above, UC-23-004-0001-With2Obstacles-ShalunRS and 30 other scenarios (including parameter variations) failed due to system errors (Simulation Health Check, Time Out etc.).

  • 186/198 pull out scenarios were successful. 12 scenarios failed.

    • UC-v2-F-15-00003_001_case01 : 1/4 case failed as Ego was not able to back up as intended.(It failed even after "act_ego nostop" condition was released.)
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : Failed -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).
    • UC-v2-F-15-00007_001_case04_cmn_general : Last week's error was solved, but 2/36 case failed with "TimeOut" and "InternalError" message="attempting to request a message part outside the valid range"

• On October 20 2024, 79/96 Dense Urban scenarios were successful on Autoware.universe scenario-dev branch. 230 scenarios failed. From this week three modified scenarios for reference design working group have been added and they all succeeded.

  • Test-SafetyPool-map1 : Has been successful for a month, but failed this time. (A simple lane change did not happen.)
  • Test-roundabout1_Shalun : Ego cannot avoid collision with the NPC in the round about. -> It looks like NPC is not detected (as the red bounding box does not appear in Rviz screen).
    • Above two scenarios succeed in the test on October 3 with the build in May 2024.
  • Test-dense-urban_ITRI(2|3|4) : EGO cannot use the space in the opposite lane when it passes/overtakes NPC/Pedestrians.
  • Test_MRM_Fault_Injection1 : MRM not conducted upon the fault injection.
  • (UC-NTR-001-0003_case2 : all 3 cases succeeded without violating excess deceleration threshold.)
  • UC-NTR-001-0005 : All 3 cases failed for "Ego moved before Npc takeover finished.
  • (UC-NTR-001-0008 : All 3 case succeeded without any Ego's deceleration overshoot (<0.8*Vslow).)
  • (UC-NTR-002-0003_case2 : All 3 cases succeeded.)
  • UC-NTR-004-0001 : EGO did not avoid the NPCs coming in the wrong direction.
  • UC-NTR-005-0001 : All 3 case succeeded without any Ego's deceleration overshoot (<0.8*Vslow).
  • UC-VRU-002-0001_case(1|2) : Failed with the excess deceleration.
  • UC-VRU-002-0005 : Failed with the excess deceleration.
    • Above two scenarios succeed in the test on October 3 with the build in May 2024.

• On October 27, 2024, 295/340 "public road bus" and "pull out" scenarios succeeded on Autoware.universe main branch.

  • 110/142 public road bus scenarios were successful.
    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops against Ego vehicle though "isBLIND:true" is set.
    • In addition to the above, UC-23-004-0001-With2Obstacles-ShalunRS and 30 other scenarios (including parameter variations) failed due to system errors (Simulation Health Check, Time Out etc.).
    • UC-PB-002-0002-Hsinchu : 1/4 case failed with a system error. -> Succeeded after a simple retry.

• On October 20 2024, 75/96 Dense Urban scenarios were successful on Autoware.universe scenario-dev branch.

  • Test-SafetyPool-map1 : Has been successful for a month, but failed this time. (A simple lane change did not happen.)
  • Test-roundabout1_Shalun : Ego cannot avoid collision with the NPC in the round about. -> It looks like NPC is not detected (as the red bounding box does not appear in Rviz screen).
    • Above two scenarios succeed in the test on October 3 with the build in May 2024.
  • Test-dense-urban_ITRI(2|3|4) : EGO cannot use the space in the opposite lane when it passes/overtakes NPC/Pedestrians.
  • Test_MRM_Fault_Injection1 : MRM not conducted upon the fault injection.
  • UC-NTR-001-0003_case2 : 1/3 case failed with excess deceleration.
  • UC-NTR-001-0005 : All 3 cases failed for "Ego moved before Npc takeover finished.
  • (UC-NTR-001-0008 : All 3 case succeeded without any Ego's deceleration overshoot (<0.8*Vslow).)
  • UC-NTR-002-0001_case1 : 1/3 case failed with TimeOut.
  • (UC-NTR-002-0003_case2 : All 3 cases succeeded.)
  • UC-NTR-004-0001 : EGO did not avoid the NPCs coming in the wrong direction.
  • UC-NTR-005-0001 : All 3 case succeeded without any Ego's deceleration overshoot (<0.8*Vslow).
  • UC-VRU-001-0001_case1 : 1/2 case failed with excess deceleration.
  • UC-VRU-002-0001_case(1|2) : Failed with the excess deceleration.
  • UC-VRU-002-0005 : Failed with the excess deceleration.
    • Above two scenarios succeed in the test on October 3 with the main build in May 2024.

• Had a Safety Pool/AWF meeting regarding the existing issues.

• Discussed with Daniel whether the scenarios are appropriately copied to LSV catalog instead of modifying the original. -> @Igata-ctrl to double check it by next week.

• [recap of the discussions until last week]

• Preliminary discussion with TIER IV Evaluator team revealed that the import of OpenSCENARIO 1.x files will be supported hopefully by the end of 2024. Currently, Scenario Simulator v2 uses the proprietary t4v2 format in yaml to better handle the parameter variations. This nee import function enables the use of out side scenario editor like AVL SCENIUS™️ Scenario Designer and help importing scenarios from Safety Pool database, though the issue of map format and the lack of destinations may remain.

• Discussion to create Evaluator improvement wish list.

  • Embed illustrations/animation in the Scenario Editor description pane or enumeration. (Armağan)
  • Integration with Vector Map Editor. (Igata) -> Clarify if some of this function is already available.
  • Create a discussion to collect needs/wants for Evaluator from larger audience.

• Discussed the improvement requests for Evaluator.

  • Will continue discussion next week, Armağan's initial ides included embedding the use case definition (slides) and the scenario (NPC behavior) preview by the time-slider in the scenario editor screen, etc.
  • @Igata-ctrl explained that Evaluator is not an OSS but a product from TIER IV, so the development team has a long wish list form TIER IV and OEM projects.
  • May need to clarify the content of the wish list from AWF before discussing the priority with TIER IV CI/CD team. @Igata-ctrl to try to collect recent situation with the team.

• Regarding the needs for calculating the accurate RSS safety distance from Vf(Npc) and Vr(Ego), TIER IV Mapping and Localization team will be able to add the functionality using UserDefinedValueCondition like UC-v2-F-15-01001_001_case01_cmn_general case. It may take more than two weeks depending on their resource margin/availability. -> @brkay54 to kindly submit an issue regarding this request mentioning Takeuchi-san of TIER IV (@kosuke55) who is working on UserDefinedValueCondition.

• For the current scenario suite for weekly execution above, the RSS definition for minimum safety distance and 2.499m/s^2 for maximum comfortable deceleration have been applied.

• @Igata-ctrl got the information in Simulation working group meeting that the resource allocated to Scenario Simulation v2 on the cloud may not be sufficient and it can make the simulation result not trustworthy enough. We may need to analyze the Simulation Log file to see if there are many waring messages in there when we get unexpected simulation result.

• @Igata-ctrl to investigate if we can use real time speed of Ego and Npc to calculate RSS Safe Distance more accurately.

• Discussion (RSS distance and comfortable deceleration) pages have been added to the Use cases for Dense Urban Area 2024 presentation.

• Reached a consensus to use RSS definition for minimum safety distance and 2.499m/s^2 for maximum comfortable deceleration for low speed shuttle.

• Discussion of the condition/threshold setting so that we can better track the Autoware maturity through scenario simulation.

• After the discussion, it turned out "min_acc" parameter should better be reverted to original value of -1.5 (this week).

• Current discussion with Safety Pool

  • [Map] Successfully converted a sample OpenDRIVE map from Safety Pool database to Lanelet2, but we have following issues. Though Mohammad (Deepen.ai/AWF Safety Assurance WG lead) proposed in May TSC that Autoware should consider accepting OpenDRIVE maps in a direct fashion as it solves all these issues, given that we may need to map-less autonomy, it may not worth switching map format from now.
    • Speed limit, lane change flag and LHT/RHT are lost through the conversion.
    • The converted Lanelet2 map contains a lot of unnecessary points on line strings. (e.g. 16,000points on 1 km one way straight road)
  • [Scenario] Autoware requires EGO's destination in each scenario, while all of the Safety Pool scenarios do not have the destination. Though Safety Pool once suggested to run Autoware in the “roaming” mode which does not require the destination in the scenario, it sounds unrealistic for Autoware to operate without destination as the destination is almost always an important part of the use case/scenario intention. (e.g. turning left/right).
    • In the Safety Pool Webinar, they demonstrated Autoware running with Safety Pool scenario, but they say that it requires some special settings for it and it is not scalable.
  • [Scenario file format] Though both Safety Pool and Autoware (Scenario Simulator v2) use OpenSCENARIO v1.x (a.k.a. OpenSCENARIO XML) standard, Autoware (Scenario Simulator v2) uses ”.yaml” file to better handle the parameters, while Safety Pool uses “.xosc” file, which is the default of the OpenSCENRIO standard. If the number of scenarios is not too big, we might be able to convert them from .xosc to .yaml format manually. (Currently, no automated converter is available.) If the scenario simulator v2 is installed on-premise, it will accept the .xosc file in a direct fashion.
  • [Scenario availability] After signing up for the Safety Pool database, the number of the accessible/visible scenarios are limited. 32,588 scenarios are visible but most of the scenarios are marked with "access restricted" and only 1957 scenarios are accessible. The total number of the scenario which Safety Pool claims is one million. Still waiting for an answer from Safety Pool

Action Item

Documents

• The bus ODD use case list which was assigned to the members is here.

• The bus ODD use case list has been also uploaded here to make it visible to anyone on the net.

• The ODD working group shared document folder is here. [Restored!]

• Discussions and Q&As in AWF Discord ODD WG channel are also encouraged.

Tools

• [Autoware Evaluator (CI/CD pipeline)]

  • Cloud based DevOps (integration of the development tools including the scenario editor and the scenario simulator below)
  • A product from TIER IV and offered for the official Autoware Foundation projects like Cargo Delivery and Public Road Bus, etc.
  • The user guide is available TIER IV document site
  • As it consumes AWS resource, the (batch) weekly execution of the scenario simulation is managed by the Software/ODD WG leads

• [Scenario Simulator V2 (Scenario testing framework)]

  • Stand alone scenario simulation tool
  • An OSS from TIER IV freely available for any Autoware developer/researcher
  • The documentation of the Scenario Testing Framework (open sourced from Tier IV) is on GitHub

• The GUI Scenario editor

  • Web based GUI scenario editor freely available from TIER IV
  • You can create and export scenarios with this web interface
  • The user guide is available TIER IV document site

• [TIER IV account]

  • The working group members who are interested in creating and testing scenarios are advised to create a free TIER IV account here.
  • Once you have created your account, please let @Igata-ctrl know the (long) User ID which appears on your login page. After @Igata-ctrl registered you to the AWF group in the Evaluator (CI/CD pipeline), you can go to AWF Autoware Evaluator page to see the simulation results, create/edit scenarios, etc.
  • If you already have a TIER IV account, your 4-digit User ID continues to work, so you do not need to register to TIER IV account again.

Administrative

• The two meetings on December 25th (Christmas day) and January 1st (New Year day) have been cancelled.

• Please check the ODD WG wiki page.

• Recurring weekly meetings have been scheduled. Please check Autoware Foundation events calendar and add this calendar to your own Google calendar by clicking the right bottom button and/or add your contact to ODD WG invitation group to receive invitations for future meetings.

• ODD WG meetings are held weekly in the following single time slot.

  • 7:00am, Monday (PST) / 6:00am, Monday (PDT) US Pacific Time
  • 10:00am, Monday (EST) / 9:00am, Monday (EDT) US Eastern Time
  • 4:00pm, Monday (CEST) / 3:00pm, Monday (CEDT) Poland time
  • 5:00pm, Monday (TRT) Turkey time
  • 10:00pm, Monday (CST) Taiwan time
  • 11:00pm, Monday (JST) Japan Time