2022-05-25 Simulation Working Group Meeting

[sglee-morai]

Useful Links

Previous Meeting Minutes

2022-05-11 Simulation Working Group meeting

Issue Board

https://github.com/orgs/autowarefoundation/projects/7/views/1

Wiki Pages

• Bus ODD E2E Simualtor Requirements

Recording from the meeting

https://drive.google.com/drive/folders/1IX3h2MLvs3TTncQrnvm3kCNRmbXHyp12

Agenda Summary

• Reviewing in-progress issues
• Bus ODD E2E Simualtor Requirements (wiki) (discussion)
  • Test Scenarios

Discussions

Reviewing in-progress issues - Metric Calculation Node Development

• The pull request is created
  • Two evaluators are created (localization, statistics of velocity)
  • In progress: refactoring, creating base evaluator which can be used for creating another evaluator, creating segmentation metric evaluator

Reviewing in-progress issues - Define Test Procedures to check Simulator's Compatibility with Autoware

• Initial version of the test code for half of the test cases is created. The plan is to create a PR, ask for feedback before working on the others.

Reviewing in-progress issues - Creating a Simulation Environment (ITRI Campus)

• Two types of the map will be created
  • map-type1: Normal Environment
  • map-type2: PCD-based Environment
• The plan is to share the road-only initial version first

Reviewing in-progress issues - Creating a Bus Model (ITRI Bus)

• TierIV and Robotec is jointly working on creating the model, an update can be found in the issue link
• Testing for vehicle model dynamics or behavior can be done in another map (other than ITRI campus map)

Bus ODD E2E Simulator Requirements

Important links: wiki page, discussion

About wiki page,

• The introduction part will be added later, the usage of the document, for example.
• For the vehicle, once modeling is finished, pictures or other stuff can be added
• For the environment, it is the same story, pictures or links to the other detailed stuff can be added later.
• Interface is being updated, which is discussed in the Architecture, therefore the requirement for the simulator will be updated accordingly.
  • @sglee-morai is attending the Autonomy Software WG and Architecture SW meetings to keep track of how the message type will be changed in the future.
• About sensor
  • For usage of radar, it's still good to keep it optional. It is planned to use one to improve the velocity prediction of detected objects but it is still in progress
  • 8 camera > this can be a problem for a simulator
  • Cameras are not used for localization. (For localization, only lidars are used), but cameras are used only for object detection.
  • When Autoware uses multiple cameras at the same time, separate machines were used to process images and send detected results to the main PC
  • For simulation usage, users wouldn't have multiple PCs for processing multiple images from camera models normally, so it would be better to start with only one camera.
  • Is it okay to reduce the resolution of the camera? a real-world camera is rather high-resolution for the simulator (picture is below, pdf link)
    
  • One thing to consider for this: Whether to run the simulator on the same PC that runs the Autoware or to have separate PCs
  • Many Autoware users would have only one PC, so it is needed to specify what performance (resolution for 1 camera) can be achieved for a certain PC spec, and recommended PC spec.
  • If a single PC configuration limits the performance too much, we can consider not using a camera model for the simulation and replace it by ground truth information (object detection information)
• Virtual/Ground Truth Sensors
  • It would be a key feature for an E2E Simulator
    • MORAI SIM: Drive has already similar features, so changing the output format slightly is the only task left.
    • TierIV SIM is also planning to develop a feature to publish GT data as well
  • Thus, the main discussion point is about the format
    • Evaluation nodes require GT data as an input, so the data format can be suggested by Dominik.
    • Making the GT data format the same as its real-world counterpart is another idea
      • Benefit: It becomes easy to replace a certain feature in the Autoware with the simulator's GT feature, making testing & developing easier
    • However, simulator providers can just define a standard GT message format, regardless of the data message format between each node in the Autoware. Since one can create a node that subscribes to the GT message and publish in another format anyway.
• Main usage of an E2E Simulator
  • Localization: won't be too difficult to make a simulator provide features for that since lidars are only used.
    • Also, the localization evaluator can be used for the test.
  • Perception: performance analysis is first required to consider detailed testing configuration or user story

Action Items

• To keep working on existing issues
• A new issue (planned): doing an initial benchmark - to see sensor performance w.r.t. to few sensor configurations, map details (ITRI campus), scenarios
• To discuss Virtual / Ground Truth Sensors (link)
• To discuss detailed use cases of an E2E simulator (e.g. Localization, Perception, and E2E Testing), test configurations, and scenarios for each use case. (link)