Minor changes for Open Sourcing.

PiperOrigin-RevId: 568780473

.github/workflows/pytest_and_autopublish.yml

@@ -28,27 +28,27 @@ jobs:
     - run: pip freeze
 
     # Run tests (in parallel)
-    - name: Run core tests
-      run: pytest -vv -n auto
-
-  # Auto-publish when version is increased
-  publish-job:
-    # Only try to publish if:
-    # * Repo is self (prevents running from forks)
-    # * Branch is `main`
-    if: |
-      github.repository == 'google-research/human-scene-transformer'
-      && github.ref == 'refs/heads/main'
-    needs: pytest-job  # Only publish after tests are successful
-    runs-on: ubuntu-latest
-    permissions:
-      contents: write
-    timeout-minutes: 30
-
-    steps:
-    # Publish the package (if local `__version__` > pip version)
-    - uses: etils-actions/pypi-auto-publish@v1
-      with:
-        pypi-token: ${{ secrets.PYPI_API_TOKEN }}
-        gh-token: ${{ secrets.GITHUB_TOKEN }}
-        parse-changelog: true
+    # - name: Run core tests
+    #  run: pytest -vv -n auto
+
+  # # Auto-publish when version is increased
+  # publish-job:
+  #   # Only try to publish if:
+  #   # * Repo is self (prevents running from forks)
+  #   # * Branch is `main`
+  #   if: |
+  #     github.repository == 'google-research/human-scene-transformer'
+  #     && github.ref == 'refs/heads/main'
+  #   needs: pytest-job  # Only publish after tests are successful
+  #   runs-on: ubuntu-latest
+  #   permissions:
+  #     contents: write
+  #   timeout-minutes: 30
+
+  #   steps:
+  #   # Publish the package (if local `__version__` > pip version)
+  #   - uses: etils-actions/pypi-auto-publish@v1
+  #     with:
+  #       pypi-token: ${{ secrets.PYPI_API_TOKEN }}
+  #       gh-token: ${{ secrets.GITHUB_TOKEN }}
+  #       parse-changelog: true

README.md

@@ -1,6 +1,6 @@
 # Human Scene Transformer
 
-![Human Scene Transformer](./images/hero.png)
+![Human Scene Transformer](./human_scene_transformer/images/hero.png)
 
 Anticipating the motion of all humans in dynamic environments such as homes and offices is critical to enable safe and effective robot navigation. Such spaces remain challenging as humans do not follow strict rules of motion and there are often multiple occluded entry points such as corners and doors that create opportunities for sudden encounters. In this work, we present a Transformer based architecture to predict human future trajectories in human-centric environments from input features including human positions, head orientations, and 3D skeletal keypoints from onboard in-the-wild sensory information. The resulting model captures the inherent uncertainty for future human trajectory prediction and achieves state-of-the-art performance on common prediction benchmarks and a human tracking dataset captured from a mobile robot adapted for the prediction task. Furthermore, we identify new agents with limited historical data as a major contributor to error and demonstrate the complementary nature of 3D skeletal poses in reducing prediction error in such challenging scenarios.
 
@@ -24,6 +24,17 @@ If you use this work please cite our paper
 ## Data
 
 ### JRDB
+
+We provide a extensive prep-processing pipeline to convert the JRDB dataset,
+JRDB was created as a detection and tracking dataset rather than a prediction
+dataset. To make the data suitable for a prediction task, we first extract the
+robot motion from the raw sensor data to account for the robot's motion.
+Further, on the JRDB training split we combine algorithmic detection with the
+ground truth labels from the tracking dataset to create authentic tracks as
+input and labels for HST.
+Note that we do not purely use the ground truth hand labeled tracks in the JRDB
+train dataset as we find them to be overly smoothed giving away the future human
+movement.
 To adapt the JRDB dataset for prediction please follow [this](/data) README.
 
 Make sure to adapt `<data_path>` in `config/<jrdb/pedestrians>/dataset_params.gin` accordingly.
@@ -38,17 +49,14 @@ Please download the raw data [here](https://github.com/StanfordASL/Trajectron-pl
 
 ### JRDB
 ```
-python train.py --model_base_dir=./model/jrdb  --gin_files=.config/jrdb/training_params.gin --gin_files=.config/jrdb/model_params.gin --gin_files=.config/jrdb/dataset_params.gin --gin_files=.config/jrdb/metrics.gin --dataset=JRDB
+python train.py --model_base_dir=./model/jrdb  --gin_files=./config/jrdb/training_params.gin --gin_files=./config/jrdb/model_params.gin --gin_files=./config/jrdb/dataset_params.gin --gin_files=./config/jrdb/metrics.gin --dataset=JRDB
 ```
 
 ### Pedestrians ETH/UCY
 ```
-python train.py --model_base_dir=./models/pedestrians_eth  --gin_files=.config/pedestrians/training_params.gin --gin_files=.config/pedestrians/model_params.gin --gin_files=.config/pedestrians/dataset_params.gin --gin_files=.config/pedestrians/metrics.gin --dataset=PEDESTRIANS
+python train.py --model_base_dir=./models/pedestrians_eth  --gin_files=..config/pedestrians/training_params.gin --gin_files=..config/pedestrians/model_params.gin --gin_files=./config/pedestrians/dataset_params.gin --gin_files=./config/pedestrians/metrics.gin --dataset=PEDESTRIANS
 ```
 
-## Checkpoints
-Coming soon!
-
 ---
 
 ## Evaluation
@@ -58,7 +66,47 @@ Coming soon!
 python jrdb/eval.py --model_path=./models/jrdb/ --checkpoint_path=./models/jrdb/ckpts/ckpt-30
 ```
 
+#### Keypoints Impact Evaluation
+```
+python jrdb/eval_keypoints.py --model_path=./models/jrdb/ --checkpoint_path=./models/jrdb/ckpts/ckpt-30
+```
+
+vs
+
+```
+python jrdb/eval_keypoints.py --model_path=./models/jrdb_no_keypoints/ --checkpoint_path=./models/jrdb_no_keypoints/ckpts/ckpt-30
+```
+
 ### Pedestrians ETH/UCY
 ```
 python pedestrians/eval.py --model_path=./models/pedestrians_eth/ --checkpoint_path=./models/pedestrians_eth/ckpts/ckpt-20
 ```
+
+---
+
+## Results
+
+Compared to the published paper we improved our data processing and fixed small
+bugs in this code release. If you compare against our method please use the
+following updated results.
+
+On the JRDB dataset with dataset options as set [here](/config/jrdb/dataset_params.py):
+
+|        | AVG  |  @ 1s | @ 2s |  @ 3s | @ 4s  |
+|--------|------|-------|------|-------|-------|
+| MinADE | 0.26 | 0.12  | 0.20 | 0.28  | 0.37  |
+| MinFDE | 0.45 | 0.21  | 0.39 | 0.56  | 0.71  |
+|  NLL   |-0.59 | -0.90 | -0.65| -0.08 | 0.32  |
+
+On the ETH/UCY Pedestrians Dataset:
+
+|        | ETH  | Hotel | Univ | Zara1 | Zara2 |  Avg  |
+|--------|------|-------|------|-------|-------|-------|
+| MinADE | 0.41 | 0.10  | 0.24 | 0.17  | 0.14  | 0.21  |
+| MinFDE | 0.73 | 0.14  | 0.44 | 0.30  | 0.24  | 0.37  |
+
+
+### Checkpoints
+You can download trained model checkpoints for both `JRDB` and `Pedestrians (ETH/UCY)` datasets [here]()(Coming Soon).
+
+To evaluate the pre-trained checkpoints you will have to adjust the path to the dataset in the respective `params/operative_config.gin` file.

human_scene_transformer/config/jrdb/dataset_params.gin

@@ -55,7 +55,7 @@ TEST_SCENES = ['clark-center-2019-02-28_1',
  'tressider-2019-04-26_3_test']
 
 
-JRDBDatasetParams.path = <dataset_path>
+JRDBDatasetParams.path = '<dataset_path>'
 
 JRDBDatasetParams.train_scenes = %TRAIN_SCENES
 JRDBDatasetParams.eval_scenes = %TEST_SCENES

human_scene_transformer/config/pedestrians/dataset_params.gin

@@ -1,4 +1,4 @@
-PedestriansDatasetParams.path = <dataset_path>
+PedestriansDatasetParams.path = '<dataset_path>'
 PedestriansDatasetParams.dataset = 'eth'
 PedestriansDatasetParams.train_config = 'train'  # train, trainval
 PedestriansDatasetParams.eval_config = 'val' # val, test

human_scene_transformer/data/README.md

@@ -9,7 +9,7 @@
 5. Download and extract `Train Detections` from the JRDB 2019 section to `<data_path>/detections`.
 
 ## Get the Leaderboard Test Set Tracks
-3. Download and extract the best leaderboard  [3D tracking result](https://jrdb.erc.monash.edu/leaderboards/download/1680) to `<data_path>/test_dataset/labels/raw_leaderboard/`.
+Download and extract this leaderboard  [3D tracking result](https://jrdb.erc.monash.edu/leaderboards/download/1605) to `<data_path>/test_dataset/labels/raw_leaderboard/`. Such that you have `<data_path>/test_dataset/labels/raw_leaderboard/00XX.txt` This is the best available leaderboard tracker at the time the code was developed.
 
 ## Get the Robot Odometry Preprocessed Keypoints