Update benchmark numbers (#411)

Co-authored-by: Akihiro Nitta <[email protected]>
pyg-team · Oct 14, 2024 · 8996b14 · 8996b14
1 parent d1f63cd
commit 8996b14
Showing 1 changed file with 13 additions and 13 deletions.
diff --git a/benchmark/README.md b/benchmark/README.md
@@ -154,19 +154,19 @@ Experimental setting: 20 Optuna search trials for XGBoost and CatBoost. 5 Optuna
 
 Experimental setting: 20 Optuna search trials for XGBoost, CatBoost and LightGBM. 3 Optuna search trials and 10 epochs training for deep learning models.
 
-|                     | dataset_0             |
-| :------------------ | :-------------------- |
-| XGBoost             | 0.966±0.000 (19327s)  |
-| CatBoost            | 0.971±0.000 (223s)    |
-| LightGBM            | **0.965±0.000 (67s)** |
-| Trompt              | 0.970±0.000 (12358s)  |
-| ResNet              | 0.970±0.000 (672s)    |
-| MLP                 | 0.973±0.000 (223s)    |
-| FTTransformerBucket | 0.970±0.000 (2071s)   |
-| ExcelFormer         | 0.969±0.000 (1785s)   |
-| FTTransformer       | 0.971±0.000 (2918s)   |
-| TabNet              | 0.970±0.000 (323s)    |
-| TabTransformer      | 0.984±0.000 (318s)    |
+|                     | dataset_0             | dataset_1             | dataset_2            | dataset_3             | dataset_4             | dataset_5             |
+| :------------------ | :-------------------- | :-------------------- | :------------------- | :-------------------- | :-------------------- | :-------------------- |
+| XGBoost             | 0.966±0.000 (19327s)  |                       |                      |                       |                       |                       |
+| CatBoost            | 0.971±0.000 (223s)    |                       |                      |                       |                       |                       |
+| LightGBM            | **0.965±0.000 (67s)** |                       |                      |                       |                       |                       |
+| Trompt              | 0.970±0.000 (12358s)  | Too slow\*            | Too slow\*           | Too slow\*            | 0.796±0.001 (157380s) | 0.799±0.001 (55577s)  |
+| ResNet              | 0.969±0.000 (5800s)   | 0.976±0.000 (57298s)  | 0.655±0.000 (4612s)  | 0.880±0.000 (20101s)  | 0.757±0.000 (53493s)  | 0.772±0.001 (3906s)   |
+| MLP                 | 0.973±0.000 (223s)    | 0.980±0.000 (1104s)   | 0.681±0.002 (91s)    | 0.908±0.001 (529s)    | 0.792±0.001 (680s)    | 0.810±0.002 (117s)    |
+| FTTransformerBucket | 0.970±0.000 (2071s)   | 0.977±0.000 (10953s)  | 0.776±0.004 (1608s)  | 0.896±0.002 (33883s)  | 0.785±0.002 (145035s) | 0.808±0.003 (10407s)  |
+| ExcelFormer         | 0.969±0.000 (1785s)   | 0.978±0.001 (47745s)  | 0.649±0.000 (15888s) | Too slow\*            | Too slow\*            | 0.782±0.001 (14729s)  |
+| FTTransformer       | 0.969±0.000 (25604s)  | 0.975±0.000 (219716s) | 0.670±0.004 (17119s) | 0.888±0.002 (323939s) | Too slow\*            | 0.788±0.001 (103434s) |
+| TabNet              | 0.968±0.001 (3177s)   | 0.974±0.000 (44035s)  | 0.655±0.001 (2679s)  | 0.891±0.001 (6800s)   | 0.767±0.001 (6242s)   | 0.784±0.001 (2307s)   |
+| TabTransformer      | 0.984±0.000 (2813s)   | 0.986±0.000 (13457s)  | 0.713±0.002 (3767s)  | 0.895±0.001 (5679s)   | 0.764±0.001 (3763s)   | 0.781±0.000 (3540s)   |
 
 ### `task_type: multiclass_classification`