Fix cost (for methods cromwell), add explicit example (#278)

* Fix cost (for methods cromwell), add explicit example * Update cost unit test to query export_time * One more that I missed * Modify README for cost --------- Co-authored-by: bshifaw <[email protected]>
broadinstitute · Apr 2, 2024 · 9c1b901 · 9c1b901
1 parent 4a485d4
commit 9c1b901
Show file tree

Hide file tree

Showing 4 changed files with 17 additions and 4 deletions.
diff --git a/README.md b/README.md
@@ -110,6 +110,19 @@ Cromshell is a CLI for submitting workflows to a Cromwell server and monitoring/
      * Requires the `bq_cost_table` key to exist in the cromshell 
        configuration file and have a value equal to the BigQuery cost table 
        for your GCP billing project.
+       * For example, your `~/.cromshell/cromshell_config.json` should contain:
+         ```json
+         {
+           "cromwell_server": "<cromwell_server_url>",
+           "requests_timeout": 5,
+           "bq_cost_table": "<table_name>"
+         }
+         ```
+         where `<table_name>` can be found by navigating to [BigQuery](https://console.cloud.google.com/bigquery), selecting the appropriate google project, and locating the table containing cost information.
+
+         <img src="developer_docs/img/bq_cost_table.png" alt="BigQuery example image" width="500"/>
+
+         Clicking on the table and opening the "DETAILS" tab, you'll find the exact path to the table in the "Table ID" section. Everything after the google project name (after the first `.`) should be included in `<table_name>`.
      * `-c/--color` Color outliers in task level cost results.
      * `-d/--detailed` Get the cost for a workflow at the task level.
 

diff --git a/developer_docs/img/bq_cost_table.png b/developer_docs/img/bq_cost_table.png
diff --git a/src/cromshell/cost/command.py b/src/cromshell/cost/command.py
@@ -182,15 +182,15 @@ def create_bq_query(detailed: bool, bq_cost_table: str) -> str:
                 AND task.key LIKE "wdl-task-name"
                 AND wfid.key LIKE "cromwell-workflow-id"
                 AND wfid.value like @workflow_id
-                AND partition_time BETWEEN @start_date AND @end_date
+                AND export_time BETWEEN @start_date AND @end_date
                 GROUP BY 1,2,3
                 ORDER BY 4 DESC
                 """
     else:
         return f"""
                 SELECT sum(cost) as cost
                 FROM {bq_cost_table}, UNNEST(labels)
-                WHERE value LIKE @workflow_id AND partition_time BETWEEN @start_date AND @end_date
+                WHERE value LIKE @workflow_id AND export_time BETWEEN @start_date AND @end_date
                 """
 
 

diff --git a/tests/unit/test_cost.py b/tests/unit/test_cost.py
@@ -23,7 +23,7 @@ class TestCost:
                 AND task.key LIKE "wdl-task-name"
                 AND wfid.key LIKE "cromwell-workflow-id"
                 AND wfid.value like @workflow_id
-                AND partition_time BETWEEN @start_date AND @end_date
+                AND export_time BETWEEN @start_date AND @end_date
                 GROUP BY 1,2,3
                 ORDER BY 4 DESC
                 """,
@@ -34,7 +34,7 @@ class TestCost:
                 """
                 SELECT sum(cost) as cost
                 FROM cost:table1, UNNEST(labels)
-                WHERE value LIKE @workflow_id AND partition_time BETWEEN @start_date AND @end_date
+                WHERE value LIKE @workflow_id AND export_time BETWEEN @start_date AND @end_date
                 """,
             ],
         ],