E2e bq test neeraj #18
Conversation
testUpsertOperationWithDedupeSourceData() testUpdateOperationWithDedupeSourceData() _ need to work on table creation hooks.
This reverts commit b5895b3.
testUpsertOperationWithDedupeSourceData() testUpdateOperationWithDedupeSourceData() _ need to work on table creation hooks.
BQ New additional scenarios added.
| @@ -161,7 +161,7 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data | |||
| Then Enter BigQuery sink property table name | |||
| Then Toggle BigQuery sink property truncateTable to true | |||
| Then Toggle BigQuery sink property updateTableSchema to true | |||
| Then Enter BigQuery sink property partition field "bqPartitionFieldTime" | |||
| Then Enter BigQuery sink property partition field "transaction_date" | |||
There was a problem hiding this comment.
Revert this change. This is I believe an existing code.
|
|
||
|
|
||
| @BQ_UPSERT_SOURCE_TEST @BQ_UPSERT_SINK_TEST | ||
| Scenario: Verify scenario form BigQuery To to ensure that upsert operations are performed without updating the destination table |
There was a problem hiding this comment.
Rephrase the scenario outline and use the dedupe word also in the scenario to make it more undestandable.
| When Select plugin: "BigQuery" from the plugins list as: "Sink" | ||
| Then Connect plugins: "BigQuery" and "BigQuery2" to establish connection | ||
| Then Navigate to the properties page of plugin: "BigQuery" | ||
| And Enter input plugin property: "referenceName" with value: "Reference" |
There was a problem hiding this comment.
We have to use the use connection step in all the scenarios as per ITN class. For more info we can connect on this
| Then Enter input plugin property: "dataset" with value: "dataset" | ||
| Then Enter input plugin property: "table" with value: "bqTargetTable" | ||
| And Select radio button plugin property: "operation" with value: "upsert" | ||
| Then Enter Value for plugin property table key : "relationTableKey" with values: "string_value" |
There was a problem hiding this comment.
Provide the values in plugin parameter file.Dont hardcode it here.
| And Select radio button plugin property: "operation" with value: "upsert" | ||
| Then Enter Value for plugin property table key : "relationTableKey" with values: "string_value" | ||
| Then Select dropdown plugin property: "dedupeBy" with option value: "DESC" | ||
| Then Enter key for plugin property: "dedupeBy" with values: "float_value" |
There was a problem hiding this comment.
Provide the values in plugin parameter file.Dont hardcode it here.
| Then Enter Value for plugin property table key : "relationTableKey" with values: "string_value" | ||
| Then Select dropdown plugin property: "dedupeBy" with option value: "DESC" | ||
| Then Enter key for plugin property: "dedupeBy" with values: "float_value" | ||
| Then Click plugin property: "updateTableSchema" |
There was a problem hiding this comment.
Scenario is for without updating the table schema. Remove this step.
| Then Wait till pipeline is in running state | ||
| Then Open and capture logs | ||
| Then Verify the pipeline status is "Succeeded" | ||
|
|
There was a problem hiding this comment.
Remove the extra lines.
|
|
||
|
|
||
| @BQ_UPDATE_SOURCE_TEST @BQ_UPDATE_SINK_TEST | ||
| Scenario: Verify form BigQuery To to ensure that update operations are performed and the ensure that the duplicate entries has been removed to sink. |
There was a problem hiding this comment.
Rephrase this scenario, its not giving the info properly. Update as per ITN class
| Then Enter input plugin property: "dataset" with value: "dataset" | ||
| Then Enter input plugin property: "table" with value: "bqTargetTable" | ||
| And Select radio button plugin property: "operation" with value: "update" | ||
| Then Enter Value for plugin property table key : "relationTableKey" with values: "string_value" |
There was a problem hiding this comment.
Value should be plugin parameter file.
| And Select radio button plugin property: "operation" with value: "update" | ||
| Then Enter Value for plugin property table key : "relationTableKey" with values: "string_value" | ||
| Then Select dropdown plugin property: "dedupeBy" with option value: "DESC" | ||
| Then Enter key for plugin property: "dedupeBy" with values: "float_value" |
There was a problem hiding this comment.
Value should be plugin parameter file.
| Then Wait till pipeline is in running state | ||
| Then Open and capture logs | ||
| Then Verify the pipeline status is "Succeeded" | ||
|
|
There was a problem hiding this comment.
Remove extra lines.
| When Select plugin: "BigQuery" from the plugins list as: "Sink" | ||
| Then Connect plugins: "BigQuery" and "BigQuery2" to establish connection | ||
| Then Navigate to the properties page of plugin: "BigQuery" | ||
| And Enter input plugin property: "referenceName" with value: "Reference" |
There was a problem hiding this comment.
Scenario need to be created using use connection as per ITN class
| Then Open and capture logs | ||
| Then Verify the pipeline status is "Succeeded" | ||
| Then Validate the values of records transferred to BQ sink is equal to the values from source BigQuery table | ||
|
|
There was a problem hiding this comment.
Remove extra lines
| Given Open Datafusion Project to configure pipeline | ||
| When Expand Plugin group in the LHS plugins list: "Source" | ||
| When Select plugin: "BigQuery" from the plugins list as: "Source" | ||
| When Expand Plugin group in the LHS plugins list: "Sink" |
There was a problem hiding this comment.
We have to re-write all the scenarios as per use connection...
|
|
||
|
|
||
| @BQ_TIME_STAMP_SOURCE_TEST @BQ_SINK_TEST | ||
| Scenario: Verify record insert from source BigQuery plugin with partition type Time (Date/timestamp/datetime). |
There was a problem hiding this comment.
Rephrase all the scenarios as per ITN class to make them more understandable.Take reference from ITN class or from alreday raised PR of Bigquery
| Then Wait till pipeline is in running state | ||
| Then Open and capture logs | ||
| Then Verify the pipeline status is "Succeeded" | ||
| #Then Validate the values of records transferred to BQ sink is equal to the values from source BigQuery table |
There was a problem hiding this comment.
Why this step is commented? Remove extra lines.
|
|
||
|
|
||
| @BQ_INSERT_SOURCE_TEST @BQ_INSERT_SECOND_SOURCE_TEST @BQ_SINK_TEST | ||
| Scenario: Verify BigQuery With Different Schema RecordName |
There was a problem hiding this comment.
Rephrase the scenario.
| Then Run the Pipeline in Runtime | ||
| Then Wait till pipeline is in running state | ||
| Then Open and capture logs | ||
| Then Verify the pipeline status is "Succeeded" |
There was a problem hiding this comment.
Remove extra lines.
| Then Open and capture logs | ||
| Then Verify the pipeline status is "Succeeded" | ||
|
|
||
| # Then Validate the values of records transferred to BQ sink is equal to the values from source BigQuery table |
There was a problem hiding this comment.
Why this step is commented?
| @@ -920,4 +921,164 @@ public static void createSourceBQUpdateTable() throws IOException, InterruptedEx | |||
| PluginPropertyUtils.addPluginProp(" bqTargetTable", bqTargetTable); | |||
| BeforeActions.scenario.write("BQ Target Table " + bqTargetTable + " updated successfully"); | |||
| } | |||
|
|
|||
There was a problem hiding this comment.
Already created hooks for existing source and sink table can be used from the already raised PR for BQ, if we have the same schema int,float,string and boolean.So the duplicity can be avoided
|
|
||
| PluginPropertyUtils.addPluginProp(" bqSourceTable", bqSourceTable); | ||
| BeforeActions.scenario.write("BQ Source Table " + bqSourceTable + " updated successfully"); | ||
| } |
There was a problem hiding this comment.
Remove extra lines
| BigQueryClient.getSoleQueryResult("INSERT INTO `" + datasetName + "." + bqSourceTable + "` " + | ||
| "(Address, transaction_date, Firstname)" + | ||
| "VALUES" + "('Agra', '2021-02-20', 'Neeraj')," + | ||
| "('Noida', '2021-02-21',''),"+"('Gudgaon', '2021-02-22', 'singhal')"); |
There was a problem hiding this comment.
Spelling for Gurgaon is incorrect.
| } | ||
| PluginPropertyUtils.addPluginProp("bqSourceTable", bqSecondSourceTable); | ||
| BeforeActions.scenario.write("BQ Source Table " + bqSecondSourceTable + " created successfully"); | ||
| } |
There was a problem hiding this comment.
Remove extra lines.
| io.cdap.e2e.utils.BigQueryClient.getSoleQueryResult("INSERT INTO `" + datasetName + "." + bqTargetTable + "` " + | ||
| "(string_value, int_value, float_value,boolean_value)" + | ||
| "VALUES" + "('string_0', 0, 0,true)," + | ||
| "('string_1', 10, 1.1,false)"); |
There was a problem hiding this comment.
Remove extra lines
| "(string_value, int_value, float_value,boolean_value)" + | ||
| "VALUES" + "('string_1', 1, 0.1,true)," + | ||
| "('string_1', 2, 0.2,false)," + | ||
| "('string_3', 3, 0.3,false)"); |
There was a problem hiding this comment.
Remove extra lines
|
|
||
| io.cdap.e2e.utils.BigQueryClient.getSoleQueryResult("create table `" + datasetName + "." + bqSourceTable + "` " + | ||
| "(string_value STRING, int_value INT64, float_value FLOAT64,boolean_value BOOL)"); | ||
|
|
There was a problem hiding this comment.
Remove extra lines
| PluginPropertyUtils.addPluginProp("bqSourceTable", bqSourceTable); | ||
| BeforeActions.scenario.write("BQ source table name - " + bqSourceTable); | ||
| BigQueryClient.getSoleQueryResult("create table `" + datasetName + "." + bqSourceTable + "` " + | ||
| "(Address STRING, transaction_date DATE, Firstname STRING)"); |
There was a problem hiding this comment.
As per ITN class data should be like: * {"string_value":"string_1","int_value":1,"float_value":0.1,"date_value":"2000-01-19"}. You can change the data values but, fields should be there of type string,int,float and date
NavdeepCS
force-pushed
the
develop
branch
4 times, most recently
from
fc2efdf
to
5f4f137
Compare
No description provided.