-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
E2e bq test neeraj #18
base: develop
Are you sure you want to change the base?
Conversation
testUpsertOperationWithDedupeSourceData() testUpdateOperationWithDedupeSourceData() _ need to work on table creation hooks.
This reverts commit b5895b3.
testUpsertOperationWithDedupeSourceData() testUpdateOperationWithDedupeSourceData() _ need to work on table creation hooks.
BQ New additional scenarios added.
@@ -161,7 +161,7 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data | |||
Then Enter BigQuery sink property table name | |||
Then Toggle BigQuery sink property truncateTable to true | |||
Then Toggle BigQuery sink property updateTableSchema to true | |||
Then Enter BigQuery sink property partition field "bqPartitionFieldTime" | |||
Then Enter BigQuery sink property partition field "transaction_date" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Revert this change. This is I believe an existing code.
|
||
|
||
@BQ_UPSERT_SOURCE_TEST @BQ_UPSERT_SINK_TEST | ||
Scenario: Verify scenario form BigQuery To to ensure that upsert operations are performed without updating the destination table |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rephrase the scenario outline and use the dedupe word also in the scenario to make it more undestandable.
When Select plugin: "BigQuery" from the plugins list as: "Sink" | ||
Then Connect plugins: "BigQuery" and "BigQuery2" to establish connection | ||
Then Navigate to the properties page of plugin: "BigQuery" | ||
And Enter input plugin property: "referenceName" with value: "Reference" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have to use the use connection step in all the scenarios as per ITN class. For more info we can connect on this
Then Enter input plugin property: "dataset" with value: "dataset" | ||
Then Enter input plugin property: "table" with value: "bqTargetTable" | ||
And Select radio button plugin property: "operation" with value: "upsert" | ||
Then Enter Value for plugin property table key : "relationTableKey" with values: "string_value" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Provide the values in plugin parameter file.Dont hardcode it here.
And Select radio button plugin property: "operation" with value: "upsert" | ||
Then Enter Value for plugin property table key : "relationTableKey" with values: "string_value" | ||
Then Select dropdown plugin property: "dedupeBy" with option value: "DESC" | ||
Then Enter key for plugin property: "dedupeBy" with values: "float_value" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Provide the values in plugin parameter file.Dont hardcode it here.
Then Enter Value for plugin property table key : "relationTableKey" with values: "string_value" | ||
Then Select dropdown plugin property: "dedupeBy" with option value: "DESC" | ||
Then Enter key for plugin property: "dedupeBy" with values: "float_value" | ||
Then Click plugin property: "updateTableSchema" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Scenario is for without updating the table schema. Remove this step.
Then Wait till pipeline is in running state | ||
Then Open and capture logs | ||
Then Verify the pipeline status is "Succeeded" | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove the extra lines.
|
||
|
||
@BQ_UPDATE_SOURCE_TEST @BQ_UPDATE_SINK_TEST | ||
Scenario: Verify form BigQuery To to ensure that update operations are performed and the ensure that the duplicate entries has been removed to sink. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rephrase this scenario, its not giving the info properly. Update as per ITN class
Then Enter input plugin property: "dataset" with value: "dataset" | ||
Then Enter input plugin property: "table" with value: "bqTargetTable" | ||
And Select radio button plugin property: "operation" with value: "update" | ||
Then Enter Value for plugin property table key : "relationTableKey" with values: "string_value" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Value should be plugin parameter file.
And Select radio button plugin property: "operation" with value: "update" | ||
Then Enter Value for plugin property table key : "relationTableKey" with values: "string_value" | ||
Then Select dropdown plugin property: "dedupeBy" with option value: "DESC" | ||
Then Enter key for plugin property: "dedupeBy" with values: "float_value" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Value should be plugin parameter file.
Then Wait till pipeline is in running state | ||
Then Open and capture logs | ||
Then Verify the pipeline status is "Succeeded" | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove extra lines.
When Select plugin: "BigQuery" from the plugins list as: "Sink" | ||
Then Connect plugins: "BigQuery" and "BigQuery2" to establish connection | ||
Then Navigate to the properties page of plugin: "BigQuery" | ||
And Enter input plugin property: "referenceName" with value: "Reference" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Scenario need to be created using use connection as per ITN class
Then Open and capture logs | ||
Then Verify the pipeline status is "Succeeded" | ||
Then Validate the values of records transferred to BQ sink is equal to the values from source BigQuery table | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove extra lines
Given Open Datafusion Project to configure pipeline | ||
When Expand Plugin group in the LHS plugins list: "Source" | ||
When Select plugin: "BigQuery" from the plugins list as: "Source" | ||
When Expand Plugin group in the LHS plugins list: "Sink" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have to re-write all the scenarios as per use connection...
|
||
|
||
@BQ_TIME_STAMP_SOURCE_TEST @BQ_SINK_TEST | ||
Scenario: Verify record insert from source BigQuery plugin with partition type Time (Date/timestamp/datetime). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rephrase all the scenarios as per ITN class to make them more understandable.Take reference from ITN class or from alreday raised PR of Bigquery
Then Wait till pipeline is in running state | ||
Then Open and capture logs | ||
Then Verify the pipeline status is "Succeeded" | ||
#Then Validate the values of records transferred to BQ sink is equal to the values from source BigQuery table |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why this step is commented? Remove extra lines.
|
||
|
||
@BQ_INSERT_SOURCE_TEST @BQ_INSERT_SECOND_SOURCE_TEST @BQ_SINK_TEST | ||
Scenario: Verify BigQuery With Different Schema RecordName |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rephrase the scenario.
Then Run the Pipeline in Runtime | ||
Then Wait till pipeline is in running state | ||
Then Open and capture logs | ||
Then Verify the pipeline status is "Succeeded" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove extra lines.
Then Open and capture logs | ||
Then Verify the pipeline status is "Succeeded" | ||
|
||
# Then Validate the values of records transferred to BQ sink is equal to the values from source BigQuery table |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why this step is commented?
@@ -920,4 +921,164 @@ public static void createSourceBQUpdateTable() throws IOException, InterruptedEx | |||
PluginPropertyUtils.addPluginProp(" bqTargetTable", bqTargetTable); | |||
BeforeActions.scenario.write("BQ Target Table " + bqTargetTable + " updated successfully"); | |||
} | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Already created hooks for existing source and sink table can be used from the already raised PR for BQ, if we have the same schema int,float,string and boolean.So the duplicity can be avoided
|
||
PluginPropertyUtils.addPluginProp(" bqSourceTable", bqSourceTable); | ||
BeforeActions.scenario.write("BQ Source Table " + bqSourceTable + " updated successfully"); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove extra lines
BigQueryClient.getSoleQueryResult("INSERT INTO `" + datasetName + "." + bqSourceTable + "` " + | ||
"(Address, transaction_date, Firstname)" + | ||
"VALUES" + "('Agra', '2021-02-20', 'Neeraj')," + | ||
"('Noida', '2021-02-21',''),"+"('Gudgaon', '2021-02-22', 'singhal')"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Spelling for Gurgaon is incorrect.
} | ||
PluginPropertyUtils.addPluginProp("bqSourceTable", bqSecondSourceTable); | ||
BeforeActions.scenario.write("BQ Source Table " + bqSecondSourceTable + " created successfully"); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove extra lines.
io.cdap.e2e.utils.BigQueryClient.getSoleQueryResult("INSERT INTO `" + datasetName + "." + bqTargetTable + "` " + | ||
"(string_value, int_value, float_value,boolean_value)" + | ||
"VALUES" + "('string_0', 0, 0,true)," + | ||
"('string_1', 10, 1.1,false)"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove extra lines
"(string_value, int_value, float_value,boolean_value)" + | ||
"VALUES" + "('string_1', 1, 0.1,true)," + | ||
"('string_1', 2, 0.2,false)," + | ||
"('string_3', 3, 0.3,false)"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove extra lines
|
||
io.cdap.e2e.utils.BigQueryClient.getSoleQueryResult("create table `" + datasetName + "." + bqSourceTable + "` " + | ||
"(string_value STRING, int_value INT64, float_value FLOAT64,boolean_value BOOL)"); | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove extra lines
PluginPropertyUtils.addPluginProp("bqSourceTable", bqSourceTable); | ||
BeforeActions.scenario.write("BQ source table name - " + bqSourceTable); | ||
BigQueryClient.getSoleQueryResult("create table `" + datasetName + "." + bqSourceTable + "` " + | ||
"(Address STRING, transaction_date DATE, Firstname STRING)"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As per ITN class data should be like: * {"string_value":"string_1","int_value":1,"float_value":0.1,"date_value":"2000-01-19"}. You can change the data values but, fields should be there of type string,int,float and date
fc2efdf
to
5f4f137
Compare
No description provided.