Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/add error handling for bqio #30081

Merged
merged 17 commits into from
Feb 21, 2024
Merged

Feature/add error handling for bqio #30081

merged 17 commits into from
Feb 21, 2024

Conversation

johnjcasey
Copy link
Contributor

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@johnjcasey
Copy link
Contributor Author

Run Java PostCommit

@github-actions github-actions bot added the build label Jan 23, 2024
@johnjcasey
wire error handling into storage write, update check to not log a war…
983b135 
…ning if errorhandler is used but error output isn't consumed.
@johnjcasey johnjcasey marked this pull request as ready for review February 9, 2024 16:45
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 9, 2024

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

@Abacn
Copy link
Contributor

Abacn commented Feb 14, 2024

"beam_PostCommit_Java" runs non-IO integration tests on direct runner. Would you mind trigger Dataflow PostCommit (which runs GCP IO integraiton tests on Dataflow) by adding file namedbeam_PostCommit_Java_DataflowV1.json ?

@Abacn
Copy link
Contributor

Abacn commented Feb 14, 2024 •
edited
Loading

I went through the changes and mostly looks straightforward and looks good to me. Due to the large number of line of changes I may missed something. Please run postcommit tests and see we do not break things (commented above). Also consider run Python xlang GCPIO Dataflow (trigger path beam_PostCommit_Python_Xlang_Gcp_Dataflow.json)

ahmedabu98
ahmedabu98 reviewed Feb 15, 2024
View reviewed changes
Copy link
Contributor

@ahmedabu98 ahmedabu98 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, left some comments.
Nothing blocking but there are a few nits

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java Show resolved Hide resolved
...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java Outdated Show resolved Hide resolved
...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java Outdated Show resolved Hide resolved
...orm/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageStreamBundleSource.java Show resolved Hide resolved
...-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageStreamSource.java Show resolved Hide resolved
...ud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiConvertMessages.java Outdated Show resolved Hide resolved
...ud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiConvertMessages.java Show resolved Hide resolved
...le-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteBundlesToFiles.java Outdated Show resolved Hide resolved
...le-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteBundlesToFiles.java Outdated Show resolved Hide resolved
@Abacn Abacn mentioned this pull request Feb 15, 2024
Merged
3 tasks
@Abacn
Copy link
Contributor

Abacn commented Feb 15, 2024

There are integration test failed:

org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOStorageReadIT > testBigQueryStorageRead1MErrorHandlingArrow FAILED
    java.lang.RuntimeException at BigQueryIOStorageReadIT.java:162

org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOStorageReadIT > testBigQueryStorageRead1MErrorHandlingAvro FAILED
    java.lang.RuntimeException at BigQueryIOStorageReadIT.java:162

See https://github.com/apache/beam/runs/21587463679 for details

@johnjcasey
Copy link
Contributor Author

"beam_PostCommit_Java" runs non-IO integration tests on direct runner. Would you mind trigger Dataflow PostCommit (which runs GCP IO integraiton tests on Dataflow) by adding file namedbeam_PostCommit_Java_DataflowV1.json ?

Turns out the tests hadn't gotten run, and I missed something. Thanks for the catch!

@johnjcasey
Merge remote-tracking branch 'origin/master' into feature/add-error-h…
a2e4b50 
…andling-for-bqio

# Conflicts:
#	sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTranslation.java
ahmedabu98
ahmedabu98 reviewed Feb 17, 2024
View reviewed changes
Copy link
Contributor

@ahmedabu98 ahmedabu98 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, left a couple non-blocking comments

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java Outdated Show resolved Hide resolved
...oud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageQueryIT.java Outdated Show resolved Hide resolved
@github-actions GitHub Actions
Copy link
Contributor

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @damondouglas for label java.
R: @damccorm for label build.
R: @damondouglas for label io.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@johnjcasey johnjcasey merged commit 6a8c27e into apache:master Feb 21, 2024
24 checks passed
@RustedBones RustedBones mentioned this pull request Sep 4, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ahmedabu98 ahmedabu98 ahmedabu98 left review comments

@Abacn Abacn Awaiting requested review from Abacn

Assignees
No one assigned
Labels
build dataflow gcp io java Next Action: Reviewers runners
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@johnjcasey @Abacn @ahmedabu98