Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable drain operation for flink jobs with requiresStableInput annotation #28567

Closed

Commits on Sep 20, 2023

  1. skip watermark hold check for requiresStableInput operator

    Kanishk Karanawat committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    911550e View commit details
    Browse the repository at this point in the history
  2. correct log message format

    Kanishk Karanawat committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    9107277 View commit details
    Browse the repository at this point in the history
  3. apply spotless

    Kanishk Karanawat committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    e6ff225 View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2023

  1. flush buffer when requiresStableInput is set

    Kanishk Karanawat committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    db67dfa View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2023

  1. re-organize imports

    Kanishk Karanawat committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    65b2dcf View commit details
    Browse the repository at this point in the history
  2. add flag in FlinkPipelineOptions to allow draining for pipelines with…

    … RequiresStableInput
    Kanishk Karanawat committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    4ca8d3a View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    e5626b7 View commit details
    Browse the repository at this point in the history

Commits on Oct 19, 2023

  1. apply spotless again

    Kanishk Karanawat committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    897d5a3 View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2023

  1. Implement java exponential histograms (apache#28903) (apache#28995)

    * Implement java exponential histograms (apache#28903)
    
    * Address comments
    
    * Address comments
    JayajP authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    10b8d10 View commit details
    Browse the repository at this point in the history
  2. Adjust a schedule to run workflows in subgroups (apache#29049)

    * cron time update
    
    * cron fix
    magicgoody authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    8c4ce51 View commit details
    Browse the repository at this point in the history
  3. Disable comment trigger for Performance and Load Tests (apache#29035)

    * reference PR 28915 issue_comment removed
    
    * post commit issue_comment fix
    
    * revert changes for postcommit
    magicgoody authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    0aa9020 View commit details
    Browse the repository at this point in the history
  4. [Security] Bump org.json:json due to CVE-2022-45688 (apache#28962)

    bvolpato authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    1f0a07f View commit details
    Browse the repository at this point in the history
  5. Add link to the Dataflow service options page (apache#29068)

    * Add link to the Dataflow service options page
    
    * Fix link format
    
    * Small text edit
    rszper authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    bffeefc View commit details
    Browse the repository at this point in the history
  6. Arc network update (apache#29058)

    * Update arc terraform to allow for coloaction in the default
    network.Allow usage of reserved ip. Allow usage of existing SA
    
    * sync beam env
    
    * move aditional runners to load based scaling
    volatilemolotov authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    f28009d View commit details
    Browse the repository at this point in the history
  7. Update base image requirements in preparation for next release (apach…

    damccorm authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    bbafe74 View commit details
    Browse the repository at this point in the history
  8. Fix file paths for test arguments (apache#29034)

    aleksandr-dudko authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    938e64e View commit details
    Browse the repository at this point in the history
  9. Fix Python PreCommit GHA recommendations_ai teardown (apache#29067)

    Abacn authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    3d6bb98 View commit details
    Browse the repository at this point in the history
  10. Add InfluxDB credentials (apache#29071)

    Amar3tto authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    26228fc View commit details
    Browse the repository at this point in the history
  11. fix workflows (apache#29009)

    andreydevyatkin authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    012eb1c View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    8145d3c View commit details
    Browse the repository at this point in the history
  13. Add 'release/trigger_all_tests.json' to trigger path of mass comment …

    …PR (apache#29064)
    
    * Add 'release/trigger_all_tests.json' to trigger path of mass comment PR
    
    * Create GItHub Action PreCommit
    
    * Create missing Google-ads IO PreCommit
    
    * Create missing RequestResponse IO PreCommit
    
    * fix leftover
    
    * Fix leftover in phrase
    
    * Update contributor-docs/release-guide.md
    
    Co-authored-by: Danny McCormick <[email protected]>
    
    ---------
    
    Co-authored-by: Danny McCormick <[email protected]>
    2 people authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    27f701b View commit details
    Browse the repository at this point in the history
  14. Bump cloud.google.com/go/profiler from 0.3.1 to 0.4.0 in /sdks (apach…

    …e#29070)
    
    Bumps [cloud.google.com/go/profiler](https://github.com/googleapis/google-cloud-go) from 0.3.1 to 0.4.0.
    - [Release notes](https://github.com/googleapis/google-cloud-go/releases)
    - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
    - [Commits](googleapis/google-cloud-go@talent/v0.3.1...v0.4.0)
    
    ---
    updated-dependencies:
    - dependency-name: cloud.google.com/go/profiler
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...
    
    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    dependabot[bot] authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    cb37c64 View commit details
    Browse the repository at this point in the history
  15. Move Java Google BOM to best effort post-release task (apache#29062)

    * Move Java Google BOM to best effort post-release task
    
    * Wording
    damccorm authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    45fb757 View commit details
    Browse the repository at this point in the history
  16. Remove command prompts from Go quickstart commands (apache#29079)

    rszper authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    9e6beca View commit details
    Browse the repository at this point in the history
  17. make BeamKafkaTable.createKafkaRead to be protected, and we can overr…

    …ide it appending options to KafkaIO.Read (apache#29051)
    gabrywu authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    7ab4879 View commit details
    Browse the repository at this point in the history
  18. TImeout StreamingDataflowWorkerTest when test stuck (apache#29065)

    Abacn authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    09035c9 View commit details
    Browse the repository at this point in the history
  19. Refactor StateFetcher (apache#28755)

    Refactor and cleanup of StateFetcher in preparation for future changes
    m-trieu authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    247b26a View commit details
    Browse the repository at this point in the history
  20. Bump google.golang.org/api from 0.147.0 to 0.148.0 in /sdks (apache#2…

    …9086)
    
    Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.147.0 to 0.148.0.
    - [Release notes](https://github.com/googleapis/google-api-go-client/releases)
    - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
    - [Commits](googleapis/google-api-go-client@v0.147.0...v0.148.0)
    
    ---
    updated-dependencies:
    - dependency-name: google.golang.org/api
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...
    
    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    dependabot[bot] authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    b7c5d6b View commit details
    Browse the repository at this point in the history
  21. Remove warning in dataframes io. (apache#29084)

    robertwb authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    1ec8d9f View commit details
    Browse the repository at this point in the history
  22. beam_performance_singlestoreio (apache#28986)

    * beam_performance_singlestoreio
    
    * Update .github/workflows/beam_PerformanceTests_SingleStoreIO.yml
    
    Add coment about the test-argumets-action
    
    Co-authored-by: Andrey Devyatkin <[email protected]>
    
    * remove fail-fast and remove comment line in config
    
    * config cleanup
    
    * influx creds and move to main pool
    
    ---------
    
    Co-authored-by: Andrey Devyatkin <[email protected]>
    2 people authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    81a99d3 View commit details
    Browse the repository at this point in the history
  23. Setup Java 21 container (apache#28918)

    * Setup Java 21 container (apache#28833)
    
    * Add Java21 container
    
    * fix spotless
    
    * Update website
    
    * Fix jamm
    
    * align configs in recent change
    
    * incremental fixes
    
    * add more comments for when jpms test enabled
    Abacn authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    6c3cca4 View commit details
    Browse the repository at this point in the history
  24. Editorial pass on content (apache#29094)

    rszper authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    2c7310f View commit details
    Browse the repository at this point in the history
  25. Fix rrio PreCommit build configs (apache#29097)

    Abacn authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    89c5380 View commit details
    Browse the repository at this point in the history
  26. Fix various issues in SplunkIO (apache#28825)

    * Fix GZIP compression in HttpEventPublisher.
    
    * Add checks to make sure the provided URL is valid.
    
    * Fix issue with DefaultCoder in AutoValue generated classes.
    
    * Add support for Splunk `fields` metadata.
    
    Also fix Coder issues for SplunkEvent.
    
    * Address comments.
    pranavbhandari24 authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    ce840a5 View commit details
    Browse the repository at this point in the history
  27. Refactor prepare arguments step for Github actions (apache#29090)

    * Refactor prepare arguments step
    
    * Rename perf/load/ pipeline options folders
    Amar3tto authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    fc9d474 View commit details
    Browse the repository at this point in the history
  28. [YAML] Improved pipeline schema definition. (apache#29083)

    Slightly stricter definitions for catching more errors, as well as avoding the use of
    anyOf which often makes it difficult to deduce what the true error is.
    
    This does mean a pipeline must have a transform (or source/sink) block rather than simply
    be itself a list of transforms.
    robertwb authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    59de1ff View commit details
    Browse the repository at this point in the history
  29. [yaml] support gcs location for main.py yaml input file (apache#29014)

    Polber authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    17df84b View commit details
    Browse the repository at this point in the history
  30. Merge pull request apache#29093 [YAML] Don't require redundant input …

    …for YamlTransform.
    robertwb authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    9858544 View commit details
    Browse the repository at this point in the history
  31. Add jsonschema dependency.

    robertwb authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    efe00ec View commit details
    Browse the repository at this point in the history
  32. Validate incoming JSON data from pubsub.

    As well as good practice, not doing so may result in much more
    obscure errors (e.g. during encoding) downstream.
    robertwb authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    872ec58 View commit details
    Browse the repository at this point in the history
  33. Regenrate python base deps files.

    robertwb authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    a1e5fff View commit details
    Browse the repository at this point in the history
  34. yapf

    robertwb authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    60ff61d View commit details
    Browse the repository at this point in the history
  35. Add workflows to README (apache#29096)

    * Add readme for PerformanceTests TextIOIT, JDBC, Kafka IO, SpannerIO, SQLBigQueryIO and BiqQueryIO Python
    
    * Update readme
    
    * PRs 28582 28584 28606 28581
    
    * PR 28738 LoadTests_Java_GBK_Dataflow
    
    * Add readme for PostCommit Java Examples Dataflow V2
    
    * Add readme for LoadTests Java CoGBK
    
    * Add readme for LoadTests Python CoGBK Dataflow
    
    * Add readme for LoadTests Python ParDo and SideInput
    
    * Add readme for LoadTests Smoke Python and Java
    
    * Update Readme
    
    * Update Readme
    
    * updated README
    
    * Update readme for Performance Tests BigQueryIO Write Python Batch
    
    * Remove Trigger Phrases for Load Tests and Performance tests
    
    * PR 28846 28730 28827 28861 28897
    
    * update readme
    
    ---------
    
    Co-authored-by: aleksandr-dudko <[email protected]>
    Co-authored-by: vitaly.terentyev <[email protected]>
    Co-authored-by: magicgoody <[email protected]>
    4 people authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    2902462 View commit details
    Browse the repository at this point in the history
  36. Bump cloud.google.com/go/spanner from 1.50.0 to 1.51.0 in /sdks (apac…

    …he#29046)
    
    Bumps [cloud.google.com/go/spanner](https://github.com/googleapis/google-cloud-go) from 1.50.0 to 1.51.0.
    - [Release notes](https://github.com/googleapis/google-cloud-go/releases)
    - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
    - [Commits](googleapis/google-cloud-go@spanner/v1.50.0...spanner/v1.51.0)
    
    ---
    updated-dependencies:
    - dependency-name: cloud.google.com/go/spanner
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...
    
    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    dependabot[bot] authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    edbc12f View commit details
    Browse the repository at this point in the history
  37. Bump golang.org/x/net from 0.7.0 to 0.17.0 in /learning/katas/go (apa…

    …che#28947)
    
    Bumps [golang.org/x/net](https://github.com/golang/net) from 0.7.0 to 0.17.0.
    - [Commits](golang/net@v0.7.0...v0.17.0)
    
    ---
    updated-dependencies:
    - dependency-name: golang.org/x/net
      dependency-type: indirect
    ...
    
    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    dependabot[bot] authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    b247ff0 View commit details
    Browse the repository at this point in the history
  38. Include ID and name of tracked thread in lull message (apache#29007)

    In order to aid debugging of lulls.
    joar authored and Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    0a44a2b View commit details
    Browse the repository at this point in the history
  39. flush buffer during drain operation for requiresStableInput operator

    correct log message format
    
    apply spotless
    
    flush buffer when requiresStableInput is set
    
    re-organize imports
    
    add flag in FlinkPipelineOptions to allow draining for pipelines with RequiresStableInput
    
    apply spotless again
    Kanishk Karanawat committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    e08578a View commit details
    Browse the repository at this point in the history