* [yaml] package kafka_clients 3.1.2 in Kafka Provider jar

Signed-off-by: Jeffrey Kinard <[email protected]>

* fix test failure

Signed-off-by: Jeffrey Kinard <[email protected]>

---------

Signed-off-by: Jeffrey Kinard <[email protected]>

This reverts commit eaf53e5.

* Delete stale and unused .test-infra/pipelines directory

* Update settings.gradle.kts to remove test-infra-pipelines

* add documentation to IcebergIO's java class

* add example; trigger ITs

* nit

* Update direct_runner.py

Improve the error message for DirectRunner.

* Update direct_runner.py

* Optimize to skip filter application when there is only a single output

* Make SparkTransformOverrides class public for testing

* add related test

* Touch trigger files

* add CHANGES.md

* Adding insertion and enrichment pipeline

* Enhanced Data Schema

* Added Apache Licensed to the notebook

* Adding Chunking Strategy

* removed unused imports

* Modified insertion logic in redis for incorporating chunking strategy

* refacted redis code

* code review changes

* Added chunking code in notebook

* Added code review changes

* Code review changes: using chunking strategy as enum

* Added Code Review Changes

* Code review changes

* Added code review changes

* Added Code Review Changes

* Code review changes

* Ingestion and Enrichment pipeline for OpenSearch Vector DB

* Added logic for reading password from .env file

* Added opensearch vector notebook

* Update credentials.env

* Added code review changes

* Added Description in opensearch notebook

* Added description in opensearch notebook

* Code review changes

Bumps [go.mongodb.org/mongo-driver](https://github.com/mongodb/mongo-go-driver) from 1.17.0 to 1.17.1.
- [Release notes](https://github.com/mongodb/mongo-go-driver/releases)
- [Commits](mongodb/mongo-go-driver@v1.17.0...v1.17.1)

---
updated-dependencies:
- dependency-name: go.mongodb.org/mongo-driver
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

The original example was not actually counting the produce but grouping the produce per season. Maybe it's better to rename the variables to reflect this, in order to not confuse the reader.

Co-authored-by: lostluck <[email protected]>
Co-authored-by: Jack McCluskey <[email protected]>

1. Added links for different handlers and removed code for unreachable conditions
2. Removed patch decorator in test

Co-authored-by: Claude <[email protected]>

* [Managed Iceberg] add GiB autosharding

* trigger iceberg integration tests

* fix test

* add to CHANGES.md

* increase GiB limits

* increase GiB limits

* data file size distribution metric; max file size 512mb

…tions#groupByKeyInGlobalWindow (#32610)

* change GroupNonMergingWindowsFunctions#groupByKeyInGlobalWindow to use partitioner

…ate that It is unused and only retained for backward compatibility (#32635)

In this commit, we add unused note on `output_fn` constructor
argument in `WriteToFiles` class and only retained for backward
compatibility.

Signed-off-by: Mohamed Awnallah <[email protected]>

Co-authored-by: damccorm <[email protected]>

* [#28187] Add gradle targets to execute python tests with prism.

* Add issues for non-expansion service skipped tests.

* Format diff.

---------

Co-authored-by: lostluck <[email protected]>

Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.27.39 to 1.27.41.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](aws/aws-sdk-go-v2@config/v1.27.39...config/v1.27.41)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.67.0 to 1.67.1.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](grpc/grpc-go@v1.67.0...v1.67.1)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

previous syntax for the snippet was in Groovy, but in step 1&2 of the Gradle conversion we use Kotlin

…ev' version

Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.17.25 to 1.17.28.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](aws/aws-sdk-go-v2@credentials/v1.17.25...credentials/v1.17.28)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ccessor" (#32694)

This reverts commit efee92b.

* Force protobuf 5.x

* restore lower bound

* Add bypass for microsecond conversion bounds

* linting

* fix incorrect unit, standardize micros_to_nanos

* remove unused constant

* add extra unit tests

* formatting

Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.43.0 to 1.44.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](googleapis/google-cloud-go@pubsub/v1.43.0...spanner/v1.44.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [golang.org/x/net](https://github.com/golang/net) from 0.29.0 to 0.30.0.
- [Commits](golang/net@v0.29.0...v0.30.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.27.41 to 1.27.42.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](aws/aws-sdk-go-v2@config/v1.27.41...config/v1.27.42)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

#32666)

* group all data files before writing a manifest file

* add to changes md

* add data file roundtrip equality test

* address comments

* create missing iceberg tables

* trigger iceberg integration tests

* add to changes md

* add to java doc

* trigger iceberg integration tests

allow numpy 2.1.x

* Enforce a size limit on StringSetData

* Make StringSetData set mutable. This avoids
  copy and create new ImutableSet every time

* adjust warning log

Signed-off-by: Jeffrey Kinard <[email protected]>

* Report File Lineage on directory

* added comments, restore lineage assert in TextIOIT

* Report bucket level Lineage for files larger than 100

* fix lint

…ame inside Work. (#32715)

Thread.setName is expensive and uses upto 4% cpu on jobs with many keys.

* fix bug all arg add as inputs

* fix bug for fillna

* Revert "fix bug for fillna"

This reverts commit 2a5736c.

* fix bug for fillna

* add test for fillna a column

* add test for fillna a column

* add test for fillna a column

* revert add test to frames_test

* Move test from transforms to frames

* Invoke teardown when DoFn throws in portable runners

* update CHANGES.md

* adjusted comment and logging

* Support ordered list state in python sdk and fnapi runner.

* Add test to verify integrity of multiple iterators

* Add fuzz tests and fix two edge cases.

* Add sortedcontainer in package dependency

* Code refactoring and add a check for the supported maximum key

* regenerate requirments for python images.

* Refactor portable runner code for ordered list state

* Return continuation tokens in portable runnner for ordered list state

* Fix some lints

* Apply yapf

* Fix lints

* Sync base image requirements with master.

* Add typing for ordered list state apis.

* Add typing to orderedliststate user state.

* Fix a typo.

* Refactor some code based on the feedback.

* Fix lints

* Remove the support of int argument type in ordered list state apis.

* Fix formats and lints

* More lints

* Refactor the code to use the continuation token logic.

* Fix lints

… in Java SDK (#32540)

* Initial changes to support processing global sequences.

* Refactor the DoFns out of the transform and into a class hierarchy.

* Next round of implementation of Global Sequence handling.

* Added ticker timers in global sequence processing.

* Corrected the emission batch logic.

* Reworked some tests and fixed the batch output logic.

* Pluggable combiner for the global sequence.

* First iteration of the efficient merging accumulator

* Mostly complete implementation of the accumulator and corresponding tests.

* Additional round of test refinements.

* Added logic to DQL the records below the global sequence range.

* Added providing a global sequence combiner through a handler.

* Added SequenceRangeAccumulatorCoder and tests. Improved logic of creating timers.

* Fixed logging levels (moved them to "trace") on several transforms.

* Round of code improvements and cleanups.

* Tests to verify that the the global sequence is correctly produced by the transform.

* Added batch processing verification to the global sequence processing.

* A round of documentation update and minor clean up.

* Fixed the description in CHANGES.md

* Polish by "spotless"

* Polish by "spotless"

* Removed unneeded logging configuration file.

* Made ContiguousSequenceRange open ended.

* Removed details from 2.60.0 section in CHANGES.md.

* Update sdks/java/extensions/ordered/src/main/java/org/apache/beam/sdk/extensions/ordered/combiner/DefaultSequenceCombiner.java

Co-authored-by: Danny McCormick <[email protected]>

* Fixed spotless related errors.

* Added a note about the new functionality to CHANGES.md

* Added clarification around the data structure used in the sequence combiner.

* Added clarification around the data structure used in the sequence combiner.

* Fixed the problem with allowed lateness being set to 0 in the global sequence tracker.

* Parameterized the GlobalSequenceTracker with the max number of events to trigger the re-evaluation. Fixed accidentally disabled unit tests.

* Made the event timer used to wait for the event arrival respect the lateness of the input.

* Created new failure reason code - "before initial sequence"

---------

Co-authored-by: Danny McCormick <[email protected]>

* Handle Date type in HCatToRow

Some initial notes:
- The issue (#20685) deals with java.sql.Date, which I wasn't able to
  reproduce fully (I can currently write hcatalog hadoop.hive date)
- On this note, 267f76f changed the
  code involved so that there's a direct cast to AbstractInstant in
  RowUtils.java. This doesn't change much, but jfyi.

* Run: ./gradlew :sdks:java:io:hcatalog:spotlessApply

* review cr: castTypes util

- s/castHDate/maybeCastHDate/ to be more concise
- move values manipulation to a separate util (hopefully, I understood
  the cr in the right way)

* Try deepcopy combine_fn and fallback to pickling if TypeError.

* Remove logging, add unit test

* Linter fixes

---------

Co-authored-by: Claude <[email protected]>

…2688)

* support timestamp, time, date types

* add to changes md

* always write java time LocalDateTime for iceberg TIMESTAMP

* update java doc

* add timezone support with Strings

* clean up; reading iceberg timestamptz will return sqltype.datetime

* support string, long, sql.datetime, and datetime; timestamp returns sql.datetime and timestamptTZ returns datetime

* add support for read with metadata in MqttIO

* Update CHANGES.md

* update javadoc

* update javadoc

* refactor : change to use SchemaCoder in MqttIO
- remove MqttRecordCoder
- refactor MqttRecord to use AutoValueSchema
- change related test

…2736)

* Attempt to fix permred test.

* Also remove token format, making it consistent with java_tests and python_tests.

* Remove the triggering file since the test is only triggered with the yaml file on master.

… in KafkaIO When Using Managed Service for Apache Kafka (#32722)

* Update setup.py to latest version of facets to fix outdated alias issue

* Add application default support in KafkaIO for users who want to authenticate their Kafka Clients with Google's server.

* Add application default support in KafkaIO for users who want to authenticate their Kafka Clients with Google's server.

---------

Co-authored-by: Ozzie Fernandez <[email protected]>

…32741)

Fix the FlinkRunner for Dataproc

* Also fix Kafka integration test version resolution

* Fix the snapshot publishing script

* Support distribution from setuptools prior to 69.3.0 as well.

Fix Pydoc Precommit

Fixed the tox and pyproject settings for numpy 2

* Fix build_wheels and build_release_candidate

Co-authored-by: Rebecca Szper <[email protected]>

Co-authored-by: Rebecca Szper <[email protected]>

Co-authored-by: Rebecca Szper <[email protected]>

Co-authored-by: Rebecca Szper <[email protected]>

Co-authored-by: Rebecca Szper <[email protected]>

Co-authored-by: Rebecca Szper <[email protected]>