Add Apache Iceberg IO connector

[damccorm]

Apache Iceberg is an open table format layer on top of different filesystems.

Beam could support it in the form of an IO connector ideally integrating with Beam's own system for SQL support.

Inspiration can be taken from the ongoing effort to support Iceberg on Flink:
apache/iceberg#788
https://docs.google.com/document/d/1idTjQCubhOCea8LkKJ_5aMTV4V_Wf1n0aYhabGsLQZc/edit

Imported from Jira BEAM-10160. Original Jira may contain additional context.
Reported by: iemejia.

[gabrywu]

any updates?

[yupbank]

any updates?

[yeshvantbhavnasi]

any updates ?

[brucearctor]

@gabrywu , @yupbank , @yeshvantbhavnasi -- nobody is assigned [ therefore potentially not being worked on ], so not clear how quickly this will progress. That said, it is also open, and therefore something that anyone could work on and contribute. Contributions welcome! :-)

[pabloem]

hi folks - I am taking a look at this...

[jotarada]

Hey folks, is there any plans regarding this?

[kellen]

@pabloem ping from Spotify

[chadlagore]

Affirm interested in this 👀 cc @lydian

[aromanenko-dev]

CC: @mosche

[iemejia]

There was some work on a sink for iceberg by @Fokko
apache/iceberg#1972

@Fokko do you have any info about this, has someone tried to build on your work to get Beam + Iceberg working together?

[Fokko]

That was some early work, and probably I would just start from scratch. I would take inspiration from the Kafka-connect repository: https://github.com/tabular-io/iceberg-kafka-connect

[gabrywu]

we use iceberg low-level API adding data files directly

[RPasecky]

Wanted to check in on this, is beam prioritizing this?

[brucearctor]

@RPasecky -- it is less 'beam' prioritizing, and a question of whether any individuals are prioritizing. I'd love for this to exist, but can't devote the time. Hopefully someone will. Contributions welcomed! :-)

[kennknowles]

Added in #30797 and released in version 2.56.0

[Fokko]

Awesome! This is a great milestone! Thanks for working on this 🎉