Add integration for openhouse-java-iceberg-1.5-runtime and openhouse-spark-3.5-runtime #221
Conversation
jiang95-dev
force-pushed
the
lejiang/iceberg-integrations
branch
from
afe2de8
to
cd3d61c
Compare
jiang95-dev
changed the title
| @@ -0,0 +1,15 @@ | |||
| plugins { | |||
There was a problem hiding this comment.
I am wondering if we should layout directories like
integrations/java/iceberg-1.2/..
integrations/java/iceberg-1.5/..
There was a problem hiding this comment.
➕ for maintaining backward compatibility, you can use settings.gradle's module renaming feature.
| @@ -0,0 +1,579 @@ | |||
| package com.linkedin.openhouse.javaclient; | |||
There was a problem hiding this comment.
Is code duplication the only way? Have we considered inheritance to share the code?
| force 'org.apache.orc:orc-core:1.8.3' | ||
| force 'com.google.guava:guava:31.1-jre' | ||
| if (it.path != ':integrations:spark-3.5:openhouse-spark-3.5-itest') { | ||
| configurations.all { |
There was a problem hiding this comment.
can't this be overwritten in openhouse-spark-3.5-itest's build.gradle? and leave the changes in this file be..
There was a problem hiding this comment.
The configuration takes precedence over the subproject's build file.
| @@ -0,0 +1,15 @@ | |||
| plugins { | |||
There was a problem hiding this comment.
➕ for maintaining backward compatibility, you can use settings.gradle's module renaming feature.
| import org.junit.jupiter.api.Assertions; | ||
| import org.junit.jupiter.api.Test; | ||
|
|
||
| public class OpenHouseTableOperationsTest { |
There was a problem hiding this comment.
why not reuse the source from other module if its unchanged?
sourceSets {
test {
java {
srcDir project(':other-module').sourceSets.test.java.srcDirs
}
resources {
srcDir project(':other-module').sourceSets.test.resources.srcDirs
}
}
}
| } | ||
|
|
||
| dependencies { | ||
| compileOnly project(':client:secureclient') |
There was a problem hiding this comment.
you can declare dependency on java moduel for 1.2 if the code is not changing. basically:
dependencies {
api (java-rntime_iceberg_1.2) {
transitive=false
}
}
| // except for source code written in the module. As a result, | ||
| // we remove chances of classpath conflicts during runtime/compiletime. | ||
| shadowJar { | ||
| dependencies { |
There was a problem hiding this comment.
this shadow block will be thin if we reuse code from iceberg 1.2 module
| * com.linkedin.openhouse.javaclient.OpenHouseMetricsReporter property | ||
| */ | ||
| @Slf4j | ||
| public class OpenHouseMetricsReporter implements MetricsReporter { |
There was a problem hiding this comment.
are there any code changes for this class?
| package com.linkedin.openhouse.javaclient; | ||
|
|
||
| import com.google.common.annotations.VisibleForTesting; | ||
| import com.google.gson.Gson; |
There was a problem hiding this comment.
are there any code changes for this class?
| @@ -0,0 +1,2 @@ | |||
| com.linkedin.openhouse.relocated.org.springframework.http.codec.ClientCodecConfigurer=com.linkedin.openhouse.relocated.org.springframework.http.codec.support.DefaultClientCodecConfigurer | |||
There was a problem hiding this comment.
wouldn't be needed if we reuse 1.2 artifact
jiang95-dev
force-pushed
the
lejiang/iceberg-integrations
branch
from
3237070
to
6050f41
Compare
Summary
Adding 2 integration artifacts: openhouse-java-iceberg-1.5-runtime and openhouse-spark-3.5-runtime. They are leveraging the iceberg-core:1.5.2 and iceberg-spark-runtime-3.5_2.12, and can only be compiled under Java 9+. This client side upgrade is the step after the server side upgrade, and apps upgrade will be the next step.
Changes
There are many duplicating files. Files worth reviewing are the following:
Testing Done
All unit tests and integration tests passed.
Additional Information
For all the boxes checked, include additional details of the changes made in this pull request.