Skip to content

Commit

Permalink
Prevent use of GraalVM Native Build Tools Maven Plugin in parallel bu…
Browse files Browse the repository at this point in the history
…ilds of project
  • Loading branch information
linghengqian committed Aug 18, 2024
1 parent cf9e6a9 commit c83b183
Show file tree
Hide file tree
Showing 9 changed files with 39 additions and 44 deletions.
3 changes: 2 additions & 1 deletion .github/workflows/graalvm.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,5 +54,6 @@ jobs:
distribution: 'graalvm-community'
github-token: ${{ secrets.GITHUB_TOKEN }}
cache: 'maven'
native-image-job-reports: 'true'
- name: Run nativeTest with GraalVM CE for ${{ matrix.java-version }}
run: ./mvnw -PnativeTestInShardingSphere -T1C -e clean test
run: ./mvnw -PnativeTestInShardingSphere -e clean test
9 changes: 2 additions & 7 deletions .github/workflows/nightly-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -123,13 +123,8 @@ jobs:
java-version: '22.0.2'
distribution: 'graalvm-community'
github-token: ${{ secrets.GITHUB_TOKEN }}
- uses: actions/cache@v4
with:
path: ~/.m2/repository
key: ${{ needs.global-environment.outputs.GLOBAL_CACHE_PREFIX }}-maven-third-party-cache-${{ github.sha }}
restore-keys: |
${{ needs.global-environment.outputs.GLOBAL_CACHE_PREFIX }}-maven-third-party-cache-
${{ needs.global-environment.outputs.GLOBAL_CACHE_PREFIX }}-maven-third-party-
cache: 'maven'
native-image-job-reports: 'true'
- uses: docker/login-action@v3
with:
registry: ${{ env.HUB }}
Expand Down
11 changes: 3 additions & 8 deletions .github/workflows/nightly-ci-dynamic.yml
Original file line number Diff line number Diff line change
Expand Up @@ -129,12 +129,7 @@ jobs:
java-version: '22.0.2'
distribution: 'graalvm-community'
github-token: ${{ secrets.GITHUB_TOKEN }}
- uses: actions/cache@v4
with:
path: ~/.m2/repository
key: ${{ needs.global-environment.outputs.GLOBAL_CACHE_PREFIX }}-maven-third-party-cache-${{ github.sha }}
restore-keys: |
${{ needs.global-environment.outputs.GLOBAL_CACHE_PREFIX }}-maven-third-party-cache-
${{ needs.global-environment.outputs.GLOBAL_CACHE_PREFIX }}-maven-third-party-
cache: 'maven'
native-image-job-reports: 'true'
- name: Run nativeTest with GraalVM CE
run: ./mvnw -PnativeTestInShardingSphere -T1C -e clean test
run: ./mvnw -PnativeTestInShardingSphere -e clean test
11 changes: 3 additions & 8 deletions .github/workflows/nightly-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -118,12 +118,7 @@ jobs:
java-version: '22.0.2'
distribution: 'graalvm-community'
github-token: ${{ secrets.GITHUB_TOKEN }}
- uses: actions/cache@v4
with:
path: ~/.m2/repository
key: ${{ needs.global-environment.outputs.GLOBAL_CACHE_PREFIX }}-maven-third-party-cache-${{ github.sha }}
restore-keys: |
${{ needs.global-environment.outputs.GLOBAL_CACHE_PREFIX }}-maven-third-party-cache-
${{ needs.global-environment.outputs.GLOBAL_CACHE_PREFIX }}-maven-third-party-
cache: 'maven'
native-image-job-reports: 'true'
- name: Run nativeTest with GraalVM CE
run: ./mvnw -PnativeTestInShardingSphere -T1C -e clean test
run: ./mvnw -PnativeTestInShardingSphere -e clean test
2 changes: 1 addition & 1 deletion distribution/proxy/src/main/release-docs/LICENSE
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,7 @@ The text of each license is the standard Apache 2.0 license.
failsafe 2.4.4: https://github.com/jhalterman/failsafe, Apache 2.0
failureaccess 1.0.1: https://github.com/google/guava, Apache 2.0
freemarker 2.3.31: https://freemarker.apache.org/, Apache 2.0
groovy 4.0.19: https://groovy.apache.org/, Apache 2.0
groovy 4.0.22: https://groovy.apache.org/, Apache 2.0
grpc-api 1.65.1: https://github.com/grpc/grpc-java, Apache 2.0
grpc-context 1.65.1: https://github.com/grpc/grpc-java, Apache 2.0
grpc-core 1.65.1: https://github.com/grpc/grpc-java, Apache 2.0
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -393,7 +393,7 @@ ClickHouse 不支持 ShardingSphere 集成级别的本地事务,XA 事务和 S
<dependency>
<groupId>io.github.linghengqian</groupId>
<artifactId>hive-server2-jdbc-driver-thin</artifactId>
<version>1.0.0</version>
<version>1.2.0</version>
<exclusions>
<exclusion>
<groupId>com.fasterxml.woodstox</groupId>
Expand Down Expand Up @@ -442,8 +442,7 @@ Args=--initialize-at-run-time=\
用户应当考虑在 ShardingSphere JDBC 中仅使用支持 ACID 的表。`apache/hive` 提供了多种事务解决方案。

第1种选择是使用 ACID 表,可能的建表流程如下。
如果用户在 ACID 表上频繁更新和删除数据,用户可能不得不在 DML 语句执行前后进行等待,
以让 HiveServer2 完成低效的 DML 操作,因为 ACID 表中数据是在文件夹级别跟踪。
由于其过时的基于目录的表格式,用户可能不得不在 DML 语句执行前后进行等待,以让 HiveServer2 完成低效的 DML 操作。

```sql
set metastore.compactor.initiator.on=true;
Expand All @@ -465,7 +464,9 @@ CREATE TABLE IF NOT EXISTS t_order
) CLUSTERED BY (order_id) INTO 2 BUCKETS STORED AS ORC TBLPROPERTIES ('transactional' = 'true');
```

第2种选择是使用 Iceberg 表,可能的建表流程如下。`apache/iceberg` 表格式有望在未来几年取代传统的 Hive 表格式。
第2种选择是使用 Iceberg 表,可能的建表流程如下。
Apache Iceberg 表格式有望在未来几年取代传统的 Hive 表格式,
参考 https://blog.cloudera.com/from-hive-tables-to-iceberg-tables-hassle-free/ 。

```sql
set iceberg.mr.schema.auto.conversion=true;
Expand Down Expand Up @@ -510,7 +511,7 @@ sudo apt-get install build-essential zlib1g-dev -y
git clone [email protected]:apache/shardingsphere.git
cd ./shardingsphere/
./mvnw -PnativeTestInShardingSphere -T1C -e clean test
./mvnw -PnativeTestInShardingSphere -e clean test
```

当贡献者发现缺少与 ShardingSphere 无关的第三方库的 GraalVM Reachability Metadata 时,应当在
Expand Down Expand Up @@ -539,5 +540,10 @@ Reachability Metadata 位于 `shardingsphere-infra-reachability-metadata` 子模
```bash
git clone [email protected]:apache/shardingsphere.git
cd ./shardingsphere/
./mvnw -PgenerateMetadata -DskipNativeTests -e -T1C clean test native:metadata-copy
./mvnw -PgenerateMetadata -DskipNativeTests -e clean test native:metadata-copy
```

在使用 GraalVM Native Build Tools 的 Maven Plugin 时,
贡献者应避免使用 Maven 的并行构建功能。
GraalVM Native Build Tools 的 Maven Plugin并不是线程安全的,
它与 https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3 不兼容。
Original file line number Diff line number Diff line change
Expand Up @@ -409,7 +409,7 @@ An example of a possible configuration is as follows,
<dependency>
<groupId>io.github.linghengqian</groupId>
<artifactId>hive-server2-jdbc-driver-thin</artifactId>
<version>1.0.0</version>
<version>1.2.0</version>
<exclusions>
<exclusion>
<groupId>com.fasterxml.woodstox</groupId>
Expand Down Expand Up @@ -459,8 +459,8 @@ In order to be able to use DML SQL statements such as `delete`, when connecting
users should consider using only ACID-supported tables in ShardingSphere JDBC. `apache/hive` provides a variety of transaction solutions.

The first option is to use ACID tables, and the possible table creation process is as follows.
If users frequently update and delete data on ACID tables, users may have to wait before and after the execution of DML statements,
to allow HiveServer2 to complete inefficient DML operations, because data in ACID tables is tracked at the folder level.
Due to its outdated catalog-based table format,
users may have to wait before and after DML statement execution to let HiveServer2 complete the inefficient DML operations.

```sql
set metastore.compactor.initiator.on=true;
Expand All @@ -482,8 +482,9 @@ CREATE TABLE IF NOT EXISTS t_order
) CLUSTERED BY (order_id) INTO 2 BUCKETS STORED AS ORC TBLPROPERTIES ('transactional' = 'true');
```

The second option is to use Iceberg table. The possible table creation process is as follows.
The `apache/iceberg` table format is expected to replace the traditional Hive table format in the next few years.
The second option is to use Iceberg table. The possible table creation process is as follows.
Apache Iceberg table format is poised to replace the traditional Hive table format in the coming years,
see https://blog.cloudera.com/from-hive-tables-to-iceberg-tables-hassle-free/ .

```sql
set iceberg.mr.schema.auto.conversion=true;
Expand Down Expand Up @@ -530,7 +531,7 @@ sudo apt-get install build-essential zlib1g-dev -y
git clone [email protected]:apache/shardingsphere.git
cd ./shardingsphere/
./mvnw -PnativeTestInShardingSphere -T1C -e clean test
./mvnw -PnativeTestInShardingSphere -e clean test
```

When contributors discover that GraalVM Reachability Metadata is missing for a third-party library not related to ShardingSphere,
Expand Down Expand Up @@ -560,5 +561,8 @@ contributors should place it on the classpath of the `shardingsphere-test-native
```bash
git clone [email protected]:apache/shardingsphere.git
cd ./shardingsphere/
./mvnw -PgenerateMetadata -DskipNativeTests -e -T1C clean test native:metadata-copy
./mvnw -PgenerateMetadata -DskipNativeTests -e clean test native:metadata-copy
```

Contributors should avoid using Maven's parallel builds feature when using the Maven Plugin for GraalVM Native Build Tools.
The Maven Plugin for GraalVM Native Build Tools is not thread-safe and is incompatible with https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3 .
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,8 @@
import org.apache.shardingsphere.infra.expr.spi.InlineExpressionParser;
import org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.condition.EnabledForJreRange;
import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.junit.jupiter.api.condition.EnabledOnOs;
import org.junit.jupiter.api.condition.JRE;
import org.junit.jupiter.api.condition.OS;

import java.util.Collections;
Expand All @@ -34,7 +33,7 @@
import static org.hamcrest.MatcherAssert.assertThat;
import static org.junit.jupiter.api.Assertions.assertThrows;

@EnabledForJreRange(min = JRE.JAVA_22)
@EnabledIfSystemProperty(named = "java.vm.vendor", matches = "GraalVM Community", disabledReason = "Github Actions device performance is too low")
@EnabledOnOs(value = OS.LINUX, architectures = "amd64", disabledReason = "See https://www.graalvm.org/jdk21/reference-manual/java-on-truffle/faq/#does-java-running-on-truffle-run-on-hotspot-too")
class EspressoInlineExpressionParserTest {

Expand Down
6 changes: 3 additions & 3 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@
<json-smart.version>2.4.10</json-smart.version>
<accessors-smart.version>2.4.9</accessors-smart.version>
<asm.version>9.3</asm.version>
<groovy.version>4.0.19</groovy.version>
<groovy.version>4.0.22</groovy.version>
<freemarker.version>2.3.31</freemarker.version>
<bytebuddy.version>1.14.13</bytebuddy.version>

Expand Down Expand Up @@ -126,7 +126,7 @@
<mariadb-java-client.version>2.4.2</mariadb-java-client.version>
<clickhouse-jdbc.version>0.6.3</clickhouse-jdbc.version>
<hive.version>4.0.0</hive.version>
<hive-server2-jdbc-driver-thin.version>1.0.0</hive-server2-jdbc-driver-thin.version>
<hive-server2-jdbc-driver-thin.version>1.2.0</hive-server2-jdbc-driver-thin.version>
<presto.version>0.282</presto.version>

<hikari-cp.version>4.0.3</hikari-cp.version>
Expand All @@ -140,7 +140,7 @@
<hamcrest.version>2.2</hamcrest.version>
<mockito.version>4.11.0</mockito.version>
<awaitility.version>4.2.0</awaitility.version>
<testcontainers.version>1.20.0</testcontainers.version>
<testcontainers.version>1.20.1</testcontainers.version>
<commons-csv.version>1.9.0</commons-csv.version>

<graal-sdk.version>24.0.2</graal-sdk.version>
Expand Down

0 comments on commit c83b183

Please sign in to comment.