Add ClasspathURLProvider implementation that reads system properties (#…

…29995) * Add ClasspathURLProvider implementation that reads system properties * Small refactoring
apache · Feb 5, 2024 · 68c07e1 · 68c07e1
1 parent bd8fbed
commit 68c07e1
Show file tree

Hide file tree

Showing 14 changed files with 402 additions and 45 deletions.
diff --git a/...l/shardingsphere-jdbc/yaml-config/jdbc-driver/known-implementation/_index.cn.md b/...l/shardingsphere-jdbc/yaml-config/jdbc-driver/known-implementation/_index.cn.md
@@ -79,5 +79,68 @@ ds_1:
 用例：
 - `jdbc:shardingsphere:absolutepath-environment:/path/to/config.yaml`
 
+### 从类路径中加载包含系统属性的配置文件
+
+加载类路径中包含系统属性的 config.yaml 配置文件的 JDBC URL，通过 `jdbc:shardingsphere:classpath-system-props:` 前缀识别。
+配置文件为 `xxx.yaml`，配置文件格式与`jdbc:shardingsphere:classpath-environment:`一致。
+与 `jdbc:shardingsphere:classpath-environment:` 的区别仅在于读取属性值的位置。
+
+假设存在以下一组系统属性，
+
+1. 存在系统属性`fixture.config.driver.jdbc-url`为`jdbc:h2:mem:foo_ds_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL`。
+2. 存在系统属性`fixture.config.driver.username`为`sa`。
+
+则对于以下 YAML 文件的截取片段，
+
+```yaml
+ds_1:
+  dataSourceClassName: com.zaxxer.hikari.HikariDataSource
+  driverClassName: $${fixture.config.driver.driver-class-name::org.h2.Driver}
+  jdbcUrl: $${fixture.config.driver.jdbc-url::jdbc:h2:mem:foo_ds_do_not_use}
+  username: $${fixture.config.driver.username::}
+  password: $${fixture.config.driver.password::}
+```
+
+此 YAML 截取片段将被解析为，
+
+```yaml
+ds_1:
+  dataSourceClassName: com.zaxxer.hikari.HikariDataSource
+  driverClassName: org.h2.Driver
+  jdbcUrl: jdbc:h2:mem:foo_ds_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL
+  username: sa
+  password:
+```
+
+在实际情况下，系统变量通常是动态定义的。
+假设如上系统变量均未定义，存在包含如上YAML截取片段的YAML文件`config.yaml`，
+可参考如下方法创建 DataSource 实例。
+
+```java
+import com.zaxxer.hikari.HikariConfig;
+import com.zaxxer.hikari.HikariDataSource;
+
+import javax.sql.DataSource;
+
+public DataSource createDataSource() {
+    HikariConfig config = new HikariConfig();
+    config.setDriverClassName("org.apache.shardingsphere.driver.ShardingSphereDriver");
+    config.setJdbcUrl("jdbc:shardingsphere:classpath-system-props:config.yaml");
+    try {
+        assert null == System.getProperty("fixture.config.driver.jdbc-url");
+        assert null == System.getProperty("fixture.config.driver.username");
+        System.setProperty("fixture.config.driver.jdbc-url", "jdbc:h2:mem:foo_ds_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL");
+        System.setProperty("fixture.config.driver.username", "sa");
+        return new HikariDataSource(config);
+    } finally {
+        System.clearProperty("fixture.config.driver.jdbc-url");
+        System.clearProperty("fixture.config.driver.username");
+    }
+}
+```
+
+用例：
+- `jdbc:shardingsphere:classpath-system-props:config.yaml`
+
 ### 其他实现
 具体可参考 https://github.com/apache/shardingsphere-plugin 。
diff --git a/...l/shardingsphere-jdbc/yaml-config/jdbc-driver/known-implementation/_index.en.md b/...l/shardingsphere-jdbc/yaml-config/jdbc-driver/known-implementation/_index.en.md
@@ -82,5 +82,71 @@ The difference from `jdbc:shardingsphere:classpath-environment:` is only where t
 Example:
 - `jdbc:shardingsphere:absolutepath-environment:/path/to/config.yaml`
 
+### Load configuration file containing system properties from classpath
+
+JDBC URL to load the config.yaml configuration file containing system properties in the classpath, 
+identified by the `jdbc:shardingsphere:classpath-system-props:` prefix.
+The configuration file is `xxx.yaml`, 
+and the configuration file format is consistent with `jdbc:shardingsphere:classpath-environment:`.
+The difference from `jdbc:shardingsphere:classpath-environment:` is only where the property value is read.
+
+Assume the following set of system properties exists,
+
+1. The system property `fixture.config.driver.jdbc-url` exists as `jdbc:h2:mem:foo_ds_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL`.
+2. The system property `fixture.config.driver.username` exists as `sa`.
+
+Then for the intercepted fragment of the following YAML file,
+
+```yaml
+ds_1:
+  dataSourceClassName: com.zaxxer.hikari.HikariDataSource
+  driverClassName: $${fixture.config.driver.driver-class-name::org.h2.Driver}
+  jdbcUrl: $${fixture.config.driver.jdbc-url::jdbc:h2:mem:foo_ds_do_not_use}
+  username: $${fixture.config.driver.username::}
+  password: $${fixture.config.driver.password::}
+```
+
+This YAML snippet will be parsed as,
+
+```yaml
+ds_1:
+  dataSourceClassName: com.zaxxer.hikari.HikariDataSource
+  driverClassName: org.h2.Driver
+  jdbcUrl: jdbc:h2:mem:foo_ds_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL
+  username: sa
+  password:
+```
+
+In real situations, system variables are usually defined dynamically.
+Assume that none of the above system variables are defined, 
+and there is a YAML file `config.yaml` containing the above YAML interception fragment,
+Users can refer to the following methods to create a DataSource instance.
+
+```java
+import com.zaxxer.hikari.HikariConfig;
+import com.zaxxer.hikari.HikariDataSource;
+
+import javax.sql.DataSource;
+
+public DataSource createDataSource() {
+    HikariConfig config = new HikariConfig();
+    config.setDriverClassName("org.apache.shardingsphere.driver.ShardingSphereDriver");
+    config.setJdbcUrl("jdbc:shardingsphere:classpath-system-props:config.yaml");
+    try {
+        assert null == System.getProperty("fixture.config.driver.jdbc-url");
+        assert null == System.getProperty("fixture.config.driver.username");
+        System.setProperty("fixture.config.driver.jdbc-url", "jdbc:h2:mem:foo_ds_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL");
+        System.setProperty("fixture.config.driver.username", "sa");
+        return new HikariDataSource(config);
+    } finally {
+        System.clearProperty("fixture.config.driver.jdbc-url");
+        System.clearProperty("fixture.config.driver.username");
+    }
+}
+```
+
+Example:
+- `jdbc:shardingsphere:classpath-system-props:config.yaml`
+
 ### Other implementations
 For details, please refer to https://github.com/apache/shardingsphere-plugin .
diff --git a/...ache/shardingsphere/driver/jdbc/core/driver/spi/absolutepath/AbsolutePathURLProvider.java b/...ache/shardingsphere/driver/jdbc/core/driver/spi/absolutepath/AbsolutePathURLProvider.java
@@ -51,7 +51,7 @@ public byte[] getContent(final String url, final String urlPrefix) {
             String line;
             while (null != (line = reader.readLine())) {
                 if (!line.startsWith("#")) {
-                    builder.append(line).append('\n');
+                    builder.append(line).append(System.lineSeparator());
                 }
             }
             return builder.toString().getBytes(StandardCharsets.UTF_8);

diff --git a/...here/driver/jdbc/core/driver/spi/absolutepath/AbsolutePathWithEnvironmentURLProvider.java b/...here/driver/jdbc/core/driver/spi/absolutepath/AbsolutePathWithEnvironmentURLProvider.java
@@ -58,7 +58,7 @@ public byte[] getContent(final String url, final String urlPrefix) {
             while (null != (line = reader.readLine())) {
                 if (!line.startsWith("#")) {
                     line = replaceEnvironmentVariables(line);
-                    builder.append(line).append('\n');
+                    builder.append(line).append(System.lineSeparator());
                 }
             }
             return builder.toString().getBytes(StandardCharsets.UTF_8);

diff --git a/...org/apache/shardingsphere/driver/jdbc/core/driver/spi/classpath/ClasspathURLProvider.java b/...org/apache/shardingsphere/driver/jdbc/core/driver/spi/classpath/ClasspathURLProvider.java
@@ -49,7 +49,7 @@ public byte[] getContent(final String url, final String urlPrefix) {
             String line;
             while (null != (line = reader.readLine())) {
                 if (!line.startsWith("#")) {
-                    builder.append(line).append('\n');
+                    builder.append(line).append(System.lineSeparator());
                 }
             }
             return builder.toString().getBytes(StandardCharsets.UTF_8);

diff --git a/...dingsphere/driver/jdbc/core/driver/spi/classpath/ClasspathWithEnvironmentURLProvider.java b/...dingsphere/driver/jdbc/core/driver/spi/classpath/ClasspathWithEnvironmentURLProvider.java
@@ -56,7 +56,7 @@ public byte[] getContent(final String url, final String urlPrefix) {
             while (null != (line = reader.readLine())) {
                 if (!line.startsWith("#")) {
                     line = replaceEnvironmentVariables(line);
-                    builder.append(line).append('\n');
+                    builder.append(line).append(System.lineSeparator());
                 }
             }
             return builder.toString().getBytes(StandardCharsets.UTF_8);

diff --git a/...dingsphere/driver/jdbc/core/driver/spi/classpath/ClasspathWithSystemPropsURLProvider.java b/...dingsphere/driver/jdbc/core/driver/spi/classpath/ClasspathWithSystemPropsURLProvider.java
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.shardingsphere.driver.jdbc.core.driver.spi.classpath;
+
+import com.google.common.base.Strings;
+import lombok.SneakyThrows;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.nio.charset.StandardCharsets;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Classpath with system properties URL provider.
+ */
+public final class ClasspathWithSystemPropsURLProvider extends AbstractClasspathURLProvider {
+
+    private static final String PATH_TYPE = "classpath-system-props:";
+
+    private static final String KEY_VALUE_SEPARATOR = "::";
+
+    private static final Pattern PATTERN = Pattern.compile("\\$\\$\\{(.+::.*)}$");
+
+    @Override
+    public boolean accept(final String url) {
+        return !Strings.isNullOrEmpty(url) && url.contains(PATH_TYPE);
+    }
+
+    @Override
+    @SneakyThrows(IOException.class)
+    public byte[] getContent(final String url, final String urlPrefix) {
+        String file = getConfigurationFile(url, urlPrefix, PATH_TYPE);
+        try (
+                InputStream stream = getResourceAsStream(file);
+                BufferedReader reader = new BufferedReader(new InputStreamReader(stream, StandardCharsets.UTF_8))) {
+            StringBuilder builder = new StringBuilder();
+            String line;
+            while (null != (line = reader.readLine())) {
+                if (!line.startsWith("#")) {
+                    line = replaceSystemProperties(line);
+                    builder.append(line).append(System.lineSeparator());
+                }
+            }
+            return builder.toString().getBytes(StandardCharsets.UTF_8);
+        }
+    }
+
+    private String replaceSystemProperties(final String line) {
+        Matcher matcher = PATTERN.matcher(line);
+        if (!matcher.find()) {
+            return line;
+        }
+        String[] systemPropNameAndDefaultValue = matcher.group(1).split(KEY_VALUE_SEPARATOR, 2);
+        String systemPropName = systemPropNameAndDefaultValue[0];
+        String systemPropValue = System.getProperty(systemPropName, systemPropNameAndDefaultValue[1]);
+        if (Strings.isNullOrEmpty(systemPropValue)) {
+            String modifiedLineWithSpace = matcher.replaceAll("");
+            return modifiedLineWithSpace.substring(0, modifiedLineWithSpace.length() - 1);
+        }
+        return matcher.replaceAll(systemPropValue);
+    }
+}
diff --git a/...-INF/services/org.apache.shardingsphere.driver.jdbc.core.driver.ShardingSphereURLProvider b/...-INF/services/org.apache.shardingsphere.driver.jdbc.core.driver.ShardingSphereURLProvider
@@ -16,6 +16,7 @@
 #
 
 org.apache.shardingsphere.driver.jdbc.core.driver.spi.absolutepath.AbsolutePathURLProvider
+org.apache.shardingsphere.driver.jdbc.core.driver.spi.absolutepath.AbsolutePathWithEnvironmentURLProvider
 org.apache.shardingsphere.driver.jdbc.core.driver.spi.classpath.ClasspathURLProvider
 org.apache.shardingsphere.driver.jdbc.core.driver.spi.classpath.ClasspathWithEnvironmentURLProvider
-org.apache.shardingsphere.driver.jdbc.core.driver.spi.absolutepath.AbsolutePathWithEnvironmentURLProvider
+org.apache.shardingsphere.driver.jdbc.core.driver.spi.classpath.ClasspathWithSystemPropsURLProvider
diff --git a/...sphere/driver/jdbc/core/driver/spi/classpath/ClasspathWithSystemPropsURLProviderTest.java b/...sphere/driver/jdbc/core/driver/spi/classpath/ClasspathWithSystemPropsURLProviderTest.java
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.shardingsphere.driver.jdbc.core.driver.spi.classpath;
+
+import org.apache.shardingsphere.driver.jdbc.core.driver.ShardingSphereURLManager;
+import org.hamcrest.Matchers;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.nullValue;
+
+public class ClasspathWithSystemPropsURLProviderTest {
+
+    private static final String FIXTURE_JDBC_URL_KEY = "fixture.config.driver.jdbc-url";
+
+    private static final String FIXTURE_USERNAME_KEY = "fixture.config.driver.username";
+
+    @BeforeAll
+    static void beforeAll() {
+        assertThat(System.getProperty(FIXTURE_JDBC_URL_KEY), Matchers.is(nullValue()));
+        assertThat(System.getProperty(FIXTURE_USERNAME_KEY), Matchers.is(nullValue()));
+        System.setProperty(FIXTURE_JDBC_URL_KEY, "jdbc:h2:mem:foo_ds_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL");
+        System.setProperty(FIXTURE_USERNAME_KEY, "sa");
+    }
+
+    @AfterAll
+    static void afterAll() {
+        System.clearProperty(FIXTURE_JDBC_URL_KEY);
+        System.clearProperty(FIXTURE_USERNAME_KEY);
+    }
+
+    @Test
+    void assertReplaceEnvironmentVariables() {
+        final String urlPrefix = "jdbc:shardingsphere:";
+        ClasspathWithSystemPropsURLProvider urlProvider = new ClasspathWithSystemPropsURLProvider();
+        byte[] actual = urlProvider.getContent("jdbc:shardingsphere:classpath-system-props:config/driver/foo-driver-system-properties-fixture.yaml", urlPrefix);
+        byte[] actualOrigin = ShardingSphereURLManager.getContent("jdbc:shardingsphere:classpath:config/driver/foo-driver-fixture.yaml", urlPrefix);
+        assertThat(actual, is(actualOrigin));
+    }
+}
diff --git a/jdbc/core/src/test/resources/config/driver/foo-driver-system-properties-fixture.yaml b/jdbc/core/src/test/resources/config/driver/foo-driver-system-properties-fixture.yaml
@@ -0,0 +1,58 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# After `ShardingSphereURLManager.getContent`, this file should be equivalent to `foo-driver-fixture.yaml` in the same folder.
+databaseName: foo_driver_fixture_db
+
+dataSources:
+  ds_0:
+    dataSourceClassName: com.zaxxer.hikari.HikariDataSource
+    driverClassName: org.h2.Driver
+    jdbcUrl: jdbc:h2:mem:foo_ds_0;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL
+    username: sa
+    password:
+  ds_1:
+    dataSourceClassName: com.zaxxer.hikari.HikariDataSource
+    driverClassName: $${fixture.config.driver.driver-class-name::org.h2.Driver}
+    jdbcUrl: $${fixture.config.driver.jdbc-url::jdbc:h2:mem:foo_ds_do_not_use}
+    username: $${fixture.config.driver.username::}
+    password: $${fixture.config.driver.password::}
+
+rules:
+  - !SHARDING
+    autoTables:
+      t_order:
+        actualDataSources: ds_0,ds_1
+        shardingStrategy:
+          standard:
+            shardingColumn: order_id
+            shardingAlgorithmName: auto_mod
+        keyGenerateStrategy:
+          column: user_id
+          keyGeneratorName: snowflake
+    shardingAlgorithms:
+      auto_mod:
+        type: HASH_MOD
+        props:
+          sharding-count: 2
+
+    keyGenerators:
+      snowflake:
+        type: SNOWFLAKE
+
+props:
+  sql-show: true