Skip to content

Commit

Permalink
Add ClasspathURLProvider implementation that reads environment variables
Browse files Browse the repository at this point in the history
  • Loading branch information
linghengqian committed Jan 29, 2024
1 parent 678dac7 commit 58927d4
Show file tree
Hide file tree
Showing 8 changed files with 321 additions and 23 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ chapter = true
在解析并加载 YAML 文件为 ShardingSphere 的元数据后,
会再次通过[模式配置](../../../java-api/mode)的相关配置决定下一步行为。讨论两种情况:
1. 元数据持久化仓库中不存在 ShardingSphere 的元数据,本地元数据将被存储到元数据持久化仓库。
2. 元数据持久化仓库中已存在与本地元数据不同的 ShardingSphere 的元数据,本地元数据将被元数据持久化仓库的元数据覆盖。
2. 元数据持久化仓库中已存在 ShardingSphere 的元数据,无论是否与本地元数据相同,本地元数据将被元数据持久化仓库的元数据覆盖。

对元数据持久化仓库的配置需参考[元数据持久化仓库](../../../../common-config/builtin-algorithm/metadata-repository)

Expand All @@ -33,5 +33,39 @@ chapter = true
用例:
- `jdbc:shardingsphere:absolutepath:/path/to/config.yaml`

### 从类路径中加载包含环境变量的配置文件

加载 classpath 中包含环境变量的 config.yaml 配置文件的 JDBC URL,通过 `jdbc:shardingsphere:classpath-environment:` 前缀识别。
配置文件为 `xxx.yaml`,配置文件格式与 [YAML 配置](../../../yaml-config)基本一致。
在涉及的 YAML 文件中,允许通过环境变量设置特定YAML属性的值,并配置可选的默认值。这常用于 Docker Image 的部署场景。
环境变量的名称和其可选的默认值通过`::`分割,在最外层通过`$${``}`包裹。

讨论两种情况。
1. 当对应的环境变量不存在时,此 YAML 属性的值将被设置为`::`右侧的默认值。
2. 当对应的环境变量和`::`右侧的默认值均不存在时,此属性将被设置为空。

假设存在以下一组环境变量,
1. 存在环境变量`FIXTURE_JDBC_URL``jdbc:h2:mem:foo_ds_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL`
2. 存在环境变量`FIXTURE_USERNAME``sa`

则对于以下 YAML 文件的截取片段,
```yaml
ds_1:
dataSourceClassName: com.zaxxer.hikari.HikariDataSource
driverClassName: org.h2.Driver
jdbcUrl: $${FIXTURE_JDBC_URL::jdbc:h2:mem:foo_ds_do_not_use}
username: $${FIXTURE_USERNAME::}
password: $${FIXTURE_PASSWORD::}
```
此 YAML 截取片段将被解析为,
```yaml
ds_1:
dataSourceClassName: com.zaxxer.hikari.HikariDataSource
driverClassName: org.h2.Driver
jdbcUrl: jdbc:h2:mem:foo_ds_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL
username: sa
password:
```
### 其他实现
具体可参考 https://github.com/apache/shardingsphere-plugin 。
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,11 @@ allows YAML configuration files to be fetched from multiple sources and File Sys

After parsing and loading the YAML file into ShardingSphere's metadata,
The next behavior will be determined again through the relevant configuration of [Mode Configuration](../../../java-api/mode). Discuss two situations:
1. ShardingSphere’s metadata does not exist in the metadata persistence warehouse, and local metadata will be stored in the metadata persistence warehouse.
2. Metadata of ShardingSphere that is different from local metadata already exists in the metadata persistence warehouse, and the local metadata will be overwritten by the metadata of the metadata persistence warehouse.
1. ShardingSphere’s metadata does not exist in the metadata repository, and local metadata will be stored in the metadata repository.
2. ShardingSphere’s metadata already exists in the metadata repository, regardless of whether it is the same as the local metadata,
the local metadata will be overwritten by the metadata of the metadata repository.

For the configuration of the metadata persistence warehouse, please refer to [Metadata Persistence Warehouse](../../../../common-config/builtin-algorithm/metadata-repository).
For the configuration of the metadata repository, please refer to [Metadata Repository](../../../../common-config/builtin-algorithm/metadata-repository).

## URL configuration

Expand All @@ -33,5 +34,40 @@ The configuration file is `xxx.yaml`, and the configuration file format is consi
Example:
- `jdbc:shardingsphere:absolutepath:/path/to/config.yaml`

### Load configuration file containing environment variables from classpath

JDBC URL to load the config.yaml configuration file that contains environment variables in classpath, identified by the `jdbc:shardingsphere:classpath-environment:` prefix.
The configuration file is `xxx.yaml`, and the configuration file format is basically the same as [YAML configuration](../../../yaml-config).
Allows setting the value of specific YAML properties via environment variables and configuring optional default values in the involved YAML files.
This is commonly used in Docker Image deployment scenarios.
The name of an environment variable and its optional default value are separated by `::` and wrapped in the outermost layer by `$${` and `}`.

Discuss two situations.
1. When the corresponding environment variable does not exist, the value of this YAML attribute will be set to the default value on the right side of `::`.
2. When the corresponding environment variable and the default value on the right side of `::` do not exist, this property will be set to empty.

Assume that the following set of environment variables exists,
1. The existing environment variable `FIXTURE_JDBC_URL` is `jdbc:h2:mem:foo_ds_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL`.
2. The existing environment variable `FIXTURE_USERNAME` is `sa`.

Then for the intercepted fragment of the following YAML file,
```yaml
ds_1:
dataSourceClassName: com.zaxxer.hikari.HikariDataSource
driverClassName: org.h2.Driver
jdbcUrl: $${FIXTURE_JDBC_URL::jdbc:h2:mem:foo_ds_do_not_use}
username: $${FIXTURE_USERNAME::}
password: $${FIXTURE_PASSWORD::}
```
This YAML snippet will be parsed as,
```yaml
ds_1:
dataSourceClassName: com.zaxxer.hikari.HikariDataSource
driverClassName: org.h2.Driver
jdbcUrl: jdbc:h2:mem:foo_ds_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL
username: sa
password:
```
### Other implementations
For details, please refer to https://github.com/apache/shardingsphere-plugin .
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.shardingsphere.driver.jdbc.core.driver.spi.classpath;

import com.google.common.base.Preconditions;
import org.apache.shardingsphere.driver.jdbc.core.driver.ShardingSphereURLProvider;

import java.io.InputStream;

/**
* Abstract classpath URL provider.
*/
public abstract class AbstractClasspathURLProvider implements ShardingSphereURLProvider {

String getConfigurationFile(final String url, final String urlPrefix, final String typePrefix) {
String configuredFile = url.substring(urlPrefix.length(), url.contains("?") ? url.indexOf('?') : url.length());
String file = configuredFile.substring(typePrefix.length());
Preconditions.checkArgument(!file.isEmpty(), "Configuration file is required in ShardingSphere URL.");
return file;
}

InputStream getResourceAsStream(final String resource) {
InputStream result = Thread.currentThread().getContextClassLoader().getResourceAsStream(resource);
result = null == result ? Thread.currentThread().getContextClassLoader().getResourceAsStream("/" + resource) : result;
if (null != result) {
return result;
}
throw new IllegalArgumentException(String.format("Can not find configuration file `%s`.", resource));
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,10 @@
* limitations under the License.
*/

package org.apache.shardingsphere.driver.jdbc.core.driver.spi;
package org.apache.shardingsphere.driver.jdbc.core.driver.spi.classpath;

import com.google.common.base.Preconditions;
import com.google.common.base.Strings;
import lombok.SneakyThrows;
import org.apache.shardingsphere.driver.jdbc.core.driver.ShardingSphereURLProvider;

import java.io.BufferedReader;
import java.io.IOException;
Expand All @@ -31,21 +29,19 @@
/**
* Classpath URL provider.
*/
public final class ClasspathURLProvider implements ShardingSphereURLProvider {
public final class ClasspathURLProvider extends AbstractClasspathURLProvider {

private static final String CLASSPATH_TYPE = "classpath:";
private static final String TYPE_PREFIX = "classpath:";

@Override
public boolean accept(final String url) {
return !Strings.isNullOrEmpty(url) && url.contains(CLASSPATH_TYPE);
return !Strings.isNullOrEmpty(url) && url.contains(TYPE_PREFIX);
}

@Override
@SneakyThrows(IOException.class)
public byte[] getContent(final String url, final String urlPrefix) {
String configuredFile = url.substring(urlPrefix.length(), url.contains("?") ? url.indexOf('?') : url.length());
String file = configuredFile.substring(CLASSPATH_TYPE.length());
Preconditions.checkArgument(!file.isEmpty(), "Configuration file is required in ShardingSphere URL.");
String file = getConfigurationFile(url, urlPrefix, TYPE_PREFIX);
try (
InputStream stream = getResourceAsStream(file);
BufferedReader reader = new BufferedReader(new InputStreamReader(stream, StandardCharsets.UTF_8))) {
Expand All @@ -59,13 +55,4 @@ public byte[] getContent(final String url, final String urlPrefix) {
return builder.toString().getBytes(StandardCharsets.UTF_8);
}
}

private InputStream getResourceAsStream(final String resource) {
InputStream result = Thread.currentThread().getContextClassLoader().getResourceAsStream(resource);
result = null == result ? Thread.currentThread().getContextClassLoader().getResourceAsStream("/" + resource) : result;
if (null != result) {
return result;
}
throw new IllegalArgumentException(String.format("Can not find configuration file `%s`.", resource));
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.shardingsphere.driver.jdbc.core.driver.spi.classpath;

import com.google.common.base.Strings;
import lombok.SneakyThrows;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.nio.charset.StandardCharsets;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

/**
* Classpath with environment variables URL provider.
*/
public final class ClasspathWithEnvironmentURLProvider extends AbstractClasspathURLProvider {

private static final String TYPE_PREFIX = "classpath-environment:";

private static final String KEY_VALUE_SEPARATOR = "::";

@SuppressWarnings("RegExpRedundantEscape")
private static final Pattern PATTERN = Pattern.compile("\\$\\$\\{(.+::.*)\\}$");

@Override
public boolean accept(final String url) {
return !Strings.isNullOrEmpty(url) && url.contains(TYPE_PREFIX);
}

@Override
@SneakyThrows(IOException.class)
public byte[] getContent(final String url, final String urlPrefix) {
String file = getConfigurationFile(url, urlPrefix, TYPE_PREFIX);
try (
InputStream stream = getResourceAsStream(file);
BufferedReader reader = new BufferedReader(new InputStreamReader(stream, StandardCharsets.UTF_8))) {
StringBuilder builder = new StringBuilder();
String line;
while (null != (line = reader.readLine())) {
if (!line.startsWith("#")) {
line = replaceEnvironmentVariables(line);
builder.append(line).append('\n');
}
}
return builder.toString().getBytes(StandardCharsets.UTF_8);
}
}

private String replaceEnvironmentVariables(final String line) {
Matcher matcher = PATTERN.matcher(line);
if (matcher.find()) {
StringBuffer modifiedLine = new StringBuffer();
String[] envNameAndDefaultValue = matcher.group(1).split(KEY_VALUE_SEPARATOR, 2);
String envName = envNameAndDefaultValue[0];
String envValue = getEnvironmentVariables(envName);
if (Strings.isNullOrEmpty(envValue) && envNameAndDefaultValue[1].isEmpty()) {
matcher.appendReplacement(modifiedLine, "");
return modifiedLine.substring(0, modifiedLine.length() - 1);
}
if (Strings.isNullOrEmpty(envValue)) {
envValue = envNameAndDefaultValue[1];
}
matcher.appendReplacement(modifiedLine, envValue);
return modifiedLine.toString();
}
return line;
}

/**
* This method is only used for mocking environment variables in unit tests and should not be used under any circumstances.
*
* @param name the name of the environment variable
* @return the string value of the variable, or null if the variable is not defined in the system environment
*/
String getEnvironmentVariables(final String name) {
return System.getenv(name);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,5 @@
#

org.apache.shardingsphere.driver.jdbc.core.driver.spi.AbsolutePathURLProvider
org.apache.shardingsphere.driver.jdbc.core.driver.spi.ClasspathURLProvider
org.apache.shardingsphere.driver.jdbc.core.driver.spi.classpath.ClasspathURLProvider
org.apache.shardingsphere.driver.jdbc.core.driver.spi.classpath.ClasspathWithEnvironmentURLProvider
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.shardingsphere.driver.jdbc.core.driver.spi.classpath;

import org.apache.shardingsphere.driver.jdbc.core.driver.ShardingSphereURLManager;
import org.junit.jupiter.api.Test;

import static org.hamcrest.CoreMatchers.is;
import static org.hamcrest.MatcherAssert.assertThat;
import static org.mockito.Mockito.spy;
import static org.mockito.Mockito.when;

public class ClasspathWithEnvironmentURLProviderTest {

@Test
void assertReplaceEnvironmentVariables() {
final String urlPrefix = "jdbc:shardingsphere:";
ClasspathWithEnvironmentURLProvider spy = spy(new ClasspathWithEnvironmentURLProvider());
when(spy.getEnvironmentVariables("FIXTURE_JDBC_URL")).thenReturn("jdbc:h2:mem:foo_ds_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL");
when(spy.getEnvironmentVariables("FIXTURE_USERNAME")).thenReturn("sa");
byte[] actual = spy.getContent("jdbc:shardingsphere:classpath-environment:config/driver/foo-driver-environment-variables-fixture.yaml", urlPrefix);
byte[] actualOrigin = ShardingSphereURLManager.getContent("jdbc:shardingsphere:classpath:config/driver/foo-driver-fixture.yaml", urlPrefix);
assertThat(actual.length, is(999));
assertThat(actual, is(actualOrigin));
}
}
Loading

0 comments on commit 58927d4

Please sign in to comment.