Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor shardingsphere-infra-expr to expose the use of Row Value Expressions SPI #28340

Merged
merged 2 commits into from
Sep 23, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions docs/document/content/dev-manual/sharding.cn.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,3 +79,20 @@ chapter = true
|--------------------------|--------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| DatabaseTimestampService | 从数据库中获取当前时间进行路由 | [`org.apache.shardingsphere.timeservice.type.database.DatabaseTimestampService`](https://github.com/apache/shardingsphere/blob/master/kernel/time-service/type/database/src/main/java/org/apache/shardingsphere/timeservice/type/database/DatabaseTimestampService.java) |
| SystemTimestampService | 从应用系统时间中获取当前时间进行路由 | [`org.apache.shardingsphere.timeservice.type.system.SystemTimestampService`](https://github.com/apache/shardingsphere/blob/master/kernel/time-service/type/system/src/main/java/org/apache/shardingsphere/timeservice/type/system/SystemTimestampService.java) |

## InlineExpressionParser

### 全限定类名

`org.apache.shardingsphere.infra.expr.core.InlineExpressionParser`

### 定义

解析行表达式

### 已知实现

| *配置标识* | *详细说明* | *全限定类名* |
|----------|-------------------|--------------------------------------------------------------------------------|
| GROOVY | 使用 Groovy 语法的行表达式 | `org.apache.shardingsphere.infra.expr.groovy.GroovyInlineExpressionParser` |
| PURELIST | 使用标准列表的行表达式 | `org.apache.shardingsphere.infra.expr.purelist.PureListInlineExpressionParser` |
17 changes: 17 additions & 0 deletions docs/document/content/dev-manual/sharding.en.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,3 +79,20 @@ Obtain the current date for routing definition
|--------------------------|--------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| DatabaseTimestampService | Get the current time from the database for routing | [`org.apache.shardingsphere.timeservice.type.database.DatabaseTimestampService`](https://github.com/apache/shardingsphere/blob/master/kernel/time-service/type/database/src/main/java/org/apache/shardingsphere/timeservice/type/database/DatabaseTimestampService.java) |
| SystemTimestampService | Get the current time from the application system for routing | [`org.apache.shardingsphere.timeservice.type.system.SystemTimestampService`](https://github.com/apache/shardingsphere/blob/master/kernel/time-service/type/system/src/main/java/org/apache/shardingsphere/timeservice/type/system/SystemTimestampService.java) |

## InlineExpressionParser

### Fully-qualified class name

`org.apache.shardingsphere.infra.expr.core.InlineExpressionParser`

### Definition

Row Value Expressions definition

### Implementation classes

| *Configuration Type* | *Description* | *Fully-qualified class name* |
|----------------------|---------------------------------------------------|--------------------------------------------------------------------------------|
| GROOVY | Row Value Expressions that uses the Groovy syntax | `org.apache.shardingsphere.infra.expr.groovy.GroovyInlineExpressionParser` |
| PURELIST | Row Value Expressions that uses a standard list | `org.apache.shardingsphere.infra.expr.purelist.PureListInlineExpressionParser` |
2 changes: 1 addition & 1 deletion docs/document/content/faq/_index.cn.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ DROP DATABASE sharding_db;

回答:

行表达式标识符可以使用 `${...}` 或 `$->{...}`,但前者与 Spring 本身的属性文件占位符冲突,因此在 Spring 环境中使用行表达式标识符建议使用 `$->{...}`。
使用 `InlineExpressionParser` SPI 的默认实现的行表达式标识符可以使用 `${...}` 或 `$->{...}`,但前者与 Spring 本身的属性文件占位符冲突,因此在 Spring 环境中使用行表达式标识符建议使用 `$->{...}`。

### [分片] inline 表达式返回结果为何出现浮点数?

Expand Down
3 changes: 2 additions & 1 deletion docs/document/content/faq/_index.en.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,8 @@ Answer:

Answer:

`${...}` or `$->{...}` can be used in inline expression identifiers, but the former one clashes with place holders in Spring property files, so `$->{...}` is recommended to be used in Spring as inline expression identifiers.
`${...}` or `$->{...}` can be used in inline expression identifiers using the default implementation of the
`InlineExpressionParser` SPI, but the former one clashes with place holders in Spring property files, so `$->{...}` is recommended to be used in Spring as inline expression identifiers.

### [Sharding] Why does float number appear in the return result of inline expression?

Expand Down
7 changes: 7 additions & 0 deletions docs/document/content/features/sharding/concept.cn.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,13 @@ db0.t_order0, db0.t_order1, db1.t_order2, db1.t_order3, db1.t_order4

对于常见的分片算法,使用 Java 代码实现并不有助于配置的统一管理。 通过行表达式书写分片算法,可以有效地将规则配置一同存放,更加易于浏览与存储。

行表达式作为字符串由两部分组成,分别是字符串开头的对应 SPI 实现的 Type Name 部分和表达式部分。 以 `<GROOVY>t_order_${1..3}` 为例,字符
串`<GROOVY>` 部分的子字符串 `GROOVY` 为此行表达式使用的对应 SPI 实现的 Type Name,其被 `<>` 符号包裹来识别。而字符串 `t_order_${1..3}`
为此行表达式的表达式部分。当行表达式不指定 Type Name 时,例如 `t_order_${1..3}`,行表示式默认将使用 `InlineExpressionParser` SPI 的
`GROOVY` 实现来解析表达式。

以下部分介绍 `GROOVY` 实现的语法规则。

行表达式的使用非常直观,只需要在配置中使用 `${ expression }` 或 `$->{ expression }` 标识行表达式即可。 目前支持数据节点和分片算法这两个部分的配置。 行表达式的内容使用的是 Groovy 的语法,Groovy 能够支持的所有操作,行表达式均能够支持。 例如:

`${begin..end}` 表示范围区间
Expand Down
9 changes: 9 additions & 0 deletions docs/document/content/features/sharding/concept.en.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,15 @@ Row expressions are designed to address the two main issues of configuration sim

For the common sharding algorithm, using Java code implementation does not help to manage the configuration uniformly. But by writing the sharding algorithm through line expressions, the rule configuration can be effectively stored together, which is easier to browse and store.

A Row Value Expressions consists of two parts as a string, the Type Name part of the corresponding SPI implementation at the beginning of the string and the expression part.

Take `<GROOVY>t_order_${1..3}` as sample, the `GROOVY` substring in the part of the `<GROOVY>` string is the Type Name used by the corresponding SPI implementation for this Row Value Expressions, which is identified by the `<>` symbol.
And the `t_order_${1..3}` string is the expression part of this Row Value Expressions. When a Row Value Expressions does
not specify a Type Name, such as `t_order_${1..3}`, the Row Value Expressions defaults to parse expressions by `GROOVY` implementation for `InlineExpressionParser` SPI.

The following sections describe the syntax rules for the `GROOVY` implementation.


Row expressions are very intuitive, just use `${ expression }` or `$->{ expression }` in the configuration to identify the row expressions. Data nodes and sharding algorithms are currently supported. The content of row expressions uses Groovy syntax, and all operations supported by Groovy are supported by row expressions. For example:

`${begin..end}` denotes the range interval
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
+++
title = "行表达式"
weight = 7
+++

## 使用 Groovy 语法的行表达式

类型:GROOVY

可配置属性:


## 使用标准列表的行表达式

类型:PURELIST

可配置属性:


## 操作步骤

使用需要使用 `行表达式` 的属性时, 如在 `数据分片` 功能中, 在 `actualDataNodes` 属性下指明特定的 SPI 实现的 Type Name 即可。

若 `行表达式` 不指明 SPI 的 Type Name,默认将使用 `GROOVY` 的 SPI 实现。

## 配置示例

```yaml
rules:
- !SHARDING
tables:
t_order:
actualDataNodes: <PURELIST>ds_0.t_order_0, ds_0.t_order_1, ds_1.t_order_0, ds_1.t_order_1
tableStrategy:
standard:
shardingColumn: order_id
shardingAlgorithmName: t_order_inline
keyGenerateStrategy:
column: order_id
keyGeneratorName: snowflake
defaultDatabaseStrategy:
standard:
shardingColumn: user_id
shardingAlgorithmName: database_inline
shardingAlgorithms:
database_inline:
type: INLINE
props:
algorithm-expression: <GROOVY>ds_${user_id % 2}
t_order_inline:
type: INLINE
props:
algorithm-expression: <GROOVY>t_order_${order_id % 2}
keyGenerators:
snowflake:
type: SNOWFLAKE
```

## 相关参考

- [核心概念](/docs/document/content/features/sharding/concept.cn.md)
- [数据分片](/docs/document/content/dev-manual/sharding.cn.md)
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
+++
title = "Row Value Expressions"
weight = 7
+++

## Row Value Expressions that uses the Groovy syntax

Type: GROOVY

Attributes:

None

## Row Value Expressions that uses a standard list

Type: PURELIST

Attributes:

None

## Procedure

When using attributes that require the use of `Row Value Expressions`, such as in the `data sharding` feature, it is
sufficient to indicate the Type Name of the specific SPI implementation under the `actualDataNodes` attribute.

If the `Row Value Expressions` does not indicate the Type Name of the SPI, the SPI implementation of `GROOVY` will be
used by default.

## Sample

```yaml
rules:
- !SHARDING
tables:
t_order:
actualDataNodes: <PURELIST>ds_0.t_order_0, ds_0.t_order_1, ds_1.t_order_0, ds_1.t_order_1
tableStrategy:
standard:
shardingColumn: order_id
shardingAlgorithmName: t_order_inline
keyGenerateStrategy:
column: order_id
keyGeneratorName: snowflake
defaultDatabaseStrategy:
standard:
shardingColumn: user_id
shardingAlgorithmName: database_inline
shardingAlgorithms:
database_inline:
type: INLINE
props:
algorithm-expression: <GROOVY>ds_${user_id % 2}
t_order_inline:
type: INLINE
props:
algorithm-expression: <GROOVY>t_order_${order_id % 2}
keyGenerators:
snowflake:
type: SNOWFLAKE
```

## Related References

- [Core Concept](/docs/document/content/features/sharding/concept.en.md)
- [Data Sharding](/docs/document/content/dev-manual/sharding.en.md)
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ Apache ShardingSphere 内置的标准分片算法实现类包括:

#### 行表达式分片算法

使用 Groovy 的表达式,提供对 SQL 语句中的 `=` 和 `IN` 的分片操作支持,只支持单分片键。
使用 `InlineExpressionParser` SPI 的默认实现的 Groovy 的表达式,提供对 SQL 语句中的 `=` 和 `IN` 的分片操作支持,只支持单分片键。
对于简单的分片算法,可以通过简单的配置使用,从而避免繁琐的 Java 代码开发,如: `t_user_$->{u_id % 8}` 表示 `t_user` 表根据 `u_id` 模 8,而分成 8 张表,表名称为 `t_user_0` 到 `t_user_7`。
详情请参见[行表达式](/cn/dev-manual/sharding/#implementation-classes)。

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,8 @@ Apache ShardingSphere built-in standard sharding algorithm are:

#### Inline Sharding Algorithm

With Groovy expressions, `InlineShardingStrategy` provides single-key support for the sharding operation of `=` and `IN` in SQL.
With Groovy expressions that uses the default implementation of the `InlineExpressionParser` SPI,
`InlineShardingStrategy` provides single-key support for the sharding operation of `=` and `IN` in SQL.
Simple sharding algorithms can be used through a simple configuration to avoid laborious Java code developments.
For example, `t_user_$->{u_id % 8}` means table t_user is divided into 8 tables according to u_id, with table names from `t_user_0` to `t_user_7`.
Please refer to [Inline Expression](/en/dev-manual/sharding/#implementation-classes) for more details.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,10 @@ services:
- `org.apache.shardingsphere.sharding.algorithm.sharding.inline.ComplexInlineShardingAlgorithm`
- `org.apache.shardingsphere.sharding.algorithm.sharding.hint.HintInlineShardingAlgorithm`

- 当前阶段,GraalVM Native Image 形态的 ShardingSphere Proxy 不支持使用带 Groovy
语法的 `行表达式`, 这首先导致 `数据分片` 功能的`actualDataNodes`属性只能使用纯列表来配置, 例如 `ds_0.t_order_0, ds_0.t_order_1`
或 `ds_0.t_user_0, ds_15.t_user_1023`。此问题在 https://github.com/oracle/graal/issues/5522 追踪。
- 当前阶段,GraalVM Native Image 形态的 ShardingSphere Proxy 不支持使用 `InlineExpressionParser` SPI 的默认实现的 `行表达式`,
这首先导致 `数据分片` 功能的`actualDataNodes` 属性只能使用其他 `InlineExpressionParser` SPI 的实现来配置, 例如使用
`InlineExpressionParser` SPI 实现为 `PURELIST` 的 `行表达式`, 即 `<PURELIST>ds_0.t_order_0, ds_0.t_order_1`
或 `<PURELIST>ds_0.t_user_0, ds_15.t_user_1023`。

- 本节假定处于 Linux(amd64,aarch64), MacOS(amd64)或 Windows(amd64)环境。
如果你位于 MacOS(aarch64/M1) 环境,你需要关注尚未关闭的 https://github.com/oracle/graal/issues/2666 。
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,16 +42,18 @@ services:
- `org.apache.shardingsphere.sharding.algorithm.sharding.inline.ComplexInlineShardingAlgorithm`
- `org.apache.shardingsphere.sharding.algorithm.sharding.hint.HintInlineShardingAlgorithm`

- At this stage, ShardingSphere Proxy in the form of GraalVM Native Image does not support the use
of `Row Value Expressions` with Groovy syntax, which first results in the `actualDataNodes` property of the `Sharding`
feature being only configurable using a pure list, such as `ds_0.t_order_0, ds_0.t_order_1`
or `ds_0.t_user_0, ds_15.t_user_1023`. This issue is tracked in https://github.com/oracle/graal/issues/5522 .
- At this stage, ShardingSphere Proxy in the form of GraalVM Native Image does not support `row expressions` using the
default implementation of the `InlineExpressionParser` SPI.
This first results in the `actualDataNodes` property of the `data sharding` feature that can only be configured using
other implementations of the `InlineExpressionParser` SPI, for example using `PURELIST` implemented
`InlineExpressionParser` SPI for `row expression`, i.e. `<PURELIST>ds_0.t_order_0, ds_0.t_order_1`
or `<PURELIST>ds_0.t_user_0, ds_15.t_user_1023`.

- This section assumes a Linux (amd64, aarch64), MacOS (amd64) or Windows (amd64) environment.
If you are on MacOS (aarch64/M1) environment, you need to follow https://github.com/oracle/graal/issues/2666 which is
not closed yet.

- 'org.apache.shardingsphere:shardingsphere-cluster-mode-repository-etcd' is affected by
- `org.apache.shardingsphere:shardingsphere-cluster-mode-repository-etcd` is affected by
https://github.com/micronaut-projects/micronaut-gcp/issues/532 and cannot be used.

## Premise
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ private void checkDataSources(final String databaseName, final Map<String, DataS

private void checkWriteDataSourceNames(final String databaseName, final Map<String, DataSource> dataSourceMap, final Collection<String> addedWriteDataSourceNames,
final ReadwriteSplittingDataSourceRuleConfiguration config, final Collection<ShardingSphereRule> rules) {
for (String each : InlineExpressionParserFactory.newInstance().splitAndEvaluate(config.getWriteDataSourceName())) {
for (String each : InlineExpressionParserFactory.newInstance(config.getWriteDataSourceName()).splitAndEvaluate()) {
ShardingSpherePreconditions.checkState(dataSourceMap.containsKey(each) || containsInOtherRules(each, rules),
() -> new DataSourceNameExistedException(String.format("Write data source name `%s` not in database `%s`.", each, databaseName)));
ShardingSpherePreconditions.checkState(addedWriteDataSourceNames.add(each),
Expand All @@ -97,7 +97,7 @@ private boolean containsInOtherRules(final String datasourceName, final Collecti
}

private void checkReadeDataSourceNames(final String databaseName, final Map<String, DataSource> dataSourceMap, final Collection<String> addedReadDataSourceNames, final String readDataSourceName) {
for (String each : InlineExpressionParserFactory.newInstance().splitAndEvaluate(readDataSourceName)) {
for (String each : InlineExpressionParserFactory.newInstance(readDataSourceName).splitAndEvaluate()) {
ShardingSpherePreconditions.checkState(dataSourceMap.containsKey(each),
() -> new DataSourceNameExistedException(String.format("Read data source name `%s` not in database `%s`.", each, databaseName)));
ShardingSpherePreconditions.checkState(addedReadDataSourceNames.add(each),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -103,10 +103,10 @@ private Map<String, ReadwriteSplittingDataSourceRule> createDataSourceRules(fina

private Map<String, ReadwriteSplittingDataSourceRule> createStaticDataSourceRules(final ReadwriteSplittingDataSourceRuleConfiguration config,
final ReadQueryLoadBalanceAlgorithm loadBalanceAlgorithm) {
List<String> inlineReadwriteDataSourceNames = InlineExpressionParserFactory.newInstance().splitAndEvaluate(config.getName());
List<String> inlineWriteDatasourceNames = InlineExpressionParserFactory.newInstance().splitAndEvaluate(config.getWriteDataSourceName());
List<String> inlineReadwriteDataSourceNames = InlineExpressionParserFactory.newInstance(config.getName()).splitAndEvaluate();
List<String> inlineWriteDatasourceNames = InlineExpressionParserFactory.newInstance(config.getWriteDataSourceName()).splitAndEvaluate();
List<List<String>> inlineReadDatasourceNames = config.getReadDataSourceNames().stream()
.map(each -> InlineExpressionParserFactory.newInstance().splitAndEvaluate(each)).collect(Collectors.toList());
.map(each -> InlineExpressionParserFactory.newInstance(each).splitAndEvaluate()).collect(Collectors.toList());
ShardingSpherePreconditions.checkState(inlineWriteDatasourceNames.size() == inlineReadwriteDataSourceNames.size(),
() -> new InvalidInlineExpressionDataSourceNameException("Inline expression write data source names size error."));
inlineReadDatasourceNames.forEach(each -> ShardingSpherePreconditions.checkState(each.size() == inlineReadwriteDataSourceNames.size(),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ public void init(final Properties props) {
private String getAlgorithmExpression(final Properties props) {
String algorithmExpression = props.getProperty(ALGORITHM_EXPRESSION_KEY, DEFAULT_ALGORITHM_EXPRESSION);
ShardingSpherePreconditions.checkNotNull(algorithmExpression, () -> new ShardingAlgorithmInitializationException(getType(), "Inline sharding algorithm expression can not be null."));
return InlineExpressionParserFactory.newInstance().handlePlaceHolder(algorithmExpression.trim());
return InlineExpressionParserFactory.newInstance(algorithmExpression.trim()).handlePlaceHolder();
}

@Override
Expand All @@ -67,7 +67,7 @@ private String doSharding(final Comparable<?> shardingValue) {
}

private Closure<?> createClosure() {
Closure<?> result = InlineExpressionParserFactory.newInstance().evaluateClosure(algorithmExpression).rehydrate(new Expando(), null, null);
Closure<?> result = InlineExpressionParserFactory.newInstance(algorithmExpression).evaluateClosure().rehydrate(new Expando(), null, null);
result.setResolveStrategy(Closure.DELEGATE_ONLY);
return result;
}
Expand Down
Loading