Merge pull request #23 from HongW2019/doc-1.1.1

[PMEM-SHUFFLE-22]Update the documents for OAP 1.1.1

CHANGELOG.md

@@ -1,13 +1,151 @@
 # Change log
-Generated on 2021-04-29
+Generated on 2021-06-02
+
+## Release 1.1.1
+
+### Native SQL Engine
+
+#### Features
+|||
+|:---|:---|
+|[#304](https://github.com/oap-project/native-sql-engine/issues/304)|Upgrade to Arrow 4.0.0|
+|[#285](https://github.com/oap-project/native-sql-engine/issues/285)|ColumnarWindow: Support Date/Timestamp input in MAX/MIN|
+|[#297](https://github.com/oap-project/native-sql-engine/issues/297)|Disable incremental compiler in CI|
+|[#245](https://github.com/oap-project/native-sql-engine/issues/245)|Support columnar rdd cache|
+|[#276](https://github.com/oap-project/native-sql-engine/issues/276)|Add option to switch Hadoop version|
+|[#274](https://github.com/oap-project/native-sql-engine/issues/274)|Comment to trigger tpc-h RAM test|
+|[#256](https://github.com/oap-project/native-sql-engine/issues/256)|CI: do not run ram report for each PR|
+
+#### Bugs Fixed
+|||
+|:---|:---|
+|[#325](https://github.com/oap-project/native-sql-engine/issues/325)|java.util.ConcurrentModificationException: mutation occurred during iteration|
+|[#329](https://github.com/oap-project/native-sql-engine/issues/329)|numPartitions are not the same|
+|[#318](https://github.com/oap-project/native-sql-engine/issues/318)|fix Spark 311 on data source v2|
+|[#311](https://github.com/oap-project/native-sql-engine/issues/311)|Build reports errors|
+|[#302](https://github.com/oap-project/native-sql-engine/issues/302)|test on v2 failed due to an exception|
+|[#257](https://github.com/oap-project/native-sql-engine/issues/257)|different version of slf4j-log4j|
+|[#293](https://github.com/oap-project/native-sql-engine/issues/293)|Fix BHJ loss if key = 0|
+|[#248](https://github.com/oap-project/native-sql-engine/issues/248)|arrow dependency must put after arrow installation|
+
+#### PRs
+|||
+|:---|:---|
+|[#332](https://github.com/oap-project/native-sql-engine/pull/332)|[NSE-325] fix incremental compile issue with 4.5.x scala-maven-plugin|
+|[#335](https://github.com/oap-project/native-sql-engine/pull/335)|[NSE-329] fix out partitioning in BHJ and SHJ|
+|[#328](https://github.com/oap-project/native-sql-engine/pull/328)|[NSE-318]check schema before reuse exchange|
+|[#307](https://github.com/oap-project/native-sql-engine/pull/307)|[NSE-304] Upgrade to Arrow 4.0.0|
+|[#312](https://github.com/oap-project/native-sql-engine/pull/312)|[NSE-311] Build reports errors|
+|[#272](https://github.com/oap-project/native-sql-engine/pull/272)|[NSE-273] support spark311|
+|[#303](https://github.com/oap-project/native-sql-engine/pull/303)|[NSE-302] fix v2 test|
+|[#306](https://github.com/oap-project/native-sql-engine/pull/306)|[NSE-304] Upgrade to Arrow 4.0.0: Change basic GHA TPC-H test target …|
+|[#286](https://github.com/oap-project/native-sql-engine/pull/286)|[NSE-285] ColumnarWindow: Support Date input in MAX/MIN|
+|[#298](https://github.com/oap-project/native-sql-engine/pull/298)|[NSE-297] Disable incremental compiler in GHA CI|
+|[#291](https://github.com/oap-project/native-sql-engine/pull/291)|[NSE-257] fix multiple slf4j bindings|
+|[#294](https://github.com/oap-project/native-sql-engine/pull/294)|[NSE-293] fix unsafemap with key = '0'|
+|[#233](https://github.com/oap-project/native-sql-engine/pull/233)|[NSE-207] fix issues found from aggregate unit tests|
+|[#246](https://github.com/oap-project/native-sql-engine/pull/246)|[NSE-245]Adding columnar RDD cache support|
+|[#289](https://github.com/oap-project/native-sql-engine/pull/289)|[NSE-206]Update installation guide and configuration guide.|
+|[#277](https://github.com/oap-project/native-sql-engine/pull/277)|[NSE-276] Add option to switch Hadoop version|
+|[#275](https://github.com/oap-project/native-sql-engine/pull/275)|[NSE-274] Comment to trigger tpc-h RAM test|
+|[#271](https://github.com/oap-project/native-sql-engine/pull/271)|[NSE-196] clean up configs in unit tests|
+|[#258](https://github.com/oap-project/native-sql-engine/pull/258)|[NSE-257] fix different versions of slf4j-log4j12|
+|[#259](https://github.com/oap-project/native-sql-engine/pull/259)|[NSE-248] fix arrow dependency order|
+|[#249](https://github.com/oap-project/native-sql-engine/pull/249)|[NSE-241] fix hashagg result length|
+|[#255](https://github.com/oap-project/native-sql-engine/pull/255)|[NSE-256] do not run ram report test on each PR|
+
+
+### SQL DS Cache
+
+#### Features
+|||
+|:---|:---|
+|[#118](https://github.com/oap-project/sql-ds-cache/issues/118)|port to Spark 3.1.1|
+
+#### Bugs Fixed
+|||
+|:---|:---|
+|[#121](https://github.com/oap-project/sql-ds-cache/issues/121)|OAP Index creation stuck issue|
+
+#### PRs
+|||
+|:---|:---|
+|[#132](https://github.com/oap-project/sql-ds-cache/pull/132)|Fix SampleBasedStatisticsSuite UnitTest case|
+|[#122](https://github.com/oap-project/sql-ds-cache/pull/122)|[ sql-ds-cache-121] Fix Index stuck issues|
+|[#119](https://github.com/oap-project/sql-ds-cache/pull/119)|[SQL-DS-CACHE-118][POAE7-1130] port sql-ds-cache to Spark3.1.1|
+
+
+### OAP MLlib
+
+#### Features
+|||
+|:---|:---|
+|[#26](https://github.com/oap-project/oap-mllib/issues/26)|[PIP] Support Spark 3.0.1 / 3.0.2 and upcoming 3.1.1|
+
+#### PRs
+|||
+|:---|:---|
+|[#39](https://github.com/oap-project/oap-mllib/pull/39)|[ML-26] Build for different spark version by -Pprofile|
+
+
+### PMEM Spill
+
+#### Features
+|||
+|:---|:---|
+|[#34](https://github.com/oap-project/pmem-spill/issues/34)|Support vanilla  spark 3.1.1|
+
+#### PRs
+|||
+|:---|:---|
+|[#41](https://github.com/oap-project/pmem-spill/pull/41)|[PMEM-SPILL-34][POAE7-1119]Port RDD cache to Spark 3.1.1 as separate module|
+
+
+### PMEM Common
+
+#### Features
+|||
+|:---|:---|
+|[#10](https://github.com/oap-project/pmem-common/issues/10)|add -mclflushopt flag to enable clflushopt for gcc|
+|[#8](https://github.com/oap-project/pmem-common/issues/8)|use clflushopt instead of clflush |
+
+#### PRs
+|||
+|:---|:---|
+|[#11](https://github.com/oap-project/pmem-common/pull/11)|[PMEM-COMMON-10][POAE7-1010]Add -mclflushopt flag to enable clflushop…|
+|[#9](https://github.com/oap-project/pmem-common/pull/9)|[PMEM-COMMON-8][POAE7-896]use clflush optimize version for clflush|
+
+
+### PMEM Shuffle
+
+#### Features
+|||
+|:---|:---|
+|[#15](https://github.com/oap-project/pmem-shuffle/issues/15)|Doesn't work with Spark3.1.1|
+
+#### PRs
+|||
+|:---|:---|
+|[#16](https://github.com/oap-project/pmem-shuffle/pull/16)|[pmem-shuffle-15] Make pmem-shuffle support Spark3.1.1|
+
+
+### Remote Shuffle
+
+#### Features
+|||
+|:---|:---|
+|[#18](https://github.com/oap-project/remote-shuffle/issues/18)|upgrade to Spark-3.1.1|
+|[#11](https://github.com/oap-project/remote-shuffle/issues/11)|Support DAOS Object Async API|
+
+#### PRs
+|||
+|:---|:---|
+|[#19](https://github.com/oap-project/remote-shuffle/pull/19)|[REMOTE-SHUFFLE-18] upgrade to Spark-3.1.1|
+|[#14](https://github.com/oap-project/remote-shuffle/pull/14)|[REMOTE-SHUFFLE-11] Support DAOS Object Async API|
+
+
 
 ## Release 1.1.0
-* [Native SQL Engine](#native-sql-engine)
-* [SQL DS Cache](#sql-ds-cache)
-* [OAP MLlib](#oap-mllib)
-* [PMEM Spill](#pmem-spill)
-* [PMEM Shuffle](#pmem-shuffle)
-* [Remote Shuffle](#remote-shuffle)
 
 ### Native SQL Engine
 
@@ -264,7 +402,7 @@ Generated on 2021-04-29
 |[#6](https://github.com/oap-project/pmem-shuffle/pull/6)|[PMEM-SHUFFLE-7] enable fsdax mode in pmem-shuffle|
 
 
-### Remote-Shuffle
+### Remote Shuffle
 
 #### Features
 |||

README.md

@@ -93,7 +93,7 @@ Master node can be co-located with one of the Hadoop data nodes.
 - Please refer to section 4.2 for configurations
 **Software:**
 -   Hadoop 2.7
--   Spark 3.0.0
+-   Spark 3.1.1
 -   Fedora 29 with ww08.2019 BKC
 
 ### 2.2. Recommended RDMA NIC
@@ -476,7 +476,7 @@ mvn install -DskipTests
 --------------------------------------------------------
 
 PMem Shuffle for spark shuffle is designed as a plugin to Spark.
-Currently the plugin supports Spark 3.0.0 and works well on various
+Currently the plugin supports Spark 3.1.1 and works well on various
 Network fabrics, including Socket, RDMA and Omni-Path. There are several
 configurations files needs to be modified in order to run PMem Shuffle. 
 

docs/OAP-Developer-Guide.md

@@ -3,13 +3,13 @@
 This document contains the instructions & scripts on installing necessary dependencies and building OAP modules. 
 You can get more detailed information from OAP each module below.
 
-* [SQL Index and Data Source Cache](https://github.com/oap-project/sql-ds-cache/blob/v1.1.0-spark-3.0.0/docs/Developer-Guide.md)
-* [PMem Common](https://github.com/oap-project/pmem-common/tree/v1.1.0-spark-3.0.0)
-* [PMem Spill](https://github.com/oap-project/pmem-spill/tree/v1.1.0-spark-3.0.0)
-* [PMem Shuffle](https://github.com/oap-project/pmem-shuffle/tree/v1.1.0-spark-3.0.0#5-install-dependencies-for-pmem-shuffle)
-* [Remote Shuffle](https://github.com/oap-project/remote-shuffle/tree/v1.1.0-spark-3.0.0)
-* [OAP MLlib](https://github.com/oap-project/oap-mllib/tree/v1.1.0-spark-3.0.0)
-* [Native SQL Engine](https://github.com/oap-project/native-sql-engine/tree/v1.1.0-spark-3.0.0)
+* [SQL Index and Data Source Cache](https://github.com/oap-project/sql-ds-cache/blob/v1.1.1-spark-3.1.1/docs/Developer-Guide.md)
+* [PMem Common](https://github.com/oap-project/pmem-common/tree/v1.1.1-spark-3.1.1)
+* [PMem Spill](https://github.com/oap-project/pmem-spill/tree/v1.1.1-spark-3.1.1)
+* [PMem Shuffle](https://github.com/oap-project/pmem-shuffle/tree/v1.1.1-spark-3.1.1#5-install-dependencies-for-pmem-shuffle)
+* [Remote Shuffle](https://github.com/oap-project/remote-shuffle/tree/v1.1.1-spark-3.1.1)
+* [OAP MLlib](https://github.com/oap-project/oap-mllib/tree/v1.1.1-spark-3.1.1)
+* [Native SQL Engine](https://github.com/oap-project/native-sql-engine/tree/v1.1.1-spark-3.1.1)
 
 ## Building OAP
 
@@ -22,18 +22,18 @@ We provide scripts to help automatically install dependencies required, please c
 # cd oap-tools
 # sh dev/install-compile-time-dependencies.sh
 ```
-*Note*: oap-tools tag version `v1.1.0-spark-3.0.0` corresponds to  all OAP modules' tag version `v1.1.0-spark-3.0.0`.
+*Note*: oap-tools tag version `v1.1.1-spark-3.1.1` corresponds to  all OAP modules' tag version `v1.1.1-spark-3.1.1`.
 
 Then the dependencies below will be installed:
 
-* [Cmake](https://help.directadmin.com/item.php?id=494)
+* [Cmake](https://cmake.org/install/)
 * [GCC > 7](https://gcc.gnu.org/wiki/InstallingGCC)
 * [Memkind](https://github.com/memkind/memkind/tree/v1.10.1)
 * [Vmemcache](https://github.com/pmem/vmemcache)
 * [HPNL](https://github.com/Intel-bigdata/HPNL)
 * [PMDK](https://github.com/pmem/pmdk)  
 * [OneAPI](https://software.intel.com/content/www/us/en/develop/tools/oneapi.html)
-* [Arrow](https://github.com/oap-project/arrow/tree/arrow-3.0.0-oap-1.1)
+* [Arrow](https://github.com/oap-project/arrow/tree/arrow-4.0.0-oap-1.1.1)
 * [LLVM](https://llvm.org/) 
 
 Run the following command to learn more.

docs/OAP-Installation-Guide.md

@@ -29,17 +29,17 @@ Create a Conda environment and install OAP Conda package.
 ```bash
 $ conda create -n oapenv -y python=3.7
 $ conda activate oapenv
-$ conda install -c conda-forge -c intel -y oap=1.1.0
+$ conda install -c conda-forge -c intel -y oap=1.1.1
 ```
 
 Once finished steps above, you have completed OAP dependencies installation and OAP building, and will find built OAP jars under `$HOME/miniconda2/envs/oapenv/oap_jars`
 
 Dependencies below are required by OAP and all of them are included in OAP Conda package, they will be automatically installed in your cluster when you Conda install OAP. Ensure you have activated environment which you created in the previous steps.
 
-- [Arrow](https://github.com/Intel-bigdata/arrow)
+- [Arrow](https://github.com/oap-project/arrow/tree/arrow-4.0.0-oap-1.1.1)
 - [Plasma](http://arrow.apache.org/blog/2017/08/08/plasma-in-memory-object-store/)
-- [Memkind](https://anaconda.org/intel/memkind)
-- [Vmemcache](https://anaconda.org/intel/vmemcache)
+- [Memkind](https://github.com/memkind/memkind/tree/v1.10.1)
+- [Vmemcache](https://github.com/pmem/vmemcache.git)
 - [HPNL](https://anaconda.org/intel/hpnl)
 - [PMDK](https://github.com/pmem/pmdk)  
 - [OneAPI](https://software.intel.com/content/www/us/en/develop/tools/oneapi.html)

docs/User-Guide.md

@@ -84,7 +84,7 @@ Master node can be co-located with one of the Hadoop data nodes.
 - Please refer to section 4.2 for configurations
 **Software:**
 -   Hadoop 2.7
--   Spark 3.0.0
+-   Spark 3.1.1
 -   Fedora 29 with ww08.2019 BKC
 
 ### 2.2. Recommended RDMA NIC
@@ -467,7 +467,7 @@ mvn install -DskipTests
 --------------------------------------------------------
 
 PMem Shuffle for spark shuffle is designed as a plugin to Spark.
-Currently the plugin supports Spark 3.0.0 and works well on various
+Currently the plugin supports Spark 3.1.1 and works well on various
 Network fabrics, including Socket, RDMA and Omni-Path. There are several
 configurations files needs to be modified in order to run PMem Shuffle. 
 

docs/index.md

@@ -84,7 +84,7 @@ Master node can be co-located with one of the Hadoop data nodes.
 - Please refer to section 4.2 for configurations
 **Software:**
 -   Hadoop 2.7
--   Spark 3.0.0
+-   Spark 3.1.1
 -   Fedora 29 with ww08.2019 BKC
 
 ### 2.2. Recommended RDMA NIC
@@ -467,7 +467,7 @@ mvn install -DskipTests
 --------------------------------------------------------
 
 PMem Shuffle for spark shuffle is designed as a plugin to Spark.
-Currently the plugin supports Spark 3.0.0 and works well on various
+Currently the plugin supports Spark 3.1.1 and works well on various
 Network fabrics, including Socket, RDMA and Omni-Path. There are several
 configurations files needs to be modified in order to run PMem Shuffle. 
 

mkdocs.yml

@@ -19,5 +19,5 @@ theme: readthedocs
 plugins:
   - search
   - mkdocs-versioning:
-      version: master
+      version: 1.1.1
       exclude_from_nav: ["image", "js", "css", "fonts", "img"]