Merge pull request #900 from Kotlin/readmes

READMEs

.github/README.md

@@ -0,0 +1,33 @@
+## GitHub Actions
+
+While publishing and testing of the library is handled by [JetBrains TeamCity](https://www.jetbrains.com/teamcity/),
+there are some CI operations that are handled by GitHub actions instead.
+
+### Publishing Docs
+
+The building of the documentation website in [docs](../docs), and the publishing of it along
+with the search-indices is handled by the [Build Docs GH Action](./workflows/main.yml). Careful: This action replaces the entire contents
+of the documentation website.
+
+### Gradle Wrapper Validation
+
+We run the [Gradle Wrapper Validation action](https://github.com/gradle/wrapper-validation-action)
+using a [GitHub Action](./workflows/gradle-wrapper-validation.yml) as well.
+This action validates the checksums of all Gradle Wrapper JAR files present in the repository and
+fails if any unknown Gradle Wrapper JAR files are found.
+
+### Auto-commit generated code
+
+Anytime the source code changes on [master](https://github.com/Kotlin/dataframe/tree/master),
+this [GitHub Action](./workflows/generated-sources-master.yml) makes sure
+[`processKDocsMain`](../KDOC_PREPROCESSING.md),
+and `korro` are run. If there have been any changes in either [core/generated-sources](../core/generated-sources) or
+[docs/StardustDocs/snippets](../docs/StardustDocs/snippets), these are auto-committed to the branch, to keep
+it up to date.
+
+### Show generated code in PR
+
+To make sure no unexpected code is auto-committed to [master](https://github.com/Kotlin/dataframe/tree/master),
+this [GitHub Action](./workflows/generated-sources.yml) runs the same code-generating tasks but on a separate branch.
+If there are changes, it will leave a message in the PR, informing you about the changes that will be done to the master
+branch if this PR were merged.

core/README.md

@@ -0,0 +1,49 @@
+## :core
+
+This is the core of the library, published as the `dataframe-core` package.
+It contains the DataFrame API and its implementation, as well as plenty of JUnit tests.
+
+I/O operations are split off into other modules, like [:dataframe-excel](../dataframe-excel),
+or [:dataframe-jdbc](../dataframe-jdbc), however, this is has not happened yet for all operations
+(see [Issue #100](https://github.com/Kotlin/dataframe/issues/100)).
+
+At the moment, these integrations are still part of the `:core` module:
+
+- csv/tsv
+- html
+- json
+- jupyter
+
+### Korro code samples
+
+Code samples for the documentation website reside
+in [core/.../test/.../samples/api](./src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) (for samples that depend solely on `dataframe-core`),
+and [tests/.../samples/api](../tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) (for samples can depend on other I/O modules)
+and they are copied over to Markdown files in [docs/StardustDocs/topics](../docs/StardustDocs/topics)
+by [Korro](https://github.com/devcrocod/korro).
+
+### Explainer dataframes
+
+Aside from code samples, `@TransformDataFrameExpressions` annotated test functions also generate sample
+dataframe HTML files that can be used as iFrames on the documentation website.
+They are tested, generated, and copied over to [docs/StardustDocs/snippets](../docs/StardustDocs/snippets) by
+our "explainer" [plugin callback proxy](./src/test/kotlin/org/jetbrains/kotlinx/dataframe/explainer),
+which hooks into [the TestBase class](./src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/TestBase.kt) and
+retrieves the intermediate DataFrame expressions thanks to our "explainer" compiler plugin
+[:plugins:expressions-converter](../plugins/expressions-converter).
+
+We can also generate "normal" DataFrame samples for the website. This can be done using the
+[OtherSamples class](./src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/OtherSamples.kt). Generated
+HTML files will be stored in [docs/StardustDocs/snippets/manual](../docs/StardustDocs/snippets/manual).
+
+### KDoc Preprocesor
+
+The code you're working on needs to be edited in [src](src), but the KDocs are processed by the
+[KDoc Preprocessor](https://github.com/Jolanrensen/docProcessorGradlePlugin) when the project is published (or the task
+is run manually). The generated sources with adjusted KDocs will be overwritten
+in [generated-sources](generated-sources).
+See the [KDoc Preprocessing Guide](../KDOC_PREPROCESSING.md) for more information.
+
+KDocs can also be exported to HTML, for them to be reused on the website.
+Elements annotated with `@ExportAsHtml` will have their generated content be copied over to
+[docs/StardustDocs/snippets/kdocs](../docs/StardustDocs/snippets/kdocs).

core/build.gradle.kts

@@ -14,13 +14,15 @@ plugins {
         alias(serialization)
         alias(jupyter.api)
         alias(korro)
-        alias(keywordGenerator)
         alias(kover)
         alias(ktlint)
         alias(docProcessor)
         alias(simpleGit)
         alias(buildconfig)
 
+        // generates keywords using the :generator module
+        alias(keywordGenerator)
+
         // dependence on our own plugin
         alias(dataframe)
 

dataframe-arrow/README.md

@@ -0,0 +1,8 @@
+## :dataframe-arrow
+
+This module, published as `dataframe-arrow`, contains all logic and tests for DataFrame to be able to work with 
+Apache Arrow.
+
+See [Read Apache Arrow formats](https://kotlin.github.io/dataframe/read.html#read-apache-arrow-formats) and
+[Writing to Apache Arrow formats](https://kotlin.github.io/dataframe/write.html#writing-to-apache-arrow-formats)
+for more information about how to use it.

dataframe-excel/README.md

@@ -0,0 +1,8 @@
+## :dataframe-excel
+
+This module, published as `dataframe-excel`, contains all logic and tests for DataFrame to be able to work with
+`xls` and `xlsx` files.
+
+See [Read from Excel](https://kotlin.github.io/dataframe/read.html#read-from-excel) and
+[Write to Excel spreadsheet](https://kotlin.github.io/dataframe/write.html#write-to-excel-spreadsheet)
+for more information about how to use it.

dataframe-jdbc/README.md

@@ -0,0 +1,7 @@
+## :dataframe-jdbc
+
+This module, published as `dataframe-jdbc`, contains all logic and tests for DataFrame to be able to work with
+JDBC data sources.
+
+See [Read from SQL databases](https://kotlin.github.io/dataframe/readsqldatabases.html) for more information
+about how to use it.

dataframe-openapi-generator/README.md

@@ -0,0 +1,14 @@
+## :dataframe-openapi-generator
+
+This module, published as `dataframe-openapi-generator` contains all logic and tests for DataFrame to be able to import
+OpenAPI specifications as auto-generated data schemas. This module is a sister module to
+[`dataframe-openapi`](../dataframe-openapi):
+
+- `dataframe-openapi-generator` is used as a dependency of the Gradle plugin and Jupyter plugin to be able to generate
+    data schemas from OpenAPI specifications. In the Gradle plugin it adds support for the `dataschemas {}` DSL and the
+    `@file:ImportDataSchema()` annotation. In Jupyter, it adds support for the `importDataSchema()` function.
+- `dataframe-openapi` must be used as a dependency of a user-project to be able to use the generated data schemas.
+
+See [Import OpenAPI Schemas in Gradle project](https://kotlin.github.io/dataframe/schemasimportopenapigradle.html) and
+[Import Data Schemas, e.g. from OpenAPI, in Jupyter](https://kotlin.github.io/dataframe/schemasimportopenapijupyter.html)
+for more information about how to use it.

dataframe-openapi/README.md

@@ -0,0 +1,14 @@
+## :dataframe-openapi
+
+This module, published as `dataframe-openapi` contains some functions to be able to use auto-generated
+data schemas from OpenAPI specifications. This module is a sister module to
+[`dataframe-openapi-generator`](../dataframe-openapi-generator):
+
+- `dataframe-openapi-generator` is used as a dependency of the Gradle plugin and Jupyter plugin to be able to generate
+  data schemas from OpenAPI specifications. In the Gradle plugin it adds support for the `dataschemas {}` DSL and the
+  `@file:ImportDataSchema()` annotation. In Jupyter, it adds support for the `importDataSchema()` function.
+- `dataframe-openapi` must be used as a dependency of a user-project to be able to use the generated data schemas.
+
+See [Import OpenAPI Schemas in Gradle project](https://kotlin.github.io/dataframe/schemasimportopenapigradle.html) and
+[Import Data Schemas, e.g. from OpenAPI, in Jupyter](https://kotlin.github.io/dataframe/schemasimportopenapijupyter.html)
+for more information about how to use it.

docs/README.md

@@ -0,0 +1,40 @@
+## The Documentation Website
+
+This folder holds the source code of our documentation website:
+[kotlin.github.io/dataframe](https://kotlin.github.io/dataframe).
+
+It's built using [Writerside](https://www.jetbrains.com/writerside/) and published
+by a [Github Action](../.github/workflows/main.yml).
+The file structure largely mirrors the default Writerside structure.
+For instance, if you want to add a new page to the website, this needs to be stored as an `.md` file in the
+[StardustDocs/topics](./StardustDocs/topics) folder,
+and included in the [StardustDocs/d.tree](./StardustDocs/d.tree) file.
+
+Images all README files can be stored in [docs/imgs](./imgs).
+
+### Explainer dataframes
+`@TransformDataFrameExpressions` annotated test functions generate sample
+dataframe HTML files that can be used as iFrames on the documentation website.
+They are tested, generated, and copied over to [docs/StardustDocs/snippets](./StardustDocs/snippets) by
+our "explainer" [plugin callback proxy](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/explainer),
+which hooks into [the TestBase class](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/TestBase.kt) and
+retrieves the intermediate DataFrame expressions thanks to
+our "explainer" compiler plugin [:plugins:expressions-converter](../plugins/expressions-converter).
+
+We can also generate "normal" DataFrame samples for the website. This can be done using the
+[OtherSamples class](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/OtherSamples.kt). Generated
+HTML files will be stored in [docs/StardustDocs/snippets/manual](./StardustDocs/snippets/manual).
+
+### KDoc Preprocessor
+KDocs can also be exported to HTML, for them to be reused on the website.
+Elements annotated with `@ExportAsHtml` will have their generated content be copied over to
+[docs/StardustDocs/snippets/kdocs](./StardustDocs/snippets/kdocs).
+
+### Korro code samples
+Code samples for the documentation website reside in [core/.../test/.../samples/api](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api)
+(for samples that depend solely on `dataframe-core`),
+and [tests/.../samples/api](../tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) (for samples can depend on other I/O modules)
+and they are copied over to Markdown files in [docs/StardustDocs/topics](./StardustDocs/topics)
+by [Korro](https://github.com/devcrocod/korro).
+
+

examples/README.md

@@ -44,3 +44,5 @@ ___
 ---
 * 0.14.0 Dev Feature Overview [notebook](notebooks/feature_overviews/0.14.0/new_features.ipynb)
   Overview of new features available in 0.14.0-dev
+
+These example notebooks are tested by the [:core module](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/jupyter).

generator/README.md

@@ -0,0 +1,14 @@
+## :generator
+
+This module holds a little Gradle plugin whose sole purpose is to provide
+[:core](../core) with the `generateKeywordsSrc` task.
+
+This task, generates three enum classes: `HardKeywords`, `ModifierKeywords`, and `SoftKeywords`.
+These enums together contain all restricted Kotlin keywords to be taken into account when generating our own
+code in Notebooks or any of our [plugins](../plugins). Words like "package", "fun", "suspend", etc...
+
+As the Kotlin language can change over time, this task ensures that any changes to the language
+will be reflected in our code generation.
+
+This module will likely be moved under [:plugins](../plugins):
+[Issue #899](https://github.com/Kotlin/dataframe/issues/899).

plugins/README.md

@@ -0,0 +1,28 @@
+## :plugins
+
+This folder holds all our Gradle- and Kotlin Compiler plugins:
+
+### [:plugins:dataframe-gradle-plugin](./dataframe-gradle-plugin)
+The Gradle plugin for DataFrame that can generate data schemas from a data sample using the simple
+task `dataframes {}`. It uses [:plugins:symbol-processor](./symbol-processor) to generate column accessors.
+
+### [:plugins:symbol-processor](./symbol-processor)
+The KSP plugin that can generate data schemas from a data sample using the `@file:ImportDataSchema` annotation.
+It is also used to generate column accessors for in the form of extension properties for 
+(both manually written- or generated) `@DataSchema` annotated classes/interfaces.
+
+### [:plugins:kotlin-dataframe](./kotlin-dataframe)
+The Kotlin 2.x Compiler plugin of DataFrame. A [work-in-progress](https://github.com/Kotlin/dataframe/issues/704)
+plugin for your Gradle project that can generate on-the-fly column accessors for the compiler and IDE even without
+having to provide data schemas!
+
+### [:plugins:expressions-converter](./expressions-converter)
+A small Kotlin Compiler plugin that provides intermediate expressions of DataFrame
+operation chains, used internally by [:core](../core) to generate "explainer dataframes" on the documentation website.
+
+### [generator](../generator) 
+A small Gradle plugin that is used internally to generate enums with restricted Kotlin keywords for the 
+[:core](../core) module.
+
+This module can probably be moved under [:plugins](../plugins):
+[Issue #899](https://github.com/Kotlin/dataframe/issues/899).

plugins/dataframe-gradle-plugin/README.md

@@ -0,0 +1,11 @@
+## :plugins:dataframe-gradle-plugin
+
+This module holds the Gradle plugin for DataFrame, published as "org.jetbrains.kotlinx.dataframe" on the
+[Gradle Plugin Portal](https://plugins.gradle.org/plugin/org.jetbrains.kotlin.plugin.dataframe).
+
+This plugin can let the user generate data schemas from a data sample using the simple
+Gradle task `dataframes {}`. It also provides an automatic dependency on [:plugins:symbol-processor](./symbol-processor)
+to generate column accessors and support the `@file:ImportDataSchema` notation.
+
+Read more about how to use the Gradle plugin at
+[Data Schemas in Gradle projects](https://kotlin.github.io/dataframe/schemasgradle.html).

plugins/expressions-converter/README.md

@@ -0,0 +1,12 @@
+## :plugins:expressions-converter
+
+This Kotlin Compiler plugin, used by [:core](../../core), can extract intermediate
+DataFrame expressions from `@TransformDataFrameExpressions` annotated functions.
+
+It is used to generate sample "explainer dataframe" HTML files that can be used as iFrames on the documentation website.
+
+Annotated functions in [core/.../test/.../samples/api](../../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api)
+are tested, generated, and copied over to [docs/StardustDocs/snippets](../../docs/StardustDocs/snippets) by
+our "explainer" [plugin callback proxy](../../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/explainer),
+which hooks into [the TestBase class](../../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/TestBase.kt) and
+retrieves the intermediate DataFrame expressions thanks to this module.

plugins/kotlin-dataframe/README.md

@@ -0,0 +1,10 @@
+## :plugins:kotlin-dataframe
+
+This module, published as "compiler-plugin-all", holds the Kotlin 2.x Compiler plugin of DataFrame.
+
+A [work-in-progress](https://github.com/Kotlin/dataframe/issues/704)
+plugin for your Kotlin project that can generate on-the-fly column accessors for the compiler and IDE even without
+having to provide data schemas!
+
+Details of how to use this compiler plugin for your own project will follow, but there is a
+[demo project](https://github.com/koperagen/df-plugin-demo) you can try already.

plugins/symbol-processor/README.md

@@ -0,0 +1,13 @@
+## :plugins:symbol-processor
+
+This module holds the KSP plugin, published as the library "symbol-processor-all" that can generate data schemas from 
+a data sample using the `@file:ImportDataSchema` annotation. This annotation can be used as an alternative to
+the `dataframes {}` syntax of the [Gradle plugin](../dataframe-gradle-plugin) that's declared closer to the source code.
+It is also used to detect (both manually written- or generated) `@DataSchema` annotated classes/interfaces to generate
+column accessors for in the form of extension properties.
+
+If you use the [Gradle plugin](../dataframe-gradle-plugin), this module and KSP are added as a dependency automatically.
+
+Read more about how to use this at
+[Data Schemas in Gradle projects](https://kotlin.github.io/dataframe/schemasgradle.html) and 
+[Extension Properties API](https://kotlin.github.io/dataframe/extensionpropertiesapi.html).

tests/README.md

@@ -0,0 +1,9 @@
+## :tests
+
+Code samples for the documentation website reside in [core/.../test/.../samples/api](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api)
+(for samples that depend solely on `dataframe-core`),
+and [tests/.../samples/api](../tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) (for samples can depend on other I/O modules)
+and they are copied over to Markdown files in [docs/StardustDocs/topics](../docs/StardustDocs/topics)
+by [Korro](https://github.com/devcrocod/korro).
+
+This module might be merged with [:core](../core): [Issue #898](https://github.com/Kotlin/dataframe/issues/898).