v0.14.0: Kotlin 2.0 and many stability improvements

[Jolanrensen]

This release can mostly be described as a quality-of-life release. While there are not many new groundbreaking features at the moment, almost every part of the library has had some improvement. See the full list of changes below, but to highlight a few:

• We now officially support Kotlin 2.0+. The library is built with 2.0.20 now, so it will work with KSP 2.0.20 too.
• We've continued our work on the DataFrame Kotlin Compiler Plugin. While it is still experimental, it introduces an exciting new approach to working with your data in a zero-boilerplate, type safe way leveraging the amazing power the Kotlin 2.0 compiler gives us. See this demo project to experiment with it yourself.
• See this notebook for some of the small yet exciting features of the 0.14 release!

Features

• Compiler plugin by @koperagen in Compiler plugin #729
• added toDataFrame for float- and double iterables by @Jolanrensen in added toDataFrame for float- and double iterables #631
• Allow any ArrowReader implementation to be use for reading Arrow data Enable use of other ArrowReader implementations #627 by @fb64 in Allow any ArrowReader implementation to be use for reading Arrow data #627 #628
• add random parameter to shuffle by @koperagen in add random parameter to shuffle #643
• apply ksp to multiplatform configs in multiplatform modules by @mgroth0 in apply ksp to multiplatform configs in multiplatform modules #647
• Add separator parameter to DataFrame.flatten by @zaleslaw in Add separator parameter to DataFrame.flatten #667
• POJO toDataFrame support (and array improvements) by @Jolanrensen in POJO toDataFrame support (and array improvements) #650
• Add JDBC credentials extraction from env variables and improve exception handling by @zaleslaw in Add JDBC credentials extraction from env variables and improve exception handling #692
• Added MS SQL support for the dataframe-jdbc module by @zaleslaw in Added MS SQL support for the dataframe-jdbc module #689
• Update SQL all table/schemas reading functions to return maps with table names by @zaleslaw in Update SQL all table/schemas reading functions to return maps with table names #718
• Add a support for H2 modes by @zaleslaw in Add a support for H2 modes #720
• Add delimiter parameter to readDelimStr by @koperagen in Add delimiter parameter to readDelimStr #743
• Add an option to read Excel cell values as a String regardless of their content type by @koperagen in Add an option to read Excel cell values as a String regardless of their content type #745
• Add castTo to help working with implicitly generated schemas in notebooks and plugin by @koperagen in Add castTo to help working with implicitly generated schemas in notebooks and plugin #747
• Replace Klaxon with kotlinx-serialization by @devcrocod in Replace Klaxon with kotlinx-serialization #603
• Add df.convertTo(schemaFrom) overload by @koperagen in Add df.convertTo(schemaFrom) overload #764
• Add Convert.asFrame function by @koperagen in Add Convert.asFrame function #781
• Add extension functions for the ResultSet by @zaleslaw in Add extension functions for the ResultSet #772

Work on the compiler plugin

• then operation in pivot column selection DSL inside aggregate by @koperagen in then operation in pivot column selection DSL inside aggregate #617
• Compiler plugin fixes by @koperagen in Compiler plugin fixes #740
• Compiler plugin update by @koperagen in Compiler plugin update #755
• Adding utils to help ensure that compile time schema ~ runtime schema by @koperagen in Adding utils to help ensure that compile time schema ~ runtime schema #767
• Improve codegen for stdlib <-> df interop workflow by @koperagen in Improve codegen for stdlib <-> df interop workflow #763
• Add initial support for CS DSL in the compiler plugin by @koperagen in Add initial support for CS DSL in the compiler plugin #783
• Refactor toDataFrame implementation in compiler plugin by @koperagen in Refactor toDataFrame implementation in compiler plugin #782
• [Compiler plugin] Avoid throwing debugging exceptions in user projects because of false positives by @koperagen in [Compiler plugin] Avoid throwing debugging exceptions in user projects because of false positives #788
• [Compiler plugin] silently abort interpretation in case of invariant errors by @koperagen in [Compiler plugin] silently abort interpretation in case of invariant errors #812
• [Compiler plugin ] Support ColumnName annotation in extension properties codegen by @koperagen in [Compiler plugin ] Support ColumnName annotation in extension properties codegen  #818
• Update compiler plugin by @koperagen in Update compiler plugin #832

Fixes

• Added "debug mode" to catch type mismatches in columns during testing only by @Jolanrensen in ☂ type: KType in DataColumnImpl mismatches actual values sometimes #713
• Removing dangerous exception in ConvenienceSchemaGeneratorPlugin by @Jolanrensen in Removing dangerous exception in ConvenienceSchemaGeneratorPlugin #655
• add gradle wrapper validation workflow by @sullis in add gradle wrapper validation workflow #648
• Fix Jupyter integration conflicts with variable type converters from other integrations. #640: Jupyter integration conflicts with variable type converters from other integrations by @ark-1 in Fix #640: Jupyter integration conflicts with variable type converters from other integrations #641
• KDoc fixes for IntelliJ 2024.X by @Jolanrensen in KDoc fixes for IntelliJ 2024.X #613
• Fix some things that break in the build with K2 enabled by @koperagen in Fix some things that break in the build with K2 enabled #665
• Add nullability inference support to dataframe-jdbc by @zaleslaw in Add nullability inference support to dataframe-jdbc #672
• Serialize BufferedImages as base64 by @ermolenkodev in Serialize BufferedImages as base64 #694
• Downsize test images to speed up unit tests execution by @ermolenkodev in Downsize test images to speed up unit tests execution #709
• KTNB-693 Send the full dataframe schema as metadata by @cmelchior in KTNB-693 Send the full dataframe schema as metadata #706
• K2 preparation by @Jolanrensen in K2 preparation #708
• File and URL names are used in JDK / suggested in IDE auto-import. Rename our classes to avoid conflict by @koperagen in File and URL names are used in JDK / suggested in IDE auto-import. Rename our classes to avoid conflict #725
• isComparable() fix for double describe() by @Jolanrensen in isComparable() fix for double describe() #726
• Fixes std() not inferring column types by @Jolanrensen in Fixes std() not inferring column types #728
• Pivot fix by @Jolanrensen in Pivot fix #735
• Enforce the project to be built with Java 11 by @Jolanrensen in Enforce the project to be built with Java 11 #736
• Rename private properties with names matching library classes by @koperagen in Rename private properties with names matching library classes #737
• Add a test for valueCounts by @koperagen in Add a test for valueCounts #756
• Revert "Add a test for valueCounts" by @koperagen in Revert "Add a test for valueCounts" #757
• Revert filter condition, fixes broken logic by @koperagen in Revert filter condition, fixes broken logic #759
• Add a test for valueCounts by @koperagen in Add a test for valueCounts #758
• Add a notice about external urls by @koperagen in Add a notice about external urls #774
• Make sortBy(ColumnReference) accept pathOf without extra cast by @koperagen in Make sortBy(ColumnReference) accept pathOf without extra cast #779
• Update MarkersExtractor.kt by @GeorgCantor in Update MarkersExtractor.kt #791
• Build config and Debug mode by @Jolanrensen in Build config and Debug mode #796
• This fixes Nothing by @Jolanrensen in This fixes Nothing #795
• Small convertTo fix by @Jolanrensen in Small convertTo fix #800
• Small implode fix by @Jolanrensen in Small implode fix #801
• GroupBy aggregate fix by @Jolanrensen in GroupBy aggregate fix #803
• Support for the serialization of DataframeConvertible values in ValueColumns has been added to enhance visualization in the KTNB plugin. by @ermolenkodev in Support for the serialization of DataframeConvertible values in ValueColumns has been added to enhance visualization in the KTNB plugin. #823
• Hide properties of intermediate objects and remove data class attributes by @koperagen in Hide properties of intermediate objects and remove data class attributes #828
• Fixes reading CSV files with BOM characters by @Jolanrensen in Fixes reading CSV files with BOM characters #831
• Improve dataframe sorting in KTNB UI by handling non-comparable columns by @ermolenkodev in Improve dataframe sorting in KTNB UI by handling non-comparable columns #836
• Empty csv fix by @Jolanrensen in Empty csv fix #835
• Added test suite for PostgreSQL local tests with covering case with URLs by @zaleslaw in Added test suite for PostgreSQL local tests with covering case with URLs #798
• Enforcing kotlinx datetime bias by @Jolanrensen in Enforcing kotlinx datetime bias #843
• allow applying ksp to custom configs by @mgroth0 in allow applying ksp to custom configs #842
• Fix duplicated jvm signature by @koperagen in Fix duplicated jvm signature #868
• Fixed flaky jsonWrite test by @Jolanrensen in Fixed flaky jsonWrite test #873
• Rerunning notebooks for 0.14.0-RC1 by @Jolanrensen in Rerunning notebooks for 0.14.0-RC1 #870
• Remove samplesTest task from kover report so "build" don't trigger it by @koperagen in Remove samplesTest task from kover report so "build" don't trigger it #865
• Improved SQL<->JDBC mapping by @zaleslaw in Improved SQL<->JDBC mapping #855
• Add Language annotations for better IDE experience by @koperagen in Add Language annotations for better IDE experience #861

Docs and Examples

• Update docs and readme for 0.13.1 by @Jolanrensen in Update docs and readme for 0.13.1 #629
• Fixed a documentation for SQL database connection in Kotlin Notebooks by @zaleslaw in Fixed a documentation for SQL database connection in Kotlin Notebooks #639
• Updated parse documentation by @Jolanrensen in Updated parse documentation #652
• Updated split documentation by @Jolanrensen in Updated split documentation #653
• Updated mentions of DataFrame to represent objects by @zaleslaw in Updated mentions of DataFrame to represent objects #664
• Remove incomplete documentation topics by @zaleslaw in Remove incomplete documentation topics #681
• Update comments and doc to provide more details about ResultSet by @zaleslaw in Update comments and doc to provide more details about ResultSet #682
• Introduce OtherSamples.kt to generate tables for writerside docs outside Korro by @koperagen in Introduce OtherSamples.kt to generate tables for writerside docs outside Korro #695
• Enable sitemap.xml by @Jolanrensen in Enable sitemap.xml #698
• Update README.md by @Jolanrensen in Update README.md #714
• Column Selector Grammar in docs by @Jolanrensen in Column Selector Grammar in docs #619
• Column selector docs function descriptions by @Jolanrensen in Column selector docs function descriptions #711
• KDocs GH actions by @Jolanrensen in KDocs GH actions #731
• Korro gh actions by @Jolanrensen in Korro gh actions #824
• [KDocs] auto preprocess only when publishing by @Jolanrensen in [KDocs] auto preprocess only when publishing #749
• google analytics script by @azhebel in google analytics script #786
• Highlight work in progress in README by @koperagen in Highlight work in progress in README #789
• Small update to the linter information for the documentation website by @Jolanrensen in Small update to the linter information for the documentation website #810
• Adding KDoc preprocessing guide by @Jolanrensen in Adding KDoc preprocessing guide #806
• Changed documentation regarding extension properties api by @Jolanrensen in Changed documentation regarding extension properties api #651
• Update links to demos by @koperagen in Update links to demos #833
• Explain the use-case for toDataFrame(columnName) in docs by @koperagen in Explain the use-case for toDataFrame(columnName) in docs #837
• Add documentation for Kotlin Notebook Plugin usage and troubleshooting by @ermolenkodev in Add documentation for Kotlin Notebook Plugin usage and troubleshooting #839

Version Updates

• Dev version bump to 0.14.0 & updating deprecations by @Jolanrensen in Dev version bump to 0.14.0 & updating deprecations #636
• update ksp to 1.9.23-1.0.20 by @koperagen in update ksp to 1.9.23-1.0.20 #654
• set docPreprocessor to 0.3.6 by @Jolanrensen in set docPreprocessor to 0.3.6 #716
• Update Kotlin version for compiler plugin by @koperagen in Update Kotlin version for compiler plugin #744
• Update kotlinDatetime version by @zaleslaw in Update kotlinDatetime version #753
• Replacing the linter by @Jolanrensen in Replacing the linter #754
• Update to Kotlin 2.0 by @Jolanrensen in Update to Kotlin 2.0 #792
• Update database URLs and dependencies by @zaleslaw in Update database URLs and dependencies #802
• Gradle bump to 8.10 and housekeeping by @Jolanrensen in Gradle bump to 8.10 and housekeeping #846
• 0.14 version bumps by @Jolanrensen in 0.14 version bumps #848

Known Issues

• isOpenApiStr logger leaks to Gradle in isOpenApiStr logger leaks to Gradle  #785
• Compiler plugin is still experimental in Kotlin DataFrame compiler plugin #704
• Jupyter importDataSchema() does not generate column accessors in Jupyter importDataSchema() does not generate column accessors #696
• Describe breaks on Number column (and other statistics inconsistencies) in ☂️ Describe breaks on Number column (and other statistics inconsistencies) #558
• CSV reading can be slow and needs some rework in ☂ CSV rework #827
• KDocs are lacking in ☂ Provide KDocs for public API: Continued #811
• Many more, see issues and feel free to help us :)

New Contributors

• @mgroth0 made their first contribution in apply ksp to multiplatform configs in multiplatform modules #647
• @sullis made their first contribution in add gradle wrapper validation workflow #648
• @ark-1 made their first contribution in Fix #640: Jupyter integration conflicts with variable type converters from other integrations #641
• @azhebel made their first contribution in google analytics script #786
• @GeorgCantor made their first contribution in Update MarkersExtractor.kt #791

Full Changelog: v0.13.1...v0.14.0

---

This discussion was created from the release v0.14.0: Kotlin 2.0 and many stability improvements.