v0.2.0

What's Changed

• Restructure the repository to distinguish/separate runtime libraries by @daw3rd in #140
• Move transform code into ray subdirectory - towards splitting transform runtimes. by @daw3rd in #143
• restore lost transforms/universal/noop/ray content by @daw3rd in #144
• New Readme file created for memory and endurance tests by @shahrokhDaijavad in #145
• LAB to Kit by @shahrokhDaijavad in #147
• Update ray/README.md by @eltociear in #148
• kfp multi jobs by @blublinsky in #142
• small fix in the init file by @blublinsky in #150
• rename make targets to be ray-specific by @daw3rd in #146
• Naming, docs and fix for recent binary file processing changes by @daw3rd in #153
• bug fixes by @blublinsky in #155
• Binary by @blublinsky in #141
• update kfp image version by @roytman in #159
• Update README.md for Broken links by @shahrokhDaijavad in #160
• adding multi_launcher tests by @blublinsky in #164
• Enable kfp in GH action for testing workflows by @revit13 in #149
• Fix paths in examples scripts. by @revit13 in #180
• Fail workflow if input size is empty. by @revit13 in #181
• library versions update by @blublinsky in #186
• Handle empty input parameter. by @Mohammad-nassar10 in #158
• Moving kfp workflows transform_workflows to transform directory. by @revit13 in #151
• update KFP docs by @roytman in #189
• Dev2 by @roytman in #191
• Modified ingress config (#130) by @D-Sai-Venkatesh in #156
• fixed flush in transform_file_processor.py by @blublinsky in #190
• added PLI related language extensions by @jitendrasinghibm in #177
• more fixes to the transform file processor by @blublinsky in #195
• Spark runtime by @cmadam in #183
• Fix white check marks in top readme. by @daw3rd in #199
• Minor fixes to kind/README.md. by @revit13 in #208
• Add utils functions to kfp support lib. by @Mohammad-nassar10 in #209
• Add Super pipeline for code transforms. by @revit13 in #172
• Tutorial README files fixes by @shahrokhDaijavad in #214
• Added copyright to the Spark files by @cmadam in #207
• Fix dependabot alert on tqdm in fdedup. by @daw3rd in #218
• Update filter_local.py by @shahrokhDaijavad in #217
• Split data-processing-lib/ray into python and ray. by @daw3rd in #213
• Enhanced the default 'make clean' rule to delete python leftovers and… by @daw3rd in #219
• small fixes by @roytman in #220
• Fixes after testing. by @revit13 in #223
• Change kfp_v1_workflow_support. by @revit13 in #227
• Split noop ray transform into ray and python runtimes. by @daw3rd in #221
• Fix tqdm security issue in ededup by @daw3rd in #224
• Tansform project conventions doc and makefile fix… by @daw3rd in #229
• Fixes after testing. by @revit13 in #232
• Runtime reorg by @daw3rd in #230
• Auto generate kfp pipelines. by @Mohammad-nassar10 in #193
• ingest to parquet rewrite by @blublinsky in #231
• KFPv2 support step 1 by @roytman in #226
• Rename of ingest_2_parquet file. by @daw3rd in #241
• Make all top level make targets pass w/o error by @daw3rd in #247
• Readme, pyproject metadata and makefile fixes in noop and filter. by @daw3rd in #240
• add retries counter to data processing by @blublinsky in #245
• Initial split of tokenization transform into ray and python by @daw3rd in #243
• add language identification transform module by @dtsuzuku-ibm in #256
• small changes to get ready for pdf by @blublinsky in #261
• Combine the common KFP support code in a shared library by @roytman in #253
• Fix tasks tags in kfp workflows. by @revit13 in #236
• Adjust ingest_2_parquet workflow. by @revit13 in #248
• Repo Root README and CONTRIBUTING clarifications by @shahrokhDaijavad in #264
• add build-language job to build-images workflow by @dtsuzuku-ibm in #268
• remove the artifactory settings by @roytman in #280
• update docs for KFPv2 by @roytman in #279
• Enhancing some README files by @shahrokhDaijavad in #278
• extended logging to print % and number processed files by @blublinsky in #272
• Updated transform readmes to reference correct runtime when describing cli params. by @daw3rd in #284
• Update advanced-transform-tutorial.md by @shahrokhDaijavad in #287
• add test-language job by @dtsuzuku-ibm in #286
• Change execution log file name. by @Mohammad-nassar10 in #251
• Update tests for KFP v2. by @revit13 in #255
• remove entire pipeline timeouts by @roytman in #270
• Randomly choose workflow to run in GH action. by @revit13 in #281
• Change the docker user as root by @takuyagt in #291
• Initial version of profiler by @blublinsky in #269
• Minimum explanation for VS Code by @shahrokhDaijavad in #290
• move logger to ensure Ray logging is correct by @blublinsky in #301
• Use dpk user for malware python image by @takuyagt in #304
• Move hack dirs to scripts dir by @revit13 in #295
• Fix issue #274 for venv corruption via make -n venv by @daw3rd in #302
• Installation of minio added to the transform README files by @shahrokhDaijavad in #303
• Minor fixes to profiler workflow by @revit13 in #308
• Ray version update by @blublinsky in #305
• update notebook by @shivdeep-singh-ibm in #310
• Split code quality, malware and proglang select transforms into python and ray. by @daw3rd in #288
• renaming of ingest_2_parquet by @blublinsky in #316
• move transform exceptions doc out of ray runtime to overview by @daw3rd in #319
• Inputcode2parquet rename by @daw3rd in #320
• fault tolerance by @blublinsky in #321
• Makefile rules updates by @revit13 in #323
• updated pyarrow version by @blublinsky in #325
• Fix make run-cli-sample for code2parquet by @daw3rd in #328
• Updated generate (simple pipeline) pipeline by @D-Sai-Venkatesh in #311
• Some new thoughts on cutting a release, especially scripts/release.sh by @daw3rd in #309
• Corrected Readme to update file path, added more detail signoff steps by @santoshborse in #330
• improve doc on transform design/expectations by @daw3rd in #331
• fix a typo by @roytman in #333
• Improvements to code2parquet transform by @daw3rd in #329
• implementing missing pyproject on transforms by @blublinsky in #327
• add new params to lang_id to store the results of language identification by @dtsuzuku-ibm in #322
• change content column name used in wf script by @dtsuzuku-ibm in #340
• small bug fixes by @blublinsky in #342
• fix typos by @roytman in #341
• update top readme table of transforms by @daw3rd in #344
• update the kfp release process by @roytman in #338
• remove globals in ray transforms that should insteads be references to the python transform globals by @daw3rd in #336
• Update K8s cluster deployment by @revit13 in #334
• Fix Instruction to create NOOP transformer by @santoshborse in #346
• Add workflow-build target by @revit13 in #348
• Update readme to point to new code2parquet transform by @Bytes-Explorer in #349
• add new release docs and stop publishing in script for 0.2.0 by @daw3rd in #337
• Add ingest2parquet step to superpipeline. by @Mohammad-nassar10 in #273
• code2parquet fixes on domain/snapshot and document_id by @daw3rd in #347
• add kfp_ray README files by @roytman in #351
• Changes in code2parquet, ingest2parquet, and advance tutorial readmes. by @daw3rd in #352
• disable debug flag, by default, in release-branch.sh by @daw3rd in #353
• Update release-branch script to not verify commits to avoid failures by @daw3rd in #354

New Contributors

• @eltociear made their first contribution in #148
• @Mohammad-nassar10 made their first contribution in #158
• @D-Sai-Venkatesh made their first contribution in #156
• @jitendrasinghibm made their first contribution in #177
• @cmadam made their first contribution in #183
• @dtsuzuku-ibm made their first contribution in #256
• @takuyagt made their first contribution in #291

Full Changelog: v0.1.0-dpk...v0.2.0