Plan for Renaming DataSet to Dataset

[noklam]

Background

This is a parent issue to document what've been done and what more to be done. This refactoring isn't simple as just changing the name of the class, and the success of this require coordination of dev and marketing. As this also affect various things (kedro.org, blogs for new Dataset contributions)

Done:

• Replace and deprecate DataSet use in class names #2500 is the main effort renaming everything from DataSet -> Dataset except AbstractDataSet
• Deprecate dataset errors with module __getattr__ #2673, Quote class names in DeprecationWarning messages #2706 are the effort enhancing the Deprecation warning show up but not everywhere.
• Remove DeprecatedClassMeta in favor of getattr #2724 attempts to improve it further with a modern Pythonic way to deprecate classes.

Ongoing:

• Replace "DataSet" with "Dataset" in Markdown files #2735 is updating the DataSet -> Dataset mentions in the markdown docs
• Deprecate abstract "DataSet" in favor of "Dataset" #2746 renames AbstractDataSet -> AbstractDataset

Things that are missing

• Update the documentation for .rst files in kedro

  kedro/docs/source/kedro.extras.datasets.rst

  Line 4 in 6e5e4e1

  .. rubric:: Description

• Do we need to update the URL on RTD? Do we need to setup the redirect because it will change the URL. https://kedro-org.slack.com/archives/C054U0LJLAC/p1687467469513519
  • https://docs.kedro.org/en/latest/kedro_datasets.api.APIDataSet.html -> https://docs.kedro.org/en/latest/kedro_datasets.api.APIDataset.html (It is case sensitive) @stichbury
• Do we need to update these extra install with Dataset as well?

  kedro/setup.py

  Lines 44 to 50 in 6e5e4e1

  	pandas_require = {
  	"pandas.CSVDataSet": [PANDAS],
  	"pandas.ExcelDataSet": [PANDAS, "openpyxl>=3.0.6, <4.0"],
  	"pandas.FeatherDataSet": [PANDAS],
  	"pandas.GBQTableDataSet": [PANDAS, "pandas-gbq>=0.12.0, <0.18.0"],
  	"pandas.GBQQueryDataSet": [PANDAS, "pandas-gbq>=0.12.0, <0.18.0"],
  	"pandas.HDFDataSet": [

• Deprecations warning for kedro-datasets (is this needed or we release the Dataset version as a breaking change?) I think we need to communicate this and have a better plan. See Replace "DataSet" with "Dataset" in Kedro-Datasets #2751.
• Starters
• Graphic in the Data Catalog section on the website

Questions:

• Do we need to finish all of this before release or we can find some midpoint?
• It is a bit awkward if we start showing Deprecation warnings in kedro but kedro-datasets are still using DataSet. Particular for users using kedro-datasets already, it may resulted in hybrid DataSet and Dataset in their catalog.yml @deepyaman

[astrojuanlu]

More:

• feat(datasets): Migrated PartitionedDataSet and IncrementalDataSet from main repository to kedro-datasets kedro-plugins#253
• fix(datasets): remove deprecation warnings kedro-plugins#255

And I agree, the priority should be to not show DeprecationWarnings that users can't action on.

[deepyaman]

Not sure the way of suppressing warnings is right...

>>> from kedro.io import MemoryDataSet
>>> from kedro.io.memory_dataset import MemoryDataSet
<stdin>:1: DeprecationWarning: 'MemoryDataSet' has been renamed to 'MemoryDataset', and the alias will be removed in Kedro 0.19.0
>>> 

We should still show DeprecationWarning.

It's annoying, but there's definitely precedent for still showing warnings, even when the user can't control them (like all the numpy warnings because of what libraries that use numpy do). Let me think if there's something better, but I think the lesser of two evils is to raise the warnings.

[astrojuanlu]

@deepyaman sorry, I clarify - I was not advocating to suppress the DeprecationWarnings. I was advocating for only showing them for code that users wrote, and when there's an alternative. In particular, kedro-datasets hasn't been migrated already, but doing from kedro_datasets.pandas import CSVDataSet will emit a DeprecationWarning that the user can't do anything about. That's what kedro-org/kedro-plugins#255 aims to fix.

[deepyaman]

@deepyaman sorry, I clarify - I was not advocating to suppress the DeprecationWarnings. I was advocating for only showing them for code that users wrote, and when there's an alternative. In particular, kedro-datasets hasn't been migrated already, but doing from kedro_datasets.pandas import CSVDataSet will emit a DeprecationWarning that the user can't do anything about. That's what kedro-org/kedro-plugins#255 aims to fix.

Understood; so are we saying we can undo the suppression in kedro/io/__init__.py then? #2746 is actually semi-choking on that (just a test, I could ignore it), hence why I started looking into it. :)

[deepyaman]

Do we need to update these extra install with Dataset as well?

kedro/setup.py

Lines 44 to 50 in 6e5e4e1

	pandas_require = {
	"pandas.CSVDataSet": [PANDAS],
	"pandas.ExcelDataSet": [PANDAS, "openpyxl>=3.0.6, <4.0"],
	"pandas.FeatherDataSet": [PANDAS],
	"pandas.GBQTableDataSet": [PANDAS, "pandas-gbq>=0.12.0, <0.18.0"],
	"pandas.GBQQueryDataSet": [PANDAS, "pandas-gbq>=0.12.0, <0.18.0"],
	"pandas.HDFDataSet": [

pip standardizes non-lowercase extras, so this would be a nice change for consistency, but it's not strictly necessary/high priority.

[zedrdave]

At the moment, it seems like kedro itself is triggering its own DeprecationWarning:

For example, the io package __init__.py contains:
from .cached_dataset import CachedDataset, CachedDataSet

(and similarly for other renamed ***Dataset classes)

This does not seem like a desirable behaviour… Or am I missing something?

[astrojuanlu]

For reference, @zedrdave question was discussed in #2882

[astrojuanlu]

Relevant: kedro-org/kedro-plugins#313

[zedrdave]

For reference, @zedrdave question was discussed in #2882

I do not think these 2 issues are the same.

One has to do with Kedro's import order (which triggers warnings when using custom classes), and the other has to do with the fact that current codebase (as of a month ago) was attempting to load both new and old class names, triggering a DeprecationWarning that users could not do anything about.

[astrojuanlu]

Ah, thanks for the clarification @zedrdave, I read too fast 👍🏽

[merelcht]

Closing this in favour of #2129 which is the full list of tasks for renaming assuming that datasets have by that point been removed from the core Kedro project.