Parameter sensitivity analytic engine. #133
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit adds doc strings and type annotation for CmfQuery class methods.
- References to `client` in the source code (e.g., CmfClient instead of CmfQuery) are removed.
- Several bugs related to checking input dict parameters are fixed (e.g., `d = d or {}` where it should be `if d is None: d= {}`). Now, the corrent object is returned when input dict is just empty.
- Missing doc strings and type annotations are added.
- Additional checks and log messages are added.
Fixing one possible bug related to accessing a column in a data frame when this data frame is empty. Adding key mapper classes to help map source to target keys when copying dictionaries.
The API implements graph-like API to traverse CMF metadata in a graph-like manner. The entry point is the `MetadataStore` class that retrieves from metadata store pipelines, stages, executions and artifacts. Users can specify search query to specify what they want to return (the query is basically the value for the `filter_query` parameter of the `ListOptions` class ML Metadata (MLMD) library). Each node mentioned above (pipeline, stages, executions and artifacts) havs its own Python wrapper class that provides developer-friendly API to access node's parameters and travers graph of machine learning concepts (e.g., get all stages of a pipeline or get all executions of a stage). The graph API also provides the `Properties` wrapper for MLMD's properties and custom_propertied nodes' fields. This wrapper implements the `Mapping` API and automatically converts MLMD's values to Python values on the fly.
- Adding unit tests (97% coverage). - Renaming certain classes, redesigning implementation of several methods. - Adding `Type` class that represents concepts such as ContextType, ExecutionType and ArtifactType in MLMD library.
- Implementation of multiple analytic functions. - Unified mechanism to traverse graph of artifacts along their dependency paths.
- New traverse API. - Base methods. - Allow users to specify selection criteria for artifacts in some methods.
- Adding HPE header. - Improving doc strings for classes that are responsible for traversing MLMD graph of artifacts.
This analytic engine is designed to compute how machine learning metrics such as accuracy depend on variations of (hyper-) parameters such as learning rate. In this initial implementation, the engine consumes data from graph or tabular data sources. It creates a `report` object that brings together various pieces of information that can be useful to build true reports. This information includes now name of a parameter, name of a metric and list of metric values.
annmary-roy
reviewed
Dec 7, 2023
There was a problem hiding this comment.
@sergey-serebryakov A session to explain these changes would be good
annmary-roy
reviewed
Dec 8, 2023
| raise NotImplementedError | ||
|
|
||
|
|
||
| class ParameterSensitivityReport: |
There was a problem hiding this comment.
@sergey-serebryakov should'nt we consider the parameter values also for sensitivity report ?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related Issues / Pull Requests
This PR depends on #131.
Description
This analytic engine is designed to compute how machine learning metrics such as accuracy depend on variations of (hyper-) parameters such as learning rate. In this initial implementation, the engine consumes data from graph or tabular data sources. It creates a
reportobject that brings together various pieces of information that can be useful to build real-world reports. This information includes name of a parameter, name of a metric and list of metric values.What changes are proposed in this pull request?
Checklist: