Skip to content
This repository has been archived by the owner on Sep 27, 2024. It is now read-only.

Latest commit

 

History

History
211 lines (138 loc) · 9.68 KB

RELEASE.md

File metadata and controls

211 lines (138 loc) · 9.68 KB
Raw

Model Card Toolkit Releases

TFX Addons follows Semantic Versioning 2.0 strategy.

Making Major and Minor Releases

  1. Create new rX.Y branch on https://github.com/tensorflow/model-card-toolkit from main.
  2. Create new PR with updates to version.py against rX.Y branch.
    • Set the correct version and suffix in version.py.
    • Ensure proper supported Python libraries are set in setup.py.
  3. Create a new release from rX.Y branch. Create a tag with vX.Y.Z name.
    • Add updates for new features, enhancements, bug fixes
    • Add contributors using git shortlog <last-version>..HEAD -s
  4. Create a new PR and merge an increase of _MINOR_VERSION in version.py on main to get ready for next release.

Making Patch Releases

  1. Cherry-pick commits to rX.Y branch. Release team can just port PR by commenting "/cherry-pick rX.Y" in a merged PR.
  2. Create new PR with increasing _PATCH_VERSION in version.py against rX.Y branch.
    • Set the correct version and suffix in version.py.
    • Ensure proper supported Python libraries are set in setup.py.
  3. Create a new release from rX.Y branch. Create a tag with vX.Y.Z name.
    • Add updates for new features, enhancements, bug fixes
    • Add contributors using git shortlog <last-version>..HEAD -s

Model Card Toolkit Release Team

Current Release Team:

  • Hannes Hapke - @hanneshapke
  • Gerard Casas Saez - @casassg
  • Suzen Fylke - @codesue

Past Releases

Release 1.3.2

Major Features and Improvements

Bug fixes and other changes

  • Remove semantic-version dependency.
  • Update jinja2 dependency.

Breaking changes and Deprecations

Release 1.3.1

Major Features and Improvements

Bug fixes and other changes

  • Fix bug where ModelCardGenerator fails if template_io is unspecified.

Breaking changes and Deprecations

Release 1.3.0

Major Features and Improvements

  • ModelCardArtifact, a TFX/MLMD artifact which references the assets generated by MCT. This can be created with model_card_toolkit.tfx.create_and_save_artifact().
  • tfx.executor.Executor, which can be used as the core MCT logic for a TFX custom component.
  • tfx.component.ModelCardGenerator, a component that can be used to generate Model Cards directly in a TFX pipeline.

Bug fixes and other changes

  • ModelCard.merge_from_json(), allowing ModelCards to be populated with JSON without overwriting all pre-existing fields.
  • Minor styling change to table CSS in default HTML template
  • Minor update to CSS linting in default HTML template

Breaking changes and Deprecations

  • source classes now uses tfx.types.Artifact instead of standard_artifacts.

Release 1.2.0

Major Features and Improvements

  • ModelCard updates
    • Fields
      • Add model_details.path.
        • This is populated with the new ModelSource object, which stores a reference to a TFX PushedModel.
      • Add model_parameters.input_format_map and model_parameters.output_format_map.
        • These are key-value pairs, and are used to render inputs and outputs in tabular form. They can be used as an alternative to the singular model_parameters.input_format and model_parameters.output_format fields.
    • Functions
      • Add from_json().
  • model_card_toolkit.source
    • This is a new submodule, and is responsible for ModelCardToolkit's' inputs (see TFX standard artifacts). It provides the following classes:
      • MlmdSource: Args to extract data from TFX artifacts in MLMD. Contains the mlmd_store and model_uri args. Previously, these were args to ModelCardToolkit.
      • Source: Args to extract data from TFX artifacts outside MLMD (by passing in a path to the artifact, or by passing in the artifact directly). Contains tfma, tfdv, and model args.

Bug fixes and other changes

  • ModelCardToolkit
    • model_card.quantitative_analysis.performance_metrics is now populated when a tfma.EvalResult is found in MLMD store.
    • export_format() and update_model_card() now accept model_card_pb2.ModelCard's, in addition to model_card.ModelCard's.
  • tfx_util
    • Add annotate_eval_result_metrics(), which appends PerformanceMetrics to a ModelCard based on a tfma.EvalResult.
    • Add read_stats_protos(), which returns dataset stats protos for all splits in the provided directory.
    • Add filter_metrics() to facilitate filtering out unwanted TFMA metrics in model cards.
    • Add filter_features() and tfx._util.read_stats_protos_and_filter_features() to facilitate filtering out unwanted TFDV features in model cards.
  • PerformanceMetrics
    • Add confidence_interval field.

Breaking changes and Deprecations

  • Replace ModelCardToolkit(output_dir, mlmd_store, model_uri) with ModelCardToolkit(output_dir, mlmd_source, source). See "Major Features and Improvements" above for details.
  • Complete deprecation of ModelCardToolkit.update_model_card_json(). Users should migrate to ModelCardToolkit.update_model_card(), which uses a proto representation. Alternatively, users can use ModelCard.to_json() and ModelCard.from_json() to interact with JSON representations.

Release 1.1.0

Major Features and Improvements

Bug fixes and other changes

  • Update TFX compatibility to TFX 1.2.
  • Fix bug where all datasets from MLMD were being compressed into one model_card.Dataset object.

Breaking changes and Deprecations

Release 1.0.0

Major Features and Improvements

  • Introduce model_card.proto. See https://developers.google.com/protocol-buffers for more info.
  • All classes in model_card_toolkit.model_card submodule now have to_proto(), merge_from_proto(), copy_from_proto(), and clear() functions.
  • ModelCardToolkit.export_format() now accepts model_card arg.
  • json_util.update(), which updates a v0.0.1 JSON object to a v0.0.2 JSON object.

Bug fixes and other changes

  • Update default template layout so charts can wrap to multiple rows
  • Installing from source now requires Bazel>=2.0.0.
  • Update model card templates to use new schema.
  • model_card_toolkit.utils.validation.validate_json_schema() can now validate both schema v0.0.1 and v0.0.2.
  • Add _jinja_loader attribute to model_card_toolkit to allow custom Jinja loaders.

Breaking changes

  • JSON schema v0.0.2 replaces JSON schema v0.0.1.
  • ModelCardToolkit.update_model_card_json() deprecated and replaced with ModelCardToolkit.update_model_card(). Writes to data/model_card.proto instead of data/model_card.json.
  • graphics.annotate_dataset_feature_statistics_plots accepts a list of stats files instead of two individual train and eval stats params.

Deprecations

Release 0.1.3

Major Features and Improvements

Bug fixes and other changes

Breaking changes

Deprecations

Release 0.1.2

Major Features and Improvements

  • In default_template.md.jinja and default_template.html.jinja, generate metrics table from quantitative_analysis.performance_metrics.

Bug fixes and other changes

  • Reference URLs in default HTML and Markdown template are now hyperlinks
  • Fix bug where Considerations div is displayed in HTML model cards, even if Considerations div is empty.
  • Update required fields in schema.
    • Removed considerations as required field.
    • Add lower_bound and upper_bound as required fields to confidence_interval.
  • Fixed the part dependencies error for new pip dependency resolver.
  • Update how UI templates are copied to be compatible with different platforms (colab, wetlab).
  • Add model_card_toolkit.validation.validate_json_schema(), a function to validate a Python dictionary against the Model Card JSON schema.
  • Fix the bug that some slices may have extra metrics that other slices does not have. e.g. ERROR metric.

Breaking changes

Deprecations

Release 0.1.1

Major Features and Improvements

  • add Markdown template

Bug fixes and other changes

  • remove quantitative_analysis from required fields
  • add input_format and export_format fields
  • add model_architecture, input_format, and export_format to HTML template
  • add Cats vs Dogs util for Standalone_Model_Card_toolkit_Demo.ipynb

Breaking changes

  • Rename _figure_to_base64str to figure_to_base64str

Deprecations

Release 0.1.0

Major Features and Improvements

Initial release of Model Card Toolkit.

Bug fixes and other changes

Breaking changes

Deprecations