Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add decorator functionality to validate_schema function #255

Open
wants to merge 20 commits into
base: planning-1.0-release
Choose a base branch
from

Commits on Jul 16, 2024

  1. All changes to make validate_schema behave both as a function and as …

    …a decorator to other functions. Also added associated changes to tests and README
    kunaljubce committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    3a36964 View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2024

  1. update column extension function names and desc in readme

    Fateme Tardasti authored and kunaljubce committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    aafb6f8 View commit details
    Browse the repository at this point in the history
  2. DOC: CONTRIBUTING.md - add details on precommit & local GitHub Action…

    …s setup
    
    Add details to `CONTRIBUTING.md` on auto-assigning issues, pre-commit installation, and GitHub Actions local setup using 'act'.
    
    * **Auto-assigning issues**: Add a section explaining auto-assigning issues on the comment 'take', referencing the configuration in `.github/workflows/assign-on-comment.yml`.
    * **Pre-commit installation and execution**: Add a section detailing pre-commit installation and execution, referencing the configuration in `.pre-commit-config.yaml`.
    * **GitHub Actions local setup using 'act'**: Add a section providing instructions for GitHub Actions local setup using 'act', referencing the configuration in `.github/workflows/ci.yml`. Include instructions for running specific jobs and handling MacBooks with M1 processors.
    nijanthanvijayakumar authored and kunaljubce committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    40a62b1 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    5e4eafa View commit details
    Browse the repository at this point in the history
  4. Change pip to poetry for pre-commit installation

    According to the review comment, make the pip installs as poetry installs
    nijanthanvijayakumar authored and kunaljubce committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    ca75b36 View commit details
    Browse the repository at this point in the history
  5. Bump jupyterlab from 3.6.7 to 3.6.8

    Bumps [jupyterlab](https://github.com/jupyterlab/jupyterlab) from 3.6.7 to 3.6.8.
    - [Release notes](https://github.com/jupyterlab/jupyterlab/releases)
    - [Changelog](https://github.com/jupyterlab/jupyterlab/blob/main/CHANGELOG.md)
    - [Commits](https://github.com/jupyterlab/jupyterlab/compare/@jupyterlab/[email protected]...@jupyterlab/[email protected])
    
    ---
    updated-dependencies:
    - dependency-name: jupyterlab
      dependency-type: direct:development
    ...
    
    Signed-off-by: dependabot[bot] <[email protected]>
    dependabot[bot] authored and kunaljubce committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    35234bb View commit details
    Browse the repository at this point in the history
  6. Drop Spark-2 support and update dependencies

    - Update deps
    - Update Ruff
    - Corresponding updates of pyproject
    - Slightly update Makefile
    - Drop Support of Spark2
    - Drop Support of Spark 3.1-3.2
    - Minimal python is 3.10 from now
    - Apply new ruff rules
    - Apply ruff linter
    
     On branch 202-drop-pyspark2
     Changes to be committed:
    	modified:   .github/workflows/ci.yml
    	modified:   Makefile
    	modified:   poetry.lock
    	modified:   pyproject.toml
    	modified:   quinn/append_if_schema_identical.py
    	modified:   quinn/dataframe_helpers.py
    	modified:   quinn/keyword_finder.py
    	modified:   quinn/math.py
    	modified:   quinn/schema_helpers.py
    	modified:   quinn/split_columns.py
    	modified:   quinn/transformations.py
    SemyonSinchenko authored and kunaljubce committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    cd42f32 View commit details
    Browse the repository at this point in the history
  7. Remove extension functions

    Related to mrpowers-io#237
    
    Remove deprecated extension functions from the codebase
    * Delete `quinn/extensions/dataframe_ext.py` and `quinn/extensions/spark_session_ext.py`
    * Remove import statements for `dataframe_ext` and `spark_session_ext` in `quinn/extensions/__init__.py`
    * Remove per-file ignores for `quinn/extensions/dataframe_ext.py` and `quinn/extensions/__init__.py` in `pyproject.toml`
    * Delete test files `tests/extensions/test_dataframe_ext.py` and `tests/extensions/test_spark_session_ext.py`
    nijanthanvijayakumar authored and kunaljubce committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    3afa41f View commit details
    Browse the repository at this point in the history
  8. Restore SparkSession extension module & delete patching row 48

    * Create the `extensions/` directory.
    * Create the `__init__.py` file within it with the license header.
    * Create the `spark_session_ext.py` file within the extensions directory
    * Update the doc-string in create_df to align with the parameter list.
    * Format modified files & lint the code using ruff (successful)
    nijanthanvijayakumar authored and kunaljubce committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    34fa8e5 View commit details
    Browse the repository at this point in the history
  9. Move the create_df function to dataframe_helpers

    As a matter of fact, a create_df function already exists under the
    dataframe_helpers.py Following are the changes introduced by this commit.
    * Update `dataframe_helpers.py` by updating the doc-string in the create_df function.
    * Remove quinn/extensions/ directory along with the two `.py` files involved.
    * Remove the ruff check from the pyproject.toml.
    * Finally, format the dataframe_helpers.py with ruff
    nijanthanvijayakumar authored and kunaljubce committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    a5fb013 View commit details
    Browse the repository at this point in the history
  10. Updates from review

    SemyonSinchenko authored and kunaljubce committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    7305223 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    501f3b6 View commit details
    Browse the repository at this point in the history
  12. update column extension function names and desc in readme

    enable UP007
    Fateme Tardasti authored and kunaljubce committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    f7cc2c2 View commit details
    Browse the repository at this point in the history
  13. improve documentation of makefile

    fpgmaas authored and kunaljubce committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    398198c View commit details
    Browse the repository at this point in the history
  14. Update the files according to the review comments

    * Remove the test_spark_connect.py file as it is not relevant anymore.
    * Update the Makefile to remove spark-connect test
    * Hardcode the hadoop version to 3 as 2 is EOL.
    nijanthanvijayakumar authored and kunaljubce committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    8345d1f View commit details
    Browse the repository at this point in the history
  15. apply hotfix

    apply hotfix
    
    update lock file
    fpgmaas authored and kunaljubce committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    49c9ca1 View commit details
    Browse the repository at this point in the history
  16. improve use of ruff

    update makefile
    
    add make command for ruff
    fpgmaas authored and kunaljubce committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    ff48cb6 View commit details
    Browse the repository at this point in the history
  17. relax pre-commit version

    fpgmaas authored and kunaljubce committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    9d034d4 View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    086a5c3 View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    1653984 View commit details
    Browse the repository at this point in the history