Crash on first commit when adding a pre-commit hook

[acdha]

I noticed an odd failure which might be related to #36. I was adding this to an existing project and added the following to my .pre-commit-config.yaml file:

    - repo: https://github.com/abravalheri/validate-pyproject
      rev: v0.12.2
      hooks:
          - id: validate-pyproject

I updated pyproject.toml to test it and ran git commit -a, waited for it to install the hook, and got this crash:

[ERROR] FileNotFoundError: [Errno 2] No such file or directory: '/Users/cadams/.cache/pre-commit/repobqc66rm2/py_env-python3/lib/python3.11/site-packages/validate_pyproject/pyproject_toml.schema.json'

What's interesting is that pre-commit run --all-files did not fail on the same repository. If I run it with -v, I see output like this indicating that it successfully validated the file:

Validate pyproject.toml..................................................Passed
- hook id: validate-pyproject
- duration: 0.09s

Valid file: pyproject.toml

If I change the pre-commit hook like this, it starts working, suggesting this is related to packaging:

    - repo: https://github.com/abravalheri/validate-pyproject
      rev: v0.12.2
      hooks:
          - id: validate-pyproject
            additional_dependencies: 
              - validate-pyproject

[abravalheri]

Hi @acdha could you please provide a minimal reproducer indicating the circumstances that this error is happening for you?

I tried to execute the following, but I cannot see anything wrong happening (so I cannot observe the error you are referring to):

> docker run --rm -it python:3.9-bullseye /bin/bash
git config --global init.defaultBranch main
git config --global user.name User
git config --global user.email [email protected]
pip install -U pre-commit
mkdir /tmp/test
cd /tmp/test
cat <<EOF > .pre-commit-config.yaml
repos:
- repo: https://github.com/abravalheri/validate-pyproject
  rev: v0.12.2
  hooks:
  - id: validate-pyproject
EOF
cat <<EOF > pyproject.toml
[project]
name = "helloworld"
version = "42"
EOF
git init .
git add .
git commit -m "Initial commit"
pre-commit install
sed -i 's/42/4.2/g' pyproject.toml
git commit -a -m "Change version"
# [INFO] Initializing environment for https://github.com/abravalheri/validate-pyproject.
# [INFO] Initializing environment for https://github.com/abravalheri/validate-pyproject:.[all].
# [INFO] Installing environment for https://github.com/abravalheri/validate-pyproject.
# [INFO] Once installed this environment will be reused.
# [INFO] This may take a few minutes...
# Validate pyproject.toml..................................................Passed
# [main 4614763] Change version
#  1 file changed, 1 insertion(+), 1 deletion(-)

[abravalheri]

Please feel free to edit the script above and adapt it to create a minimal reproducer for the error you are seeing.

[acdha]

It no longer reproduces but I noticed something interesting when running pre-commit with a clean cache:

[INFO] Initializing environment for https://github.com/abravalheri/validate-pyproject.
[INFO] Initializing environment for https://github.com/abravalheri/validate-pyproject:.[all].

That happens whether or not I put the additional_dependencies line in whereas before only the first line shows up, like the other hooks. I'm going to see if I can reproduce this with an earlier version of pre-commit than the current one.

[jvacek]

So I'm experiencing this right now.

repos:
  - repo: https://github.com/abravalheri/validate-pyproject
    rev: v0.18
    hooks:
      - id: validate-pyproject
        # Optional extra validations from SchemaStore:
        files: '.*pyproject\.toml'
        additional_dependencies: ["validate-pyproject-schema-store[all]"]

Note, that version 0.19 is released right now. The fix for me is to either:

• remove the additional dependency
• update to 0.19

So I'm guessing something breaks when the first time the pre-commit initialises is when it's for a version that's not latest, and the additional dependencies are no longer available or something like that?

[abravalheri]

@jvacek I suspect that what is happening in your case is that probably validate-ppyroject-schema-store == "2024.08.19" gets installed (because there is no version constraints) and that might depend on validate-pyproject>=0.19.

If you either restrict validate-pyproject-schema-store[all] to an older version or update rev to v0.19 that should work I guess...

[henryiii]

It shouldn’t depend on anything new. I’ve not touched it besides the auto SchemaStore updates in a while. Is the error message the same as above?

[abravalheri]

@henryiii, does validate-pyproject-schema-store[all] uses the schemas distributed by validate-pyproject itself to feed the snapshots?

In 0.19 I introduced a new format, I was assuming that the crash was because the schema needs this new format. But probably I am wrong then.

@jvacek, could you please provide the full error trace?

[jvacek]

@abravalheri how exactly would you like me to do that? I'm just using this via pre-commit, I'm not running the tool directly

[abravalheri]

On a crash, pre-commit should give you a log file, right? I believe there should be some form of trace in that file.

Otherwise, could you please send here the error message you saw and help me with the reproducer below? Right now it is not crashing. Could you try to modify it to produce the same behaviour you are experiencing?

# > docker run --rm -it python:3.12.5-bookworm /bin/bash
mkdir /tmp/proj
cd /tmp/proj

cat <<EOF > pyproject.toml
[project]
name = "proj"
version = "42"
EOF

cat <<EOF > .pre-commit-config.yaml
repos:
  - repo: https://github.com/abravalheri/validate-pyproject
    rev: v0.18
    hooks:
      - id: validate-pyproject
        # Optional extra validations from SchemaStore:
        files: '.*pyproject\.toml'
        additional_dependencies: ["validate-pyproject-schema-store[all]"]
EOF
git init .
git add pyproject.toml
git add .pre-commit-config.yaml
python -m venv .venv
.venv/bin/python -m pip install pre-commit
.venv/bin/pre-commit install
.venv/bin/pre-commit run --verbose
# [INFO] Initializing environment for https://github.com/abravalheri/validate-pyproject.
# [INFO] Initializing environment for https://github.com/abravalheri/validate-pyproject:validate-pyproject-schema-store[all].
# [INFO] Installing environment for https://github.com/abravalheri/validate-pyproject.
# [INFO] Once installed this environment will be reused.
# [INFO] This may take a few minutes...
# Validate pyproject.toml..................................................Passed
# - hook id: validate-pyproject
# - duration: 1.0s
# 
# Valid file: pyproject.toml

[jvacek]

On a crash, pre-commit should give you a log file, right

Doesn't seem like it. This is the best I got.

$ pre-commit run -v
[WARNING] Unstaged files detected.
[INFO] Stashing unstaged files to /Users/jvacek/.cache/pre-commit/patch1724251202-13884.
markdownlint-fix.....................................(no files to check)Skipped
- hook id: markdownlint-fix
Lint YAML................................................................Passed
- hook id: yamllint
- duration: 0.12s
shfmt................................................(no files to check)Skipped
- hook id: shfmt
shellcheck...........................................(no files to check)Skipped
- hook id: shellcheck
trim trailing whitespace.................................................Passed
- hook id: trailing-whitespace
- duration: 0.05s
fix end of files.........................................................Passed
- hook id: end-of-file-fixer
- duration: 0.04s
check toml...............................................................Passed
- hook id: check-toml
- duration: 0.04s
Validate pyproject.toml..................................................Failed
- hook id: validate-pyproject
- duration: 0.1s
- exit code: 1

[ERROR] FileNotFoundError: [Errno 2] No such file or directory: '/Users/jvacek/.cache/pre-commit/repo_vlx0urg/py_env-python3.12/lib/python3.12/site-packages/validate_pyproject/pyproject_toml.schema.json'

[INFO] Restored changes from /Users/jvacek/.cache/pre-commit/patch1724251202-13884.

[henryiii]

Have you checked to see if that file exists? Is that the correct directory structure?

[abravalheri]

Yeah, I am failing to see why the file would not be present in the disk:

• It exists in the repository: https://github.com/abravalheri/validate-pyproject/blob/v0.18/src/validate_pyproject/pyproject_toml.schema.json
• It exists in the sdist: https://inspector.pypi.io/project/validate-pyproject/0.18/packages/db/2b/ea513aeb27dad0d5cff01de749cca1923dfe9240a095b1fffb2d5f80af70/validate_pyproject-0.18.tar.gz/validate_pyproject-0.18/src/validate_pyproject/pyproject_toml.schema.json
• It exists in the wheel: https://inspector.pypi.io/project/validate-pyproject/0.18/packages/68/50/da93904c1006fe884b5936cb789ca2a19eb0e2908efd2b27036da4a16846/validate_pyproject-0.18-py3-none-any.whl/validate_pyproject/pyproject_toml.schema.json

@jvacek did you had the chance to check the path as suggested by Henry?

If you can have a look on the script I shared in #99 (comment) and help to evolve it into a reproducer, that would be very helpful (since so far I did not manage to reproduce the problem).

[jvacek]

/Users/jvacek/.cache/pre-commit/repoliqbup2z/py_env-python3.12/lib/python3.12/site-packages/validate_pyproject/ exists and there is no schema in there.

And now this happens when i've got 0.19 in the config too.

Every here and there, I will run pre-commit clean to reset the things and it fixes it.

I'll see if I can do something to repro this, but not sure when I'll have the time.