-
Notifications
You must be signed in to change notification settings - Fork 539
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pipy dependency generated py_library ignores .pyc files in installed package. #2178
Comments
This bug was mentioned in a recent PR, so I'll repost what I said there. The basic gist is that pyc files have been problematic for build determinism and correctness. Usually pyc files are created by the runtime when they are first imported. This is problematic for Bazel in two ways:
All that said, this behavior is somewhat surprising and perplexing. By default, individual files are symlinked, and they're in a sandbox, so it's not clear how pycs were getting created in the underlying repo directory. But, all the CI issues went away after ignoring pyc files. |
@rickeylev Thanks for the explanation. I might not fully grasp the details but I get your point on
What I am curious is:
|
Not really. Both cases are "source" files. So to Bazel, whether it was installed from the package, created by a user, or created by some other process, they look the same. The closest we're able to do is prevent the interpreter from being able to write the files, e.g. by making everything read-only. Windows strikes here again, though, because it's security model allows an admin user to ignore such read-only attributes. Alternatively, we could force the interpreter to use DONTWRITEBYTECODE. That might be doable (setting interpreter flags/env has been trickier than expected; this is the sort of thing we have to try to flush out edge cases).
Yes, but it doesn't prevent the deleting-open-file race condition I mentioned. This only affects windows (only windows gives an error about deleting an open file). It'd be acceptable to have windows ignore pyc, I suppose; our windows support is borderline and best-effort. The main thing is preventing the build errors that occur, where the only thing you can do is re-run bazel and hope things interweave successfully. |
Also, to clarify:
Part of me suspects this undesirable behavior is limited to (a) just the interpreter and its stdlib, and/or (b) local, non-sandboxed execution of some sort. That's just a theory -- if it could be shown that theory is correct, maybe we could think of some more solutions. Two more thoughts. Precompiling is a builtin feature of the rules now. By setting If the pypi package is a pyc-only package (rare, but supported), then I think it's reasonble for py_library to accept |
In my use case, I am okay with the build errors because the package that I am concerned with is a pyc-only library with pyi files for tooling. I am happy with any special case solution that just works with pure pyc packages. |
🐞 bug report
Affected Rule
https://github.com/bazelbuild/rules_python/blob/main/docs/pypi-dependencies.md
Specifically
py_library
withname = "pkg"
generated byIs this a regression?
I don't know.
Description
I couldn't import a module from pip package that ships python modules with .pyi and .pyc files.
For example, I have a example_package that ships with the following file structure
According to https://peps.python.org/pep-3147/#flow-chart,
Should be successful. But the
py_library
generated specifically excludes"**/*.pyc"
fromdata
. If I manually remove the"**/*.pyc"
exclusion, the import statement works.How
🔬 Minimal Reproduction
import foo from example_package
should work but errors out with module not found error.🔥 Exception or Error
🌍 Your Environment
Operating System:
Output of
bazel version
:Rules_python version:
Anything else relevant?
The text was updated successfully, but these errors were encountered: