pydeps randomly generates compiled pyc files when examine python source file

[ZiqianXu]

Hi pydeps team, thanks for implementing this great tool!

When using the tool to examine python source files, we noticed sometimes a compiled pyc file would be automatically generated for the all python files inside the same directory. For example pyyaml.

yaml $ ls -la __pycache__/
ls: cannot access '__pycache__/': No such file or directory
yaml $ pydeps dumper.py -vvvvvvvv --show-deps --no-output --noshow --include-missing
run_script 'dumper.py' 
    load_module(PY_SOURCE) fqname=__main__, fp=fp, pathname=dumper.py 
    import_hook: name(emitter) caller(Module(name=__main__, file='dumper.py', path=None)) fromlist(None) level(1) 
        determine_parent Module(name=__main__, file='dumper.py', path=None) 1 
        ImportError: 'relative importpath too deep' 
        import_hook: name(serializer) caller(Module(name=__main__, file='dumper.py', path=None)) fromlist(None) level(1) 
            determine_parent Module(name=__main__, file='dumper.py', path=None) 1 
            ImportError: 'relative importpath too deep' 
            import_hook: name(representer) caller(Module(name=__main__, file='dumper.py', path=None)) fromlist(None) level(1) 
                determine_parent Module(name=__main__, file='dumper.py', path=None) 1 
                ImportError: 'relative importpath too deep' 
                import_hook: name(resolver) caller(Module(name=__main__, file='dumper.py', path=None)) fromlist(None) level(1) 
                    determine_parent Module(name=__main__, file='dumper.py', path=None) 1 
                    ImportError: 'relative importpath too deep' 
                load_module -> Module(name=__main__, file='dumper.py', path=None) 
{
    "dumper.py": {
        "bacon": 0,
        "imports": [
            "emitter",
            "representer",
            "resolver",
            "serializer"
        ],
        "name": "dumper.py",
        "path": null
    },
    ...
}
yaml $ ls -la __pycache__/
total 172
drwxr-xr-x. 1 ziqian ziqian   752 May  3 07:07 .
drwxr-xr-x. 1 ziqian ziqian   372 May  3 07:07 ..
-rw-r--r--. 1 ziqian ziqian  3562 May  3 07:07 composer.cpython-38.pyc
-rw-r--r--. 1 ziqian ziqian 20821 May  3 07:07 constructor.cpython-38.pyc
-rw-r--r--. 1 ziqian ziqian  3410 May  3 07:07 cyaml.cpython-38.pyc
-rw-r--r--. 1 ziqian ziqian  1822 May  3 07:07 dumper.cpython-38.pyc
-rw-r--r--. 1 ziqian ziqian 25352 May  3 07:07 emitter.cpython-38.pyc
-rw-r--r--. 1 ziqian ziqian  2299 May  3 07:07 error.cpython-38.pyc
-rw-r--r--. 1 ziqian ziqian  3973 May  3 07:07 events.cpython-38.pyc
-rw-r--r--. 1 ziqian ziqian 11844 May  3 07:07 __init__.cpython-38.pyc
-rw-r--r--. 1 ziqian ziqian  2163 May  3 07:07 loader.cpython-38.pyc
-rw-r--r--. 1 ziqian ziqian  1724 May  3 07:07 nodes.cpython-38.pyc
-rw-r--r--. 1 ziqian ziqian 11923 May  3 07:07 parser.cpython-38.pyc
-rw-r--r--. 1 ziqian ziqian  4536 May  3 07:07 reader.cpython-38.pyc
-rw-r--r--. 1 ziqian ziqian 10068 May  3 07:07 representer.cpython-38.pyc
-rw-r--r--. 1 ziqian ziqian  5497 May  3 07:07 resolver.cpython-38.pyc
-rw-r--r--. 1 ziqian ziqian 25268 May  3 07:07 scanner.cpython-38.pyc
-rw-r--r--. 1 ziqian ziqian  3319 May  3 07:07 serializer.cpython-38.pyc
-rw-r--r--. 1 ziqian ziqian  4934 May  3 07:07 tokens.cpython-38.pyc

python version is 3.8 and pydeps version is v1.12.20.

Is this expected behavior due to checking import-opcodes in python bytecodes for imports? But it does not happen for all python source file directory, so far we only find pyyaml filepath having the issue. What's the reason for that? Cam we disable the auto-generation of compiled python file?

Thanks for looking into this!

[thebjorn]

Hi @ZiqianXu and thank you for your interest in Pydeps.

This is indeed an artifact of how modulefinder and Python works. modulefinder is a Python standard module that pydeps uses to find the importgraph, see https://docs.python.org/3/library/modulefinder.html

I'm not aware of any hooks to prevent saving .pyc files (https://github.com/python/cpython/blob/main/Lib/modulefinder.py)

[ZiqianXu]

Hey @thebjorn, thanks for the quick response!

But what's the reason for .pyc only generated for some python files like pyyaml but not for other packages like cffi?

[thebjorn]

I'm guessing because it's mostly written in C...?

[ZiqianXu]

Hey @thebjorn, I think the compiled .pyc for yaml created when yaml is imported in pydeps. Somehow python module loader fails to compile to the system python directory(like /usr/local/lib/python3.8/dist-packages/yaml/) but the current yaml directory( like /tmp/testDir/yaml/)

So the issue is not about compiled .pyc got created when finding imports, but it's created in the wrong directory.