Import Error when importing complex Python libraries

[Embers-of-the-Fire]

I'm not a native speaker, so if it's convenient please do not use too many slangs and idioms as it's hard to understand.

Overview

I'm trying to use pyo3 to embed a python lib to my rust project, however when I'm trying to import the lib I got Import Errors.

Reproduce

Here's the simplest way to reproduce:

Python File Structure:

│  app.py
│
└─ utils
        bar.py
        foo.py
        __init__.py

# app.py
from utils import foo

def run():
    return foo()

# bar.py
def bar():
    return 12

# foo.py
from utils.bar import bar

def foo():
    return 20 + bar()

# utils/__init__.py
from utils.bar import bar
from utils.foo import foo

And my rust code is:

let py_foo = include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/python/utils/foo.py"));
let py_bar = include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/python/utils/bar.py"));
let py_app = include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/python/app.py"));
let from_python = Python::with_gil(|py| -> PyResult<u32> {
    PyModule::from_code(py, "", "utils/__init__.py", "utils")?;
    PyModule::from_code(py, py_foo, "utils/foo.py", "utils")?;
    PyModule::from_code(py, py_bar, "utils/bar.py", "utils")?;
    let app: Py<PyAny> = PyModule::from_code(py, py_app, "app.py", "")?
        .getattr("run")?
        .into();
    app.call0(py)?.extract(py)
});

println!("py: {}", from_python?);

And then I receive such error:

Error: ModuleNotFoundError: No module named 'utils.bar'; 'utils' is not a package

How could I solve this?

[Embers-of-the-Fire]

Or maybe I could zip my python code and unzip after distribution, then use sys.path to add my lib to interpreter's paths and run locally?

[davidhewitt]

Usually with these kinds of distribution issues I recommend taking a look at PyOxidizer, which is able to zip your project and dependencies into a single executable.

[Embers-of-the-Fire]

Thanks for your suggestion. However actually I want to use the python lib as a rust lib for micro-services but not a complete executable. I'll try to configure pyoxidizer, but I hope there could be another better solution.

In addition, is it possible to build packages like the above using pyo3? Or it's not designed to work in this way?

[davidhewitt]

I see. It's not something we've directly got a solution for at the moment. I think zipping and including in your lib as a byte stream is the way to go. If you want to avoid unzipping you might be able to add a loader to the import system which can use the zipimport mechanism to read directly from the bytes embedded in your lib, but that's pure speculation.