Parallel run #[pyfunction] to get multi PyObject in Rust

[sun-rs]

#[pyfunction]
pub fn get_dict(py: Python) -> PyObject {
        let map = PyDict::new(py); 
        SYMBOL_MAP.iter().for_each(|(k, v)| {
            map.set_item(v.name.clone(), k.to_string()).expect("Failed to set_item on dict");
        });
        map.into()
    }    

If I call this func as a python extension function it would trigger gil definitely which makes multi thread not efficiency.

I want to create multi Dict in rust and give the dict to python after, if I just call this func in pure Rust env with rayon, would it be real parallelism?

[davidhewitt]

You can't interact with Python objects from multiple threads, so you'll need to use a thread-safe datastructure to build the map. Perhaps hashbrown::HashMap with the hashbrown/rayon feature enabled. You can then use pyo3/hashbrown feature to enable conversion of hash brown::HashMap to PyDict once the multithreaded part is complete.

[sun-rs]

The process that pyo3 convert hashmap to PyDict still must be one by one rather than parallel？

If I need to iter two vec to create a dict, set item to a PyDict directly is faster than generate a hashmap first then converted by pyo3 automaticlly？

[davidhewitt]

A Python dict is not thread-safe so you can only build it from a single thread, yes.

The only way to answer which is faster is benchmark your use case and determine what performs better in your context.