Providers of Large Size: Import is very slow in Python

[brent-at-aam]

Expected Behavior

When using Python, the time taken for a synth is only correlated to the volume of resources within a stack.

Actual Behavior

When using Python, running any operations with larger providers takes a long time due to lengthy module import load times. You could have a single resource and it would still take ~30s to even get started with the synth.

Steps to Reproduce

1. Use Python
2. Add the cdktf-cdktf-provider-aws library to your project
3. Make a small stack that uses any resource from the aws provider module
4. Run a synth

Versions

Running with the latest versions

Providers

No response

Gist

No response

Possible Solutions

This is really just a general issue for all providers, but only becomes a big problem for the large ones like AWS. Looking through the issues, this is related to #2792, which was ostensibly fixed by #3030.

Perhaps this is a regression due to some newer behavior in upstream packages but importing AWS is back to taking around 30 seconds and has been for quite a while. I've switched a few projects from python to typescript to get away from it but really it would be nice to not have to do that.

From what I can tell, the bulk of the time is spent loading the very large gzipped assembly in _jsii.

The root of the module loads it:

/init.py

from ._jsii import *

And worth noting that resource modules also load it:

/foo/init.py

from .._jsii import *

I know much of this is really just behavior of other libraries, and thus this might not be something you can control here. I also realize this is something you have probably already considered, but there is no discussion about it in issues I could find. Is it possible to instruct the package generation to build separate assemblies for each of the submodules of a provider package?

And you would then remove from ._jsii import * from the root. And resources would just import their specific jsii assembly.

Workarounds

None that are feasible

Anything Else?

No response

References

• azurerm provider: Python package cdktf-cdktf-provider-azurerm extremely slow to import  #2792
• feat: remove root-level import of all modules in python #3030

Help Wanted

• I'm interested in contributing a fix myself

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment