Restrict Python Version Mismatch between Pickled Object and Remote Envrionment

[Mecoli1219]

Tracking Issues

flyteorg/flyte#5907

Why are the changes needed?

In the Jupyter Notebook environment, we are using cloudpickle to package the entity at the binary level. When we load the pickled object with a different Python version, it will throw an unexpected error. Therefore, we should restrict loading pickled objects with the mismatched Python version.

What changes were proposed in this pull request?

Let's assume the Python version for the pickled object is X.

1. If we load it in a Python version smaller than X, it is possible to crash. Therefore, we need to handle loading exceptions.
2. If we successfully load the object, we need to check whether the Python version is matched. I added a new metadata in the pickled object to store the Python version when pickling, and this will be used for validation after loading.
3. Enable logging error messages when trying to extract the output from remote execution so that the user doesn't need to go to Console to get the error message.

How was this patch tested?

I set the Python version of Jupyter Notebook to 3.11 and built 3 images with different Python versions for remote execution, including 3.9, 3.11, and 3.12.

Setup process

Screenshots

• Remote Python version = 3.12
  
• Remote Python version = 3.11
  
• Remote Python version = 3.9
  

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

[Mecoli1219]

cc @thomasjpfan

[codecov]

Codecov Report

Attention: Patch coverage is 79.16667% with 5 lines in your changes missing coverage. Please review.

Project coverage is 78.50%. Comparing base (3fc51af) to head (21ae75c).

Files with missing lines	Patch %	Lines
flytekit/core/python_auto_container.py	77.77%	2 Missing ⚠️
flytekit/remote/remote.py	85.71%	2 Missing ⚠️
flytekit/remote/executions.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #2848       +/-   ##
===========================================
+ Coverage   45.53%   78.50%   +32.96%     
===========================================
  Files         196      196               
  Lines       20418    20462       +44     
  Branches     2647     2649        +2     
===========================================
+ Hits         9298    16064     +6766     
+ Misses      10658     3699     -6959     
- Partials      462      699      +237     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.