sam build obscures pip ssl errors

[alex-lamoureaux]

Description:

When running in an environment that requires you to pass a cert to pip, running sam build fails with an ssl error. There is no way to directly specify a cert file via a flag to the sam build command, the way you can when running pip directly. In many cases (e.g. when all your requirements are already available locally) the traceback you receive makes no mention of an ssl error.

The only way to specify a cert is to write a pip config file that sets the ssl file path, which is a bit cumbersome for a CI pipeline run.

Steps to reproduce:

Run sam build in an environment that requires you to pass a cert file to pip

Observed result:

56 2021-01-04 16:20:42,731 | Running PythonPipBuilder:ResolveDependencies
57 2021-01-04 16:20:42,756 | calling pip download -r <truncated>/requirements.txt --dest /tmp/tmpv51uhu6u
58 2021-01-04 16:20:51,284 | PythonPipBuilder:ResolveDependencies failed
59 Traceback (most recent call last):
60   File "/usr/lib/python3.8/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 39, in execute
61     package_builder.build_dependencies(
62   File "/usr/lib/python3.8/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 137, in build_dependencies
63     self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
64   File "/usr/lib/python3.8/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 204, in build_site_packages
65     wheels, packages_without_wheels = self._download_dependencies(scratch_directory, requirements_filepath)
66   File "/usr/lib/python3.8/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 228, in _download_dependencies
67     deps = self._download_all_dependencies(requirements_filename, directory)
68   File "/usr/lib/python3.8/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 310, in _download_all_dependencies
69     self._pip.download_all_dependencies(requirements_filename, directory)
70   File "/usr/lib/python3.8/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 599, in download_all_dependencies
71     raise NoSuchPackageError(str(package_name))
72 aws_lambda_builders.workflows.python_pip.packager.NoSuchPackageError: Could not satisfy the requirement: path
73 During handling of the above exception, another exception occurred:
74 Traceback (most recent call last):
75   File "/usr/lib/python3.8/site-packages/aws_lambda_builders/workflow.py", line 269, in run
76     action.execute()
77   File "/usr/lib/python3.8/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 45, in execute
78     raise ActionFailedError(str(ex))
79 aws_lambda_builders.actions.ActionFailedError: Could not satisfy the requirement: path
80 Build Failed
81 2021-01-04 16:20:51,287 | Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 8931, 'exitReason': 'WorkflowFailedError', 'exitCode': 1, 'requestId': '84efccb7-1d4a-40a3-8cac-e107768315e4', 'installationId': '102a9a74-feb5-4f9f-aaac-429c61e7d91c', 'sessionId': 'af7ec9a7-63c9-45d3-9d3d-1f428159db2d', 'executionEnvironment': 'CLI', 'pyversion': '3.8.5', 'samcliVersion': '1.15.0'}}]}
82 2021-01-04 16:20:51,599 | Telemetry response: 200
83 Error: PythonPipBuilder:ResolveDependencies - Could not satisfy the requirement: path

Expected result:

The error output clearly states that the build failed because a cert needs to be specified, and there is a more intuitive, documented way of specifying that cert.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: amazon linux 2
2. sam --version: SAM CLI, version 1.15.0

Add --debug flag to command you are running

[hoffa]

I agree, that's not a very helpful message. The way it's currently done is by executing pip download and checking for non-zero exit code. For NoSuchPackageError, stderr is not displayed.

I don't have an easy way to reproduce this right now; but assuming pip download prints the clear details in stderr (would be super helpful to confirm if you have example error output from pip at hand!), we could include that as part of NoSuchPackageError.

[alex-lamoureaux]

18 $ pip download path
19 WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /simple/path/
20 WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /simple/path/
21 WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /simple/path/
22 WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /simple/path/
23 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /simple/path/
24 Could not fetch URL https://pypi.org/simple/path/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/path/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))) - skipping
25 ERROR: Could not find a version that satisfies the requirement path (from versions: none)
26 ERROR: No matching distribution found for path

This is the output I get from running pip download without configuring the cert. Not sure what cert chain it's looking at that has a self-signed cert in there. There are probably more common errors that would cause failures, too. Configuring pip to look for certs in an empty file might get some decent sample error output.

[kigawas]

After half an hour trying, I found this can be worked around by

1. Save below as pip.conf in your project folder

[global]
trusted-host = pypi.python.org
               pypi.org
               files.pythonhosted.org
               github.com
               YOUR_HOST.com

2. Set environment variable to specify pip config file: PIP_CONFIG_FILE=pip.conf sam build
   Or if you build in container: sam build --use-container --container-env-var PIP_CONFIG_FILE=pip.conf

Hooray!

[andrewhall-premier]

@kigawas This solved my problem! Mucho gracias!

[mndeveci]

Thanks for providing a resolution for this issue @kigawas !

I will move this issue to Lambda Builders repository to add more detailed error messages for such cases.