python arm build issue still failing in some cases #348
Comments
|
Thanks for reporting the issue, we will be investigating it further. Please watch this channel for more updates, and feel free to reach out. |
|
another function with the following requirements: requests plus a layer with this: dynamodb-encryption-sdk that was failing to build before is now building successfully, if that helps. |
|
@blathersbugs thanks for raising this issue. I've tried with following dependencies with Can you provide more details for these ones (python version for instance) to test it further? I also tried with When I check PyPI, I can't find the |
mndeveci
added
blocked/more-info-needed
|
hi @mndeveci work as expected, so there is nothing to reproduce there. if you download the py7zr source files it lists the following as its depedencies: |
|
hi @mndeveci @mildaniel any updates? |
aws-sam-cli-bot
added
maintainer/need-followup
and removed
maintainer/need-followup
labels
|
Hi @blathersbugs , I've retired with following function definition; with following when I run I remember this was failing before so please let me know if I am missing something here. Otherwise I would assume the library have fixed the issue with their dependencies that we discussed earlier. Thanks |
|
hi @mndeveci I can report that the build is now succeeding for me as well, I see no reason not to close this issue. |
|
@mndeveci this has started occurring again for me, it did work for a while and now it isn't. Installing py7zr via pip works fine on a t4g amazon linux 2 instance if that helps, I tried that several times and it worked consistently. It failed on something different this time, here is the logs: 2022-07-14 18:26:00,608 | Building codeuri: /codebuild/output/src567373820/src/lambda/hp-png-cap-unzip runtime: python3.9 metadata: {} architecture: arm64 functions: unzip894 | 2022-07-14 18:26:00,609 | Building to following folder /codebuild/output/src567373820/src/.aws-sam/build/unzip |
|
@blathersbugs This is a dependency problem. From the output: |
jfuss
added
blocked/close-if-inactive
|
@jfuss So I don't think it is a dependency at all, I think there is some sort of issue with the dependency resolution used by aws-lambda-builders. |
|
hi @mndeveci @jfuss it fails with inflate64, here is the build output: Fetching public.ecr.aws/sam/build-python3.9:latest-arm64 Docker container image...............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................193 | 2022-07-15 20:43:34,641 | Mounting /codebuild/output/src341811617/src/lambda/helloworld as /tmp/samcli/source:ro,delegated inside runtime container let me check if any of the other dependencies cause this as well EDIT: all other dependencies succeed in building I downloaded the inflate64 source code from pypi and the word "sdist" only appears in a readme file, it isn't in any of the code or config files. |
|
@blathersbugs If you take a look at (which is whereFull dependency closure: {unknown==0.0.0(sdist)} gets logged from), you will see we first download the dependencies using pip (command 208 | calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i from your log). So somewhere in your dependency closure, pip is getting unknown and therefore we are trying to build it.
This looks to me like an issue with your system or dependency closure. |
|
@jfuss |
|
@blathersbugs Can you provide output from The other thing you can try is doing the |
|
@jfuss |
We output "sdist" because we couldn't find a |
|
@jfuss am I correct in assuming that outputting "sdist" is supposed to trigger building the package from source? If that is so, then what I think is happening is it is finding the package sdist https://pypi.org/project/sdist/ that someone uploaded on pypi and attempting to install that package instead. This then fails because that package does not have a compatible version. or at least I think this might have something to do with what is happening |
moelasmar
removed
blocked/more-info-needed
|
@jfuss @mndeveci this is still failing for inflate64==0.1.4 Fetching public.ecr.aws/sam/build-python3.9:latest-arm64 Docker container image..........................................................................................................................................................................................................................................................................................................................................................................................................................267 | 2023-02-10 18:07:34,019 | Mounting /codebuild/output/src174021131/src/lambda/helloworld as /tmp/samcli/source:ro,delegated inside runtime container it looks like that sdist package on pypi is the issue, at least to me with my limited knowledge of python packaging. |
|
@blathersbugs I am not really sure where |
|
requirements.txt |
|
@blathersbugs Got it. Will try to build locally with that to try and repro. Will comment back with more details once I have something |
|
@blathersbugs I tried locally with I starting poking deeper into pip because pip is finding something that it can't resolve as a dependency and returning unknown. SAM CLI/Lambda Builders then think that is a package it needs to handle. I still need to understand deeper but that is how it is looking by some comments in the pip codebase.
You mentioned above that newer versions don't have this issue, is there any reason you can't migrate to a newer version? |
|
More "googling" seems to suggest an outdated version of I think failing for when we see |
|
@jfuss There is nothing stopping me from updating the dependency at this point, the problem is that this issue might still be happening for someone else with another arm package that lacks a wheel, unless we are certain it is only this package. |
|
@blathersbugs I wasn't trying to suggest to close. I think this or something targeting the "unknown" dependency needs to be in the repo until we fully fix it or have better error handling. I was just trying to give details on what is going on (or at least what I think is). I wanted to make sure you are not blocked by this first, which is why I was bringing up the upgrading. I doubt this is only this package, honestly but I don't think this is common either. |
|
I am able to reproduce this issue. |
|
I believe that the source of this issue is the legacy arm64 platforms are missed here |
Description:
python lambda function builds using the sam cli (which calls this library) are still failing for certain dependencies
Steps to reproduce the issue:
Observed result:
build fails on resolve dependencies
Expected result:
build completes
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
aws codebuild
image
aws/codebuild/amazonlinux2-x86_64-standard:3.0
buildspec.yaml:
version: 0.2
env:
secrets-manager:
DOCKER_PASSWORD: *******
DOCKER_USER: ******
phases:
install:
runtime-versions:
python: 3.9
build:
commands:
- printenv
- pip install --upgrade aws-sam-cli
- docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
- echo $DOCKER_PASSWORD | docker login --username $DOCKER_USER --password-stdin
- sam build -u --debug
- sam package --s3-bucket ******i --output-template-file template-out.yaml
artifacts:
files:
- template-out.yaml
sam build -u --debug output near failure:
2022-03-22 18:35:43,680 | Building codeuri: /codebuild/output/src531839338/src/lambda/******** runtime: python3.9 metadata: {} architecture: arm64 functions: ['*****']
2022-03-22 18:35:43,680 | Building to following folder /codebuild/output/src531839338/src/.aws-sam/build****
Fetching public.ecr.aws/sam/build-python3.9:latest-arm64 Docker container image......
2022-03-22 18:35:44,617 | Mounting /codebuild/output/src531839338/src/lambda/****** as /tmp/samcli/source:ro,delegated inside runtime container
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Registering workflow 'NodejsNpmEsbuildBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm-esbuild', application_framework=None)'
Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i
Full dependency closure: {brotli==1.0.9(wheel), pycryptodomex==3.14.1(sdist), texttable==1.6.4(wheel), zipfile-deflate64==0.0.0(sdist), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel)}
initial compatible: {brotli==1.0.9(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel)}
initial incompatible: {pycryptodomex==3.14.1(sdist), zipfile-deflate64==0.0.0(sdist)}
Downloading missing wheels: {pycryptodomex==3.14.1(sdist), zipfile-deflate64==0.0.0(sdist)}
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_aarch64 --implementation cp --abi cp39 --dest /tmp/samcli/scratch pycryptodomex==3.14.1
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_aarch64 --implementation cp --abi cp39 --dest /tmp/samcli/scratch zipfile-deflate64==0.0.0
compatible wheels after second download pass: {brotli==1.0.9(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel)}
Build missing wheels from sdists (C compiling True): {pycryptodomex==3.14.1(sdist), zipfile-deflate64==0.0.0(sdist)}
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/pycryptodomex-3.14.1.tar.gz
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/zipfile-deflate64-0.2.0.tar.gz
compatible after building wheels (no C compiling): {brotli==1.0.9(wheel), pycryptodomex==3.14.1(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel), zipfile-deflate64==0.2.0(wheel)}
Build missing wheels from sdists (C compiling False): {zipfile-deflate64==0.0.0(sdist)}
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/zipfile-deflate64-0.2.0.tar.gz
compatible after building wheels (C compiling): {brotli==1.0.9(wheel), pycryptodomex==3.14.1(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel), zipfile-deflate64==0.2.0(wheel)}
Final compatible: {brotli==1.0.9(wheel), pycryptodomex==3.14.1(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel), zipfile-deflate64==0.2.0(wheel)}
Final incompatible: set()
Final missing wheels: {zipfile-deflate64==0.0.0(sdist)}
PythonPipBuilder:ResolveDependencies failed
Traceback (most recent call last):
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 51, in execute
package_builder.build_dependencies(
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 156, in build_dependencies
self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 261, in build_site_packages
raise MissingDependencyError(packages_without_wheels)
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {zipfile-deflate64==0.0.0(sdist)}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 302, in run
action.execute()
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 57, in execute
raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: {zipfile-deflate64==0.0.0(sdist)}
Builder workflow failed
Traceback (most recent call last):
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 51, in execute
package_builder.build_dependencies(
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 156, in build_dependencies
self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 261, in build_site_packages
raise MissingDependencyError(packages_without_wheels)
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {zipfile-deflate64==0.0.0(sdist)}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 302, in run
action.execute()
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 57, in execute
raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: {zipfile-deflate64==0.0.0(sdist)}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/main.py", line 118, in main
builder.build(
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/builder.py", line 164, in build
return workflow.run()
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 96, in wrapper
func(self, *args, **kwargs)
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 309, in run
raise WorkflowFailedError(workflow_name=self.NAME, action_name=action.NAME, reason=str(ex))
aws_lambda_builders.exceptions.WorkflowFailedError: PythonPipBuilder:ResolveDependencies - {zipfile-deflate64==0.0.0(sdist)}
2022-03-22 18:43:02,875 | Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "PythonPipBuilder:ResolveDependencies - {zipfile-deflate64==0.0.0(sdist)}"}}
Build Failed
2022-03-22 18:43:03,179 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '64785e7d-a17c-4f7f-a347-aca7b7adc804', 'installationId': '75a7108d-f9a0-4aa3-ad9a-b637b8ea0647', 'sessionId': 'c7ec421f-a00b-47f1-b1e6-b34189e99768', 'executionEnvironment': 'AWSCodeBuild', 'ci': True, 'pyversion': '3.9.5', 'samcliVersion': '1.41.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'metricSpecificAttributes': {'projectType': 'CFN'}, 'duration': 667631, 'exitReason': 'BuildInsideContainerError', 'exitCode': 1}}]}
2022-03-22 18:43:03,217 | Telemetry response: 200
Error: PythonPipBuilder:ResolveDependencies - {zipfile-deflate64==0.0.0(sdist)}
[Container] 2022/03/22 18:43:03 Command did not exit successfully sam build -u --debug exit status 1
[Container] 2022/03/22 18:43:03 Phase complete: BUILD State: FAILED
[Container] 2022/03/22 18:43:03 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: sam build -u --debug. Reason: exit status 1
[Container] 2022/03/22 18:43:03 Entering phase POST_BUILD
[Container] 2022/03/22 18:43:03 Phase complete: POST_BUILD State: SUCCEEDED
[Container] 2022/03/22 18:43:03 Phase context status code: Message:
[Container] 2022/03/22 18:43:03 Expanding base directory path: .
[Container] 2022/03/22 18:43:03 Assembling file list
[Container] 2022/03/22 18:43:03 Expanding .
[Container] 2022/03/22 18:43:03 Expanding file paths for base directory .
[Container] 2022/03/22 18:43:03 Assembling file list
[Container] 2022/03/22 18:43:03 Expanding template-out.yaml
[Container] 2022/03/22 18:43:03 Skipping invalid file path template-out.yaml
[Container] 2022/03/22 18:43:03 Phase complete: UPLOAD_ARTIFACTS State: FAILED
[Container] 2022/03/22 18:43:03 Phase context status code: CLIENT_ERROR Message: no matching artifact paths found
The text was updated successfully, but these errors were encountered: