AWS CLI hangs in proxy mode

[Aurrix]

Hi,

I am trying to make a POC with iamlive, it seems that somehow iamlive proxy results in aws cli hanging on most of the operations.

I have set up env variables as per documentation; iamlive seems to correctly intercept calls:

However, the terminal with aws command hangs without any output. So I have run it with debug mode. It seems to be requests stops at and retries:

2022-03-08 22:49:04,436 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (3): s3.eu-central-1.amazonaws.com:443

Full logs below:

GET
/

host:s3.eu-central-1.amazonaws.com
x-amz-content-sha256:/hidden/
x-amz-date:20220308T204904Z

host;x-amz-content-sha256;x-amz-date
/hidden/
2022-03-08 22:49:04,434 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20220308T204904Z
20220308/eu-central-1/s3/aws4_request
/hidden/
2022-03-08 22:49:04,436 - MainThread - botocore.auth - DEBUG - Signature:
/hidden/
2022-03-08 22:49:04,436 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://s3.eu-central-1.amazonaws.com/, headers={'User-Agent': b'aws-cli/2.4.18 Python/3.8.8 Windows/10 exe/AMD64 prompt/off command/s3api.list-buckets', 'X-Amz-Date': b'20220308T204904Z', 'X-Amz-Content-SHA256': /hidden/', 'Authorization': b'AWS4-HMAC-SHA256 Credential=/hidden//20220308/eu-central-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=/hidden/'}>
2022-03-08 22:49:04,436 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (3): s3.eu-central-1.amazonaws.com:443
2022-03-08 22:50:07,102 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListBuckets: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x0000028AD3E5FB50>>
2022-03-08 22:56:00,193 - MainThread - botocore.retries.standard - DEBUG - Max attempts of 3 reached.
2022-03-08 22:56:00,224 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2022-03-08 22:56:00,225 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListBuckets: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x0000028AD3E5FBB0>>
2022-03-08 22:56:00,225 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "urllib3\connectionpool.py", line 449, in _make_request
  File "<string>", line 3, in raise_from
  File "urllib3\connectionpool.py", line 444, in _make_request
  File "http\client.py", line 1347, in getresponse
  File "http\client.py", line 307, in begin
  File "http\client.py", line 268, in _read_status
  File "socket.py", line 669, in readinto
  File "ssl.py", line 1241, in recv_into
  File "ssl.py", line 1099, in read
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "awscli\botocore\httpsession.py", line 344, in send
  File "urllib3\connectionpool.py", line 785, in urlopen
  File "urllib3\util\retry.py", line 525, in increment
  File "urllib3\packages\six.py", line 770, in reraise
  File "urllib3\connectionpool.py", line 703, in urlopen
  File "urllib3\connectionpool.py", line 451, in _make_request
  File "urllib3\connectionpool.py", line 340, in _raise_timeout
urllib3.exceptions.ReadTimeoutError: AWSHTTPSConnectionPool(host='s3.eu-central-1.amazonaws.com', port=443): Read timed out. (read timeout=60)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "awscli\clidriver.py", line 459, in main
  File "awscli\clidriver.py", line 594, in __call__
  File "awscli\clidriver.py", line 770, in __call__
  File "awscli\clidriver.py", line 901, in invoke
  File "awscli\clidriver.py", line 913, in _make_client_call
  File "awscli\botocore\client.py", line 281, in _api_call
  File "awscli\botocore\client.py", line 595, in _make_api_call
  File "awscli\botocore\client.py", line 615, in _make_request
  File "awscli\botocore\endpoint.py", line 102, in make_request
  File "awscli\botocore\endpoint.py", line 156, in _send_request
  File "awscli\botocore\endpoint.py", line 200, in _do_get_response
  File "awscli\botocore\endpoint.py", line 269, in _send
  File "awscli\botocore\httpsession.py", line 379, in send
botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "https://s3.eu-central-1.amazonaws.com/"

Read timeout on endpoint URL: "https://s3.eu-central-1.amazonaws.com/"

---

Command: aws s3api list-buckets
OS: Windows 11

[iann0036]

Hi @Aurrix,

I'm unsure why this specifically would be happening. Are there specific services that work and some that don't?

[Aurrix]

Not really, it seems entire aws cli hangs. Eg, I used also describe resources and terraform plan.

[iann0036]

Hey @Aurrix,

There shouldn't be any issues with the latest release. Could you confirm you're doing the following:

Window 1:

iamlive --set-ini --mode proxy

Window 2:

set HTTP_PROXY=http://127.0.0.1:10080
set HTTPS_PROXY=http://127.0.0.1:10080
set AWS_CA_BUNDLE=~/.iamlive/ca.pem
aws s3 ls

[Aurrix]

Hi @iann0036 ,

I have again double-checked the configuration on my personal and workstation.

It works perfectly ok on the corporate machine, but I can't understand why it doesn't on my personal laptop.

Besides having different OS Windows 10/11; I only have different authentication mechanisms.

I am using an access key on my personal laptop while saml based authentication on a company-provided workstation.

Everything else seems to be the same including AWS CLI, Python version, and Git Bash configuration.

Could you perhaps suggest the direction of the search?

PS The set up you suggested is the same I have tried.

[iann0036]

Hey @Aurrix,

I'm a little lost for your configuration. I'd suggest looking into whether the port comes up and is listening, whether admin privs help, and perhaps open Wireshark to see what's really going on.

[timblaktu]

@Aurrix, I'm having a similar problem and wondered if you made any progress on this? It sounds like we may have similar goals, In my case, I'm running all the client aws cli calls inside a container running terraform, and using docker compose to orchestrate it all. I may file a separate issue if I don't resolve my connection-through-proxy issue..

[Aurrix]

@timblaktu Unfortunately, my team decided not to proceed with iamlive further. I am afraid I won't be able to help you.

[timblaktu]

Ah, I see. Hope you found some good alternative.

Closing the loop on this, my connection problem was fixed by me setting HTTP_PROXY and HTTPS_PROXY env vars to be identical, both pointing at the single port iamlive listens on, and both using http:.

@iann0036 this would seem a good candidate for closing, since it hasn't been reproducible and the issuer has moved on.