googleapi.com HTTP 503 causes crash

[TMDeal]

Description

The program crashed when run against a GCP environment using a service account json file. It ran a good amount of the way and then crashed unexpectedly when googleapi.com returned an HTTP 503 stating that the "Policy checks are unavailable.".

To Reproduce

This is the command that was run when cartography crashed.

GOOGLE_APPLICATION_CREDENTIALS=./credentials.json cartography --neo4j-uri bolt://localhost:7687 --neo4j-user neo4j --neo4j-password-prompt

Stacktrace

ERROR:cartography.sync:Unhandled exception during sync stage 'gcp'
Traceback (most recent call last):
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/sync.py", line 109, in run
    stage_func(neo4j_session, config)
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/util.py", line 178, in timed
    return method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/intel/gcp/__init__.py", line 250, in start_gcp_ingestion
    _sync_multiple_projects(neo4j_session, resources, projects, config.update_tag, common_job_parameters)
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/intel/gcp/__init__.py", line 208, in _sync_multiple_projects
    _sync_single_project(neo4j_session, resources, project_id, gcp_update_tag, common_job_parameters)
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/intel/gcp/__init__.py", line 177, in _sync_single_project
    compute.sync(neo4j_session, resources.compute, project_id, gcp_update_tag, common_job_parameters)
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/intel/gcp/compute.py", line 1235, in sync
    sync_gcp_subnets(neo4j_session, compute, project_id, regions, gcp_update_tag, common_job_parameters)
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/util.py", line 178, in timed
    return method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/intel/gcp/compute.py", line 1140, in sync_gcp_subnets
    subnet_res = get_gcp_subnets(project_id, r, compute)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/util.py", line 178, in timed
    return method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/intel/gcp/compute.py", line 130, in get_gcp_subnets
    return req.execute()
           ^^^^^^^^^^^^^
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/googleapiclient/http.py", line 938, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 503 when requesting https://compute.googleapis.com/compute/v1/projects/<REDACTED>/regions/us-east5/subnetworks?alt=json returned "Policy checks are unavailable.". Details: "[{'message': 'Policy checks are unavailable.', 'domain': 'global', 'reason': 'backendError'}]">
Traceback (most recent call last):
  File "/root/.local/bin/cartography", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/cli.py", line 709, in main
    sys.exit(CLI(prog='cartography').main(argv))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/cli.py", line 690, in main
    return cartography.sync.run_with_config(self.sync, config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/sync.py", line 187, in run_with_config
    return sync.run(neo4j_driver, config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/sync.py", line 109, in run
    stage_func(neo4j_session, config)
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/util.py", line 178, in timed
    return method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/intel/gcp/__init__.py", line 250, in start_gcp_ingestion
    _sync_multiple_projects(neo4j_session, resources, projects, config.update_tag, common_job_parameters)
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/intel/gcp/__init__.py", line 208, in _sync_multiple_projects
    _sync_single_project(neo4j_session, resources, project_id, gcp_update_tag, common_job_parameters)
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/intel/gcp/__init__.py", line 177, in _sync_single_project
    compute.sync(neo4j_session, resources.compute, project_id, gcp_update_tag, common_job_parameters)
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/intel/gcp/compute.py", line 1235, in sync
    sync_gcp_subnets(neo4j_session, compute, project_id, regions, gcp_update_tag, common_job_parameters)
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/util.py", line 178, in timed
    return method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/intel/gcp/compute.py", line 1140, in sync_gcp_subnets
    subnet_res = get_gcp_subnets(project_id, r, compute)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/util.py", line 178, in timed
    return method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/cartography/intel/gcp/compute.py", line 130, in get_gcp_subnets
    return req.execute()
           ^^^^^^^^^^^^^
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/pipx/venvs/cartography/lib/python3.11/site-packages/googleapiclient/http.py", line 938, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 503 when requesting https://compute.googleapis.com/compute/v1/projects/<REDACTED>/regions/us-east5/subnetworks?alt=json returned "Policy checks are unavailable.". Details: "[{'message': 'Policy checks are unavailable.', 'domain': 'global', 'reason': 'backendError'}]">

Version Info

• cartography: 0.85.3
• Python version: 3.11.5