"filters" endpoint leads to "argument of type 'NoneType' is not iterable" exception

[christiankf]

Hi there since the 11-12 of June 2024 I got a consistent error when the Pipedrive integration run during extraction.

Here is the relevant part of the log:

2024-06-17 21:00:28,647Z    tap - CRITICAL argument of type 'NoneType' is not iterable
2024-06-17 21:00:28,648Z    tap - Traceback (most recent call last):
2024-06-17 21:00:28,648Z    tap -   File "tap-env/bin/tap-pipedrive", line 10, in <module>
2024-06-17 21:00:28,648Z    tap -     sys.exit(main())
2024-06-17 21:00:28,648Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/utils.py", line 229, in wrapped
2024-06-17 21:00:28,648Z    tap -     return fnc(*args, **kwargs)
2024-06-17 21:00:28,648Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_pipedrive/cli.py", line 26, in main
2024-06-17 21:00:28,648Z    tap -     pipedrive_tap.do_sync(catalog)
2024-06-17 21:00:28,648Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_pipedrive/tap.py", line 237, in do_sync
2024-06-17 21:00:28,648Z    tap -     self.do_paginate(stream, stream_metadata)
2024-06-17 21:00:28,648Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_pipedrive/tap.py", line 273, in do_paginate
2024-06-17 21:00:28,648Z    tap -     stream.paginate(response)
2024-06-17 21:00:28,648Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_pipedrive/stream.py", line 74, in paginate
2024-06-17 21:00:28,648Z    tap -     if 'additional_data' in payload and 'pagination' in payload['additional_data']:
2024-06-17 21:00:28,648Z    tap - TypeError: argument of type 'NoneType' is not iterable

I had to disable to replicate the filters so the rest runs through properly.

I would expect the following fixes:

1. Handle the error in this line. Looks easy to catch (except if it's broken contract that none of them there should be "None")
2. Add error handling that makes sure that even if one table/endpoint breaks, the others are still run through normally

[RushiT0122]

PR#137 fixes TypeError: argument of type 'NoneType' is not iterable.