Add Kafka DLQ Subscriber Preview functionality (GSI-1127)

[TheByronHimes]

🚧 Under Construction! 🚧

KafkaConfig

• Adds one config option, kafka_preview_limit, which is 1 by default.

ExtractedEventInfo

• The type_ assignment used to fall back to the header value, but didn't remove it. The result is that the "type" would be found both in the top-level type_ variable as well as in headers["type"]. Now, the value is popped.

KafkaDLQSubscriber

• Added a preview method that return at most N=kafka_preview_limit events from the configured DLQ topic.
• Moved logic for run(ignore=True) into a new method, ignore().
• Renamed run(ignore=False) to process().
• Removed the subclassing of InboundProviderBase because this is actually a specialized class that has little overlap with the normal subscriber provider class.
• The __init__ function now takes the whole config instance and internally breaks out the values it needs (topics/preview limit)

The preview operation uses AIOKafkaConsumer.getmany with max_records set to the configured preview limit with an arbitrary but reasonable timeout. The point of the timeout is to retrieve the records without hanging forever, since we can't know if or how many records will be available to fetch.

[coveralls]

Pull Request Test Coverage Report for Build 11686548317

Details

• 20 of 20 (100.0%) changed or added relevant lines in 2 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.08%) to 92.601%

---

Totals
Change from base Build 11680285342:	0.08%
Covered Lines:	2040
Relevant Lines:	2203

---

💛 - Coveralls