[feat][Proposal-2] Support including the topic name to the metadata(#836)

[RobertIndie]

Proposal link: #860

Special release for #816

See #816

• Add new configuration includeTopicToMetadata

(cherry picked from commit 94620e4)

Motivation

Currently, we don't include the topic name to the metadata. And there is not intuitive workaround for it.

This proposal introduces a new configuration includeTopicName to the Cloud Storage sink connector. When
activated(true), the connector will include the topic name to the metadata in the sink file.

Modifications

Introduce a new configuration includeTopicToMetadata to support including the Pulsar topic name into the metadata.

The new data format of the cloud storage format would be like:

{
  "key": "value",
  "__message_metadata__": {
    "messageId": "CAgQADAA",
    "topic": "persistent://public/default/test-s3",
    "properties": {}
  }
}

A new key, topic, would be added to the metadata, containing the Pulsar topic name.

Tests

{"test-message":"test-value","__message_metadata__":{"messageId":"17:0:-1","topic":"persistent://public/default/test-s3","properties":{}}}
{"test-message":"test-value","__message_metadata__":{"messageId":"17:1:-1","topic":"persistent://public/default/test-s3","properties":{}}}
{"test-message":"test-value","__message_metadata__":{"messageId":"17:2:-1","topic":"persistent://public/default/test-s3","properties":{}}}
{"test-message":"test-value","__message_metadata__":{"messageId":"17:3:-1","topic":"persistent://public/default/test-s3","properties":{}}}
{"test-message":"test-value","__message_metadata__":{"messageId":"17:4:-1","topic":"persistent://public/default/test-s3","properties":{}}}
{"test-message":"test-value","__message_metadata__":{"messageId":"17:5:-1","topic":"persistent://public/default/test-s3","properties":{}}}
{"test-message":"test-value","__message_metadata__":{"messageId":"17:6:-1","topic":"persistent://public/default/test-s3","properties":{}}}
{"test-message":"test-value","__message_metadata__":{"messageId":"17:7:-1","topic":"persistent://public/default/test-s3","properties":{}}}
{"test-message":"test-value","__message_metadata__":{"messageId":"17:8:-1","topic":"persistent://public/default/test-s3","properties":{}}}
{"test-message":"test-value","__message_metadata__":{"messageId":"17:9:-1","topic":"persistent://public/default/test-s3","properties":{}}}

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

Documentation

Check the box below.

Need to update docs?

• doc-required

  (If you need help on updating docs, create a doc issue)

• no-need-doc

  (Please explain why)

• doc

  (If this PR contains doc changes)

[github-actions]

@RobertIndie:Thanks for your contribution. For this PR, do we need to update docs?
(The PR template contains info about doc, which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)

[github-actions]

@RobertIndie:Thanks for providing doc info!

[github-actions]

@RobertIndie:Thanks for your contribution. For this PR, do we need to update docs?
(The PR template contains info about doc, which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)