Add a multi_value_processor plugin to the tide_data_pipeline module to handle multiple values.

[vincent-gao]

Jira

https://digital-vic.atlassian.net/browse/SDPAP-9391

Change

1. This PR adds a new DatasetTransform plugin called multi_value_processor. The plugin splits delimited string into arrays and optionally applies a callback function to each resulting value.
2. This PR introduces a custom ElasticSearch destination plugin (TideElasticSearchDestination) that extends the base ElasticSearchDestination plugin. The main improvements are:
   • Flexible index_id generation:
     • Adds support for a hash prefix in addition to the existing prefix.
     • Implements a new getFullIndexId() method that generates the index_id based on the presence of hash prefix and regular prefix.
   • Custom configuration:
     • Adds a new 'hash_prefix' configuration option.
   • Improved cleanup process:
     • Updates the processCleanup() method to use the new index_id generation logic. more details see the conversation from the Slack

Related

This change has applied to https://github.com/dpc-sdp/content-solar-vic-gov-au/pull/148, just for a reference

Note on implementation approach

We've implemented this as a custom plugin rather than submitting a patch to the data_pipelines module. The hash code prefix functionality is specific to SDP requirements and lacks the generality needed for its upstream.

Example

pipeline_with_str_replace:
  label: 'Multiple Value Processor with str_replace'
  transforms:
    field:
      Suburbs:
        - plugin: multi_value_processor
          separator: ';'
          callback: str_replace
          parameters:
            - 'a'
            - 'A'
          value_position: 2

pipeline_with_substr:
  label: 'Multiple Value Processor with substr'
  transforms:
    field:
      Suburbs:
        - plugin: multi_value_processor
          separator: ';'
          callback: substr
          parameters:
            - '0'
            - 'true'

[vincent-gao]

All unit/kernal tests have passed. The failure is only because our DevOps team is debugging why phpdbg doesn't work.

[anthony-malkoun]

We've discussed the approach so happy to approve this. Good find with the delete issues.

[anthony-malkoun]

@vincent-gao are we doing anything about the builds failing and removing the circle jobs?

[vincent-gao]

@vincent-gao are we doing anything about the builds failing and removing the circle jobs?

Yes, currently, we are doing a mono-repo, should we remove the .circle directory in the mono-repo?
and I will also shut down the coverage test for now until Guy fixes the phpdbg issue