Connector/Source: Publish the *request* object in the record transformation like the config, stream_slice, stream_interval, etc. are

[rpopov]

Discussed in #49971

^{Originally posted by rpopov December 20, 2024}
Status in Airbyte 1.2.0
The record transformation allows removing existing fields from the record and adding new fields by calculating them using:

• JINJA templates
• pre-defined objects found in the transformation's context:
  • record
  • stream_state
  • stream_slice
  • stream_interval
  • stream_partition
• the access to the response raw data is redirected through the record object, which by definition is only part of the response. Converting the response into valuable records sometimes it needs data beyond the scope of the record object, which is not accessible.

Example
The JIRA /issue response:

{
  "expand": "renderedFields,names,schema,operations,editmeta,changelog,versionedRepresentations",
  "id": "2759961",
  "self": "https://jira-test.paysafe.cloud/rest/api/2/issue/2759961",
  "key": "EF-3131",
  "fields": {
    "customfield_19440": {
      "self": "https://jira-test.paysafe.cloud/rest/api/2/customFieldOption/23882",
      "value": "Perú",
      "id": "23882",
      "disabled": false
    },
...
    "customfield_12073": "EF-3130"
  },
  "names": {
    "customfield_19442": "End Time",
...
    "customfield_17709": "BU Legal Lead"
  },
  "schema": {
    "customfield_19442": {
      "type": "date",
      "custom": "com.atlassian.jira.plugin.system.customfieldtypes:datepicker",
      "customId": 19442
    },
...
    "customfield_17709": {
      "type": "string",
      "custom": "com.atlassian.jira.plugin.system.customfieldtypes:textfield",
      "customId": 17709
    }
  },
}
]

In JIRA the list of custom fields is highly dynamic, and it makes no practical sense to have a single record per issue, combining in it all standard and custom fields.
Instead, the JIRA issue can be represented using 2 tables: ISSUE 1 --- * CUSTOM_FIELD in one-to-many / master-detail relation.
Turning the response into a list of records, one per custom field, could be done by iterating over the schema.* sub-list in the response, but taking their values from the fields map and taking their human-readble names from the name map.

Problem
In this configuration, the fields and names maps are not accessible. They could be, if the response object were available in the Tranformations' context. The response object exists in the context of the Pagination section.

Suggestion

• Publish the response object in the context of the Transformations section/phase, as available in the Pagination section.