Duplicate Circular emails distributed

[jracusin]

Description

Some circulars are being distributed via email multiple times. Some recent examples are Circs: 38023, 38021, 38016, 38009, 37989, 37987, 37980, 37979, 37963, 37962.

Similar issue in #924, but that seemed to have resolved the problem, at least temporarily.

Acceptance Criteria

Distribute only 1 copy of every circular to subscribed users.

[lpsinger]

Similar issue in #924, but that seemed to have resolved the problem, at least temporarily.

The root cause is probably not related to #924.

[lpsinger]

The pattern that I see is that duplicate GCN Circulars emails correspond to invocations of the circulars table stream lambda that log the following errors:

2024-12-04T13:47:18.740Z	630c00d0-3149-4ca2-8fda-ac8b0ea15924	ERROR	{
    "level": "ERROR",
    "timestamp": "2024-12-04T13:47:18.740Z",
    "logger": "kafkajs",
    "message": "[BrokerPool] outgoing request timed out after 3500ms",
    "retryCount": 0,
    "retryTime": 287
}
2024-12-04T13:47:18.741Z	630c00d0-3149-4ca2-8fda-ac8b0ea15924	ERROR	Invoke Error 	{
    "errorType": "Error",
    "errorMessage": "oy: outgoing request timed out after 3500ms",
    "stack": [
        "Error: oy: outgoing request timed out after 3500ms",
        "    at Object.intoError (file:///var/runtime/index.mjs:46:16)",
        "    at Object.textErrorLogger [as logError] (file:///var/runtime/index.mjs:684:56)",
        "    at postError (file:///var/runtime/index.mjs:801:27)",
        "    at done (file:///var/runtime/index.mjs:833:11)",
        "    at fail (file:///var/runtime/index.mjs:845:11)",
        "    at file:///var/runtime/index.mjs:872:20"
    ]
}

Probably Lambda is retrying the failed invocations, hence the duplicate emails.

[lpsinger]

The outgoing request timed out after 3500ms is being reported by KafkaJS, but it is originating inside node-openid-client, here: https://github.com/panva/openid-client/blob/v5.6.1/lib/helpers/request.js#L137

So the authentication is timing out.

[lpsinger]

I think that #2754 helped, but I recently got two copies of https://gcn.nasa.gov/circulars/38476. Here's the server log:

2024-12-09T05:56:29.521Z	6cb876d0-b08b-4341-9fcd-8a9a69c783f8	ERROR	Invoke Error 	{
    "errorType": "Error",
    "errorMessage": "ThrottlingException: Rate exceeded",
    "stack": [
        "Error: ThrottlingException: Rate exceeded",
        "    at Object.intoError (file:///var/runtime/index.mjs:46:16)",
        "    at Object.textErrorLogger [as logError] (file:///var/runtime/index.mjs:684:56)",
        "    at postError (file:///var/runtime/index.mjs:801:27)",
        "    at done (file:///var/runtime/index.mjs:833:11)",
        "    at fail (file:///var/runtime/index.mjs:845:11)",
        "    at file:///var/runtime/index.mjs:872:20",
        "    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
    ]
}

It's hard to tell what is timing out because the stack trace is referring to minified source.