enhancement(http sink): Retrying the HTTP sink in case of 404s and request timeouts

[noble-varghese]

Adding the functionality to retry the log sending on getting a not found error(404), forbidden request (403) or a request timeout (408) from the HTTP sink. Since the defualt behavior is to drop the events during such events, this would affect the log propagation.

solves: #10870

[bits-bot]

All committers have signed the CLA.

[pront]

Hi @noble-varghese, thank you for the contribution. I think retrying for these codes is reasonable, I will take a closer look tomorrow.

Note: Need to update the docs as well. See https://vector.dev/docs/reference/configuration/sinks/http/#retry-policy.

[noble-varghese]

@pront Thanks for pointing it out ! I have updated the docs page and added unit tests on the http utils as well :)

[pront]

Thanks!

[pront]

Hey @noble-varghese, I noticed the E2E test http_to_http failed. This change actually is a breaking one. Since it is hard to evaluate the risk, it is better to err on the side of caution. After some discussion with the team, I created #21469 to layout a solution for this.

[pront]

To unblock this PR, perhaps we can add 408 to the defaults? Still a breaking change, but seems desirable and low risk. cc @jszwedko