K8SPG-619: restart backup jobs on failure

[pooknull]

https://perconadev.atlassian.net/browse/K8SPG-619

DESCRIPTION

Problem:
The backup pod currently fails on the first attempt, resulting in the creation of a new pod on failure. This behavior may not be reliable in all Kubernetes environments, due to potential delays in establishing communication with the Kubernetes API.

Cause:
The backup job’s restartPolicy is set to Never, preventing the existing pod from retrying after a failure.

Solution:
Add new .spec.backups.pgbackrest.jobs.restartPolicy and .spec.backups.pgbackrest.jobs.backoffLimit fields to the cr.yaml file so that the user can change it to suit their needs.

CHECKLIST

Jira

• Is the Jira ticket created and referenced properly?
• Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• Is an E2E test/test case added for the new feature/change?
• Are unit tests added where appropriate?

Config/Logging/Testability

• Are all needed new/changed options added to default YAML files?
• Are all needed new/changed options added to the Helm Chart?
• Did we add proper logging messages for operator actions?
• Did we ensure compatibility with the previous version or cluster upgrade process?
• Does the change support oldest and newest supported PG version?
• Does the change support oldest and newest supported Kubernetes version?

[hors]

Please do not change the defaults. We need to add the possibility of configuring it, but we need to have the old behavior by default.

[pooknull]

39c7804

[JNKPercona]

Test name	Status
custom-extensions	passed
custom-tls	passed
demand-backup	passed
finalizers	passed
init-deploy	passed
monitoring	passed
one-pod	passed
operator-self-healing	passed
pitr	passed
scaling	passed
scheduled-backup	passed
self-healing	passed
sidecars	passed
start-from-backup	passed
tablespaces	passed
telemetry-transfer	passed
upgrade-consistency	passed
upgrade-minor	passed
users	passed
We run 19 out of 19

commit: d8b4d5d
image: perconalab/percona-postgresql-operator:PR-969-d8b4d5de5