Increase the amount of secrets supported by Secret provider

[InbalZilberman]

Feature Overview & Customer Need
We would like to increase the amount of secrets supported by Secret provider.
Secret provider should support in minimum 550 secrets as secrets can be PAS accounts which means 2750 conjur variables.

As Secret provider supports multiple apps we need to provide as much secrets as possible to support as much applications hence minimise the variable failures. If some variables managed to be fetch and some did not then we need to have a log stating what vars have failed and the status of the secret provider pod should be reported as “partly successful” and if not possible failed in probes. A summary log should appear at the end of Secret provider initialization how many vars were successfully retrieved and how many failed. For those who failed lets write what k8s secrets they affect.

For example- secret provider is serving 3 apps A, B & C that uses K8s secrets A, B & C respectively
In K8s Secret A 30 Conjur secrets are mapped
In K8s Secret B Conjur secret Y and X are mapped
In K8s Secret C other 30 Conjur secrets are mapped

Secret provider initiates and tries to retrieve all 62 secrets all successful but secret Y. Fetching secret Y has failed for any reason (for example it was erased) hence K8s secret A, C & B were updated yet K8s secret B has only have key for secret X and not Y.
Secret provider finish with status “partly successful” and if not possible failed in probes.
A summary log should be written in error level: "Secret provider retrieved 61 variables. one variable X has failed to be retrieved and updated in k8s secret B"

Process Logic \ XDD

1. Martin defined Secret provider host and provided it permissions on 550*5 = 2750 variables.
2. Liz defined K8s secrets and mapped these variables into K8s secrets using conjur map
3. Secret provider initiated and accumulate the k8s secrets with the DAP/Conjur variables values.

Secret provider Documentation
As part of this Epic we will need to create a formal documentation that describes that the amount of variables Secret provider can support (2750 variables or more ) and the test we performed to get to it.

Moreover, SLA is shared in our official documentation == readme file of secret provider and online help
Examples for SLA sharing: https://docs.cyberark.com/Product-Doc/OnlineHelp/AAM-DAP/Latest/en/Content/Conjur/cv_limitations.htm?tocpath=Integrations%7CCyberArk%20Vault%20Synchronizer%7C_____14#GeneralVaultSynchronizerlimitations

Assumptions:

• Variable ID length (similar to synchronizer) 47
• 1 Secret Provider
• Same condition as our latest tests

DOD

• Delivery Plan
• Implement a way to increase the amount of secrets supported by Secret provider
• Test plan written and reviewed by PO & QAA
• Security review was done and issues were raised
• Performance Documentation is created and SLA is shared in our official documentation

[sigalsax]

@oburstein-hub @InbalZilberman Regarding performance, I would like to get more insight on the Conjur-side because this will impact the results we get from our tests.
For example:

1. Are we running OSS or DAP?
2. How many Followers?
3. Are Applications + Followers in same namespace

@InbalZilberman As we saw from this discussion we might need to increase the secret ID

[sigalsax]

At the beginning of the doc the following is written:

Secret provider should support in minimum 550 secrets as secrets can be PAS accounts.

and then at the middle under Process Logic \ XDD

Martin defined Secret provider host and provided it permissions on 550*5 = 2750 variables.

@InbalZilberman is 550 550 K8s Secrets or Conjur secrets? We have a couple of numbers here so should we be aiming to support 550 or 2750?

[sigalsax]

@InbalZilberman as discussed with @Tovli please provide a link to a performance doc template that would help me structure the performance tests according to the requirements