Reconciliation stuck because of Sealed Secrets

[jmsariron]

Describe the bug

We are having this issue, where, not sure how exactly, but sometimes the Flux reconciliation is failing because of Sealed Secrets, which are (As far as I know) correct.

Kustomization gets stuck in Reconciliation in Progress, getting the details I get a:

Warning  HealthCheckFailed  19m   kustomize-controller  health check failed after 59m30.097886897s: timeout waiting for: [SealedSecret/myapp/myapp-secrets status: 'InProgress'] 

Checking the kustomize-controllerpod logs, it shows a server-side apply completed with all elements being unchanged and then the Reconciation Failed log message, I'm kinda confused.

Going to the SealedSecret on the Namespace it shos as correctly applied and Synced, with the corresponding regular Secret generated.

This is happening only sometimes and we can't figure out why. Since I can't reproduce it 100% of time I'm looking for some help to debug this behaviour.

Steps to reproduce

We are using SealedSecrets on other projects and AFAIK the same way, sometimes it fails sometimes not, so I don't know how to reproduce it exactly

Expected behavior

I should reconcile just fine

Screenshots and recordings

No response

OS / Distro

N/A

Flux version

N/A

Flux check

N/A

Git provider

No response

Container Registry provider

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[jmsariron]

Adding a bit more info. for this case particular I:

1. Deleted the Sealed Secret inside Kubernetes
2. Deleted the kustomize-controller pod

When the new kustomize-controller started syncing everything, it fixed the kustomization and now appears as Synced with the Sealed Secret recreated from the git repo, so it seems to be something related to he controller?

[fculpo]

Hi, same issue here, restarting sealed secret pod also solve this.
Any idea ?

[stefanprodan]

SealedSecret does now comply with the kstatus standard condition (Ready=true), to solve this we'll need to implement #4528 and make use of CEL to define a custom health check that looks at the Synced status.