Gatekeeper setup: insecureTLSSkipVerify: true to work with current state alpha and http

[mathieu-benoit]

Signed-off-by: Mathieu Benoit [email protected]

Proposing to have the current state of this repo working with alpha and http by adding the insecureTLSSkipVerify: true parameter.

Here are the errors fixed with that:

• If installing Gatekeeper 3.11 (latest as we speak), getting this error: Support Gatekeeper v3.11 External Data Beta API #20.
• If installing Gatekeeper 3.10, getting this error below:

"manifest/provider.yaml": error when patching "manifest/provider.yaml": admission webhook "validation.gatekeeper.sh" denied the request: only HTTPS scheme is supported for this Provider. To enable HTTP scheme, set insecureTLSSkipVerify to true

Also adding an explicit note and instruction to install Gatekeeper 3.10 to have this project working until #20 is fixed and implemented. Otherwise, nobody can use this project with Gatekeeper 3.11. So this PR is temporary workaround of the Gatekeeper setup.

Important note: even after this setup, there is another issue related to #16 when trying to deploy both scenarios: signed or unsigned containers. But that's not related to what this PR is trying to fix/propose here.

[NiniiGit]

If we can amend README.md sooner, would help as one of my colleague was stuck for hours to understand why it wasn't working at first

[NiniiGit]

Hello @mathieu-benoit / @dlorenc ,
Were you successfully able to verify your images from the private repository using alpha and HTTP and by adding the insecureTLSSkipVerify: true parameter.

In my case, though I am able to install cosign-gatekeeper-provider (with gatekeeper version 3.10 ), I could see in my logs of it (using kubectl logs cosign-gatekeeper-provider-67798c57c8-v85w4 -n cosign-gatekeeper-provider command), its constantly mentioned one of verifier or root certs is required and so it was not able to verify any images I believe.

verify signature for: spotify/backstage:latest
2023/02/07 00:34:19 http: panic serving 10.0.3.172:34976: creating root cert pool: retrieving trusted root; local cache may be corrupt: initializing root client: tuf: failed to decode root.json: encoding/hex: invalid byte: U+002D '-'
goroutine 9 [running]:
net/http.(*conn).serve.func1()
	/usr/local/go/src/net/http/server.go:1801 +0xb9
panic({0x1b9fa40, 0xc000257c20})
	/usr/local/go/src/runtime/panic.go:1047 +0x266
github.com/sigstore/cosign/cmd/cosign/cli/fulcio/fulcioroots.initRoots()
	/go/pkg/mod/github.com/sigstore/[email protected]/cmd/cosign/cli/fulcio/fulcioroots/fulcioroots.go:67 +0x235
github.com/sigstore/cosign/cmd/cosign/cli/fulcio/fulcioroots.Get.func1()
	/go/pkg/mod/github.com/sigstore/[email protected]/cmd/cosign/cli/fulcio/fulcioroots/fulcioroots.go:45 +0x17
sync.(*Once).doSlow(0xc0003ac138, 0x18)
	/usr/local/go/src/sync/once.go:68 +0xd2
sync.(*Once).Do(...)
	/usr/local/go/src/sync/once.go:59
github.com/sigstore/cosign/cmd/cosign/cli/fulcio/fulcioroots.Get()
	/go/pkg/mod/github.com/sigstore/[email protected]/cmd/cosign/cli/fulcio/fulcioroots/fulcioroots.go:44 +0x31
github.com/sigstore/cosign/cmd/cosign/cli/fulcio.GetRoots(...)
	/go/pkg/mod/github.com/sigstore/[email protected]/cmd/cosign/cli/fulcio/fulcio.go:197
main.validate({0x218ee50, 0xc000562000}, 0xc000560300)
	/go/src/github.com/developer-guy/cosign-gatekeeper-provider/provider.go:72 +0x408
net/http.HandlerFunc.ServeHTTP(0x7fabe0ff4aa0, {0x218ee50, 0xc000562000}, 0xc000562000)
	/usr/local/go/src/net/http/server.go:2046 +0x2f
net/http.(*ServeMux).ServeHTTP(0x0, {0x218ee50, 0xc000562000}, 0xc000560300)
	/usr/local/go/src/net/http/server.go:2424 +0x149
net/http.serverHandler.ServeHTTP({0x2182d08}, {0x218ee50, 0xc000562000}, 0xc000560300)
	/usr/local/go/src/net/http/server.go:2878 +0x43b
net/http.(*conn).serve(0xc00041c3c0, {0x2199b40, 0xc000651980})
	/usr/local/go/src/net/http/server.go:1929 +0xb08
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3033 +0x4e8
verify signature for: chanwit/gitops-api:latest
one of verifier or root certs is required
verify signature for: gcr.io/google-samples/kubernetes-bootcamp:v1
one of verifier or root certs is required

It ideally should not create any deployment for any of the unsigned images but in my case, while creation it's not throwing any error but still creating a deployment objects (with no pods) - is it normal?

kubectl get deployments |grep -i error-deployment
error-deployment         0/0     0            0           4h53

[mathieu-benoit]

Hi @NiniiGit, sorry if it was misleading, but I didn't go that far with my tests. This PR is just fixing the Gatekeeper setup part, I just clarified a little bit the description of that PR in order to avoid any confusion.

With that said, I just tested the deployment part and like you, I'm seeing the 2 types of errors in the logs:

• 1st with panic serving 10.0.3.172:34976: creating root cert pool: retrieving trusted root... happening for any first intent to deploy any container --> that's a known issue apparently: Not able to verify container image in my k8s cluster #16.
• 2nd with one of verifier or root certs is required happening after the first intent to deploy any container --> same issue apparently: Not able to verify container image in my k8s cluster #16.

And this error happens for both signed or unsigned containers.

As for the creation of Deployments, you are right, they shouldn't be created, and that's what I see, so not sure to know what's happening on your end. FYI: here is my sequence:

kubectl apply -f policy/template.yaml
# wait until this template/CRD is installed
kubectl apply -f policy/constraint.yaml
# now testing:
kubectl apply -f policy/examples/valid.yaml
# getting this error: Error from server (Forbidden): error when creating "policy/examples/valid.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [cosign-gatekeeper-provider] invalid response: {"errors": [], "responses": [], "status_code": 200, "system_error": "VerifyImageSignatures: one of verifier or root certs is required"
# and verifying that I don't have any deployments in default namespace:
kubectl get deployments -n default
# getting this message: No resources found in default namespace.

[NiniiGit]

Thanks @mathieu-benoit for all the detailed insights and replies! Let me retry the same sequence as you have (to observe behaviour again on my EKS cluster). Meanwhile IMO (yet to test), I think till this issue gets fixed, instead of gatekeeper version v3.10.0, maybe should try version v3.8.0 for gatekeepr+Cosign for image signature validation; as Gatekeeper has hardened the security posture from 3.9 for external data, feature
ref: https://open-policy-agent.github.io/gatekeeper/website/docs/v3.10.x/externaldata/#tls-and-mutual-tls-support
Excerpt:

To further harden the security posture of the external data feature, starting from Gatekeeper v3.9.0, 
TLS and mutual TLS (mTLS) via HTTPS protocol are supported between Gatekeeper and external 
data providers.

[NiniiGit]

Meanwhile, I can confirm that with opa-gatekeeper version v3.10.0, I am also getting the same behaviour as what Mathieu mentioned above for both valid and error.yaml based resources

kubectl apply -f policy/examples/valid.yaml
# getting this error: Error from server (Forbidden): error when creating "policy/examples/valid.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [cosign-gatekeeper-provider] invalid response: {"errors": [], "responses": [], "status_code": 200, "system_error": "VerifyImageSignatures: one of verifier or root certs is required"
# and verifying that I don't have any deployments in default namespace:

[Dentrax]

Thanks!