bug: kcp workload sync command does not work with EKS/AKS/GKE clusters

[siddhi-khanna]

Describe the bug

I have Installed KCP in my EKS cluster with the following version details:
Client Version: v1.25.5
Kustomize Version: v4.5.7
Server Version: v1.24.3+kcp-v0.11.0

After installing, when I am executing the "kubectl kcp workload sync kind --syncer-image ghcr.io/kcp-dev/kcp/syncer:v0.10.0 -o syncer-kind-main.yaml" command in my eks/aks/gke cluster, initially the syncer pod shows that its in running state but after a while its going into error state and the syncer pod is unable to connect to the kind cluster. Also, I can notice that it is looking for kind cluster instead of EKS/AKS/GKE cluster.

ERROR in syncer pod:

I0711 10:03:17.781845 1 syncer.go:83] "syncing" resource-types=[]
I0711 10:03:17.783302 1 syncer.go:79] "starting syncer" syncTarget.workspace="1b3sbf8cq5r4epjn" syncTarget.name="kind"
I0711 10:03:17.783726 1 syncer.go:106] "attempting to retrieve the Syncer virtual workspace URL" syncTarget.workspace="1b3sbf8cq5r4epjn" syncTarget.name="kind"
E0711 10:03:47.785391 1 run.go:74] "command failed" err="Get `\"[https://17********/clusters/1b3sbf8cq5r4epjn/apis/workload.kcp.dev/v1alpha1/synctargets/kind\":]` dial tcp 17********: i/o timeout"

So is there a way I can run the syncer with EKS/AKS/GKE instead of kind cluster?

Steps To Reproduce

1. Create the eks/aks/gke cluster
2. Install kcp
3. Add Syncer

Expected Behaviour

The syncer should be added properly and the syncer pod should be in running state in eks/aks/gke cluster

Additional Context

No response

[jyotimahapatra]

I believe the syncer is installed in the EKS/GKE/AKS cluster, which would indicate its running as a pod on the worker nodes. Going by the error message https://17********/ , the pod is trying to connect to a vpc local address, is that true? I believe there is plumbing missing that does not allow the pod connectivity to 172.**

[mjudeikis]

/transfer-issue contrib-tmc

[mjudeikis]

/transfer-issue contrib-tmc