Cruise Control with MSK - Not able to remove broker due to capacity limit

[danielap-ma]

Hello,

I've connected Cruise Control to my MSK cluster, and the metrics are drawn from Prometheus.
When I try to disconnect a broker, no matter which hard goal I set, the operation fails due to insufficient capacity. Example:

ERROR: Error processing POST request '/remove_broker' due to: 'com.linkedin.kafka.cruisecontrol.exception.OptimizationFailureException: [DiskCapacityGoal] Utilization (137393.11) of broker 2 violated capacity limit (-0.99) for resource disk. Add at least 1 broker for disk. Add at least 1 broker for disk.'.

The disk usage is less than 140GB, and the broker disk size is 310GB.

Even when the usage is 0 like with Network Outbound metrics, I get the same error:

ERROR Error processing POST request '/remove_broker' due to: 'com.linkedin.kafka.cruisecontrol.exception.OptimizationFailureException: [NetworkOutboundCapacityGoal] Utilization (0.00) of host b-2.XXXXX.kafka.us-east-1.amazonaws.com violated capacity limit (-0.80) for resource networkOutbound. Add at least 1 broker for networkOutbound. Add at least 1 broker for networkOutbound.'. (com.linkedin.kafka.cruisecontrol.KafkaCruiseControlRequestHandler)

Any idea what could be the issue here?

Thank you