node_is_alive returns unknown when patroni is not reachable

[MLyssens]

Hi,

When I stop patroni on a node and run node_is_alive the check returns unknown:

check_patroni -vvv -e http://x.x.x.x:8008 node_is_alive
DEBUG - Trying to connect to http://x.x.x.x:8008/liveness with cert: None verify: None
DEBUG - Starting new HTTP connection (1): x.x.x.x:8008
DEBUG - HTTPConnectionPool(host='x.x.x.x', port=8008): Max retries exceeded with url: /liveness (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4205724790>: Failed to establish a new connection: [Errno 111] Connection refused'))
NODEISALIVE UNKNOWN - Connection failed for all provided endpoints
unknown: Connection failed for all provided endpoints

When it tries to connect and it fails I would suspect it gives a critical:

connect(3, {sa_family=AF_INET, sin_port=htons(8008), sin_addr=inet_addr("x.x.x.x")}, 16) = -1 ECONNREFUSED (Connection refused)
close(3)                                = 0
alarm(0)                                = 2
write(1, "NODEISALIVE UNKNOWN - Connection failed for all provided endpoints\nunknown: Connection failed for all provided endpoints\n", 121NODEISALIVE UNKNOWN - Connection failed for all provided endpoints
unknown: Connection failed for all provided endpoints
) = 121

[blogh]

Hi!

Sorry for the wait and thank you for the patch/report.

The actual value is set on purpose, because:

• it could also be a configuration error which should be matched to UNKNOWN according to nagios;
• I think remember checking other checks like check_pgactivity who does the same for similar situations.

[blogh]

I recognise that given the probe's description, it's not obvious ..

Check if the node is alive ie patroni is running. This is a liveness check as defined in Patroni's documentation.

My Idea was to map to the /liveness check: 200 = OK, <>200 cirtical, failed connection UNKNOWN.

GET /liveness: returns HTTP status code 200 if Patroni heartbeat loop is properly running and 503 if the last run was
more than ttl seconds ago on the primary or 2*ttl on the replica. Could be used for livenessProbe.