Failed to get PMM Server parameters from local pmm-agent

[Cosmicoppai]

Description

I'm facing some weird issues on my server.

where the systemd service is running but I still get the following errors

 pmm-admin list
Failed to get PMM Server parameters from local pmm-agent: Post "http://127.0.0.1:7777/local/Status": dial tcp 127.0.0.1:7777: connect: connection refused.
Please use --server-url flag to specify PMM Server URL.

service pmm-agent status
● pmm-agent.service - pmm-agent
     Loaded: loaded (/lib/systemd/system/pmm-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-03-18 17:51:16 UTC; 8 months 24 days ago
   Main PID: 3094 (pmm-agent)
      Tasks: 31 (limit: 77121)
     Memory: 86.0M
     CGroup: /system.slice/pmm-agent.service
             └─3094 /usr/sbin/pmm-agent --config-file=/usr/local/percona/pmm2/config/pmm-agent.yaml

Nov 27 13:06:49 --- pmm-agent[3094]: WARN[2024-11-27T13:06:49.964+00:00] Failed to sen>
Nov 27 13:13:04 --- pmm-agent[3094]: WARN[2024-11-27T13:13:04.330+00:00] ts=2024-11-27>
Dec 11 09:08:37 --- pmm-agent[3094]: WARN[2024-12-11T09:08:37.360+00:00] Failed to sen>
Dec 11 09:08:37 --- pmm-agent[3094]: WARN[2024-12-11T09:08:37.362+00:00] Failed to sen>
Dec 11 09:08:37 --- pmm-agent[3094]: WARN[2024-12-11T09:08:37.362+00:00] Failed to sen>
Dec 11 09:08:37 --- pmm-agent[3094]: WARN[2024-12-11T09:08:37.362+00:00] Failed to sen>
Dec 11 09:08:37 --- pmm-agent[3094]: WARN[2024-12-11T09:08:37.362+00:00] Failed to sen>
Dec 11 09:08:37 --- pmm-agent[3094]: WARN[2024-12-11T09:08:37.363+00:00] Failed to sen>
Dec 11 09:08:37 --- pmm-agent[3094]: WARN[2024-12-11T09:08:37.364+00:00] Failed to sen>
Dec 11 09:08:37 --- pmm-agent[3094]: WARN[2024-12-11T09:08:37.377+00:00] Failed to sen>

Expected Results

 pmm-admin list
Service type        Service name        Address and port        Service ID

Agent type           Status           Metrics Mode        Agent ID                                              Service ID        Port
pmm_agent            Connected                            /agent_id/f9f5775a-867d-4d62-afb2-03fdaf1ada42                          0
node_exporter        Running          push                /agent_id/a38a3fc5-6334-419f-ade1-1718b11b269d                          42001
vmagent              Running          push                /agent_id/2c602700-3fd7-44e6-ac96-15e3c66aa17f                          42000

Actual Results

Alerts are firing

 pmm-admin list
Failed to get PMM Server parameters from local pmm-agent: Post "http://127.0.0.1:7777/local/Status": dial tcp 127.0.0.1:7777: connect: connection refused.
Please use --server-url flag to specify PMM Server URL.

Version

# pmm-admin --version
ProjectName: pmm-admin
Version: 2.31.0
PMMVersion: 2.31.0
Timestamp: 2022-09-23 13:21:08 (UTC)
FullCommit: 86df37b0a6913391ee28b100a46761174052097b

Steps to reproduce

No response

Relevant logs

Nov 27 11:47:35 --- pmm-agent[3094]: WARN[2024-11-27T11:47:35.367+00:00] ts=2024-11-27T11:47:35.367Z caller=node_exporter.go:186 level=warn msg="Node Exporter is running as root user. This exporter is designed to run as unpriviledg>
Nov 27 11:47:47 --- pmm-agent[3094]: WARN[2024-11-27T11:47:47.226+00:00] Failed to send StateChanged request.          component=client
Nov 27 11:47:47 --- pmm-agent[3094]: WARN[2024-11-27T11:47:47.226+00:00] Failed to send StateChanged request.          component=client
Nov 27 11:47:47 --- pmm-agent[3094]: WARN[2024-11-27T11:47:47.226+00:00] Failed to send StateChanged request.          component=client
Nov 27 11:47:47 --- pmm-agent[3094]: WARN[2024-11-27T11:47:47.226+00:00] Failed to send StateChanged request.          component=client
Nov 27 11:47:47 --- pmm-agent[3094]: WARN[2024-11-27T11:47:47.226+00:00] Failed to send StateChanged request.          component=client
Nov 27 11:47:47 --- pmm-agent[3094]: WARN[2024-11-27T11:47:47.228+00:00] Failed to send StateChanged request.          component=client
Nov 27 11:47:47 --- pmm-agent[3094]: WARN[2024-11-27T11:47:47.229+00:00] Failed to send StateChanged request.          component=client
Nov 27 11:47:47 --- pmm-agent[3094]: WARN[2024-11-27T11:47:47.229+00:00] Failed to send StateChanged request.          component=client
Nov 27 12:24:16 --- pmm-agent[3094]: ERRO[2024-11-27T12:24:16.514+00:00] Failed to establish two-way communication channel: failed to send message: EOF.  component=client
Nov 27 13:06:45 --- pmm-agent[3094]: ERRO[2024-11-27T13:06:45.588+00:00] Failed to establish two-way communication channel: Kicked.  component=client
Nov 27 13:06:47 --- pmm-agent[3094]: WARN[2024-11-27T13:06:47.873+00:00] ts=2024-11-27T13:06:47.873Z caller=node_exporter.go:186 level=warn msg="Node Exporter is running as root user. This exporter is designed to run as unpriviledg>
Nov 27 13:06:49 --- pmm-agent[3094]: WARN[2024-11-27T13:06:49.684+00:00] Failed to send StateChanged request.          component=client
Nov 27 13:06:49 --- pmm-agent[3094]: WARN[2024-11-27T13:06:49.684+00:00] Failed to send StateChanged request.          component=client
Nov 27 13:06:49 --- pmm-agent[3094]: WARN[2024-11-27T13:06:49.684+00:00] Failed to send StateChanged request.          component=client
Nov 27 13:06:49 --- pmm-agent[3094]: WARN[2024-11-27T13:06:49.684+00:00] Failed to send StateChanged request.          component=client
Nov 27 13:06:49 --- pmm-agent[3094]: WARN[2024-11-27T13:06:49.685+00:00] Failed to send StateChanged request.          component=client
Nov 27 13:06:49 --- pmm-agent[3094]: WARN[2024-11-27T13:06:49.685+00:00] Failed to send StateChanged request.          component=client
Nov 27 13:06:49 --- pmm-agent[3094]: WARN[2024-11-27T13:06:49.961+00:00] Failed to send StateChanged request.          component=client
Nov 27 13:06:49 --- pmm-agent[3094]: WARN[2024-11-27T13:06:49.961+00:00] Failed to send StateChanged request.          component=client
Nov 27 13:06:49 --- pmm-agent[3094]: WARN[2024-11-27T13:06:49.964+00:00] Failed to send StateChanged request.          component=client
Nov 27 13:13:04 --- pmm-agent[3094]: WARN[2024-11-27T13:13:04.330+00:00] ts=2024-11-27T13:13:04.329Z caller=node_exporter.go:186 level=warn msg="Node Exporter is running as root user. This exporter is designed to run as unpriviledg>
Dec 11 09:08:37 --- pmm-agent[3094]: WARN[2024-12-11T09:08:37.360+00:00] Failed to send StateChanged request.          component=client
Dec 11 09:08:37 --- pmm-agent[3094]: WARN[2024-12-11T09:08:37.362+00:00] Failed to send StateChanged request.          component=client
Dec 11 09:08:37 --- pmm-agent[3094]: WARN[2024-12-11T09:08:37.362+00:00] Failed to send StateChanged request.          component=client
Dec 11 09:08:37 --- pmm-agent[3094]: WARN[2024-12-11T09:08:37.362+00:00] Failed to send StateChanged request.          component=client
Dec 11 09:08:37 --- pmm-agent[3094]: WARN[2024-12-11T09:08:37.362+00:00] Failed to send StateChanged request.          component=client
Dec 11 09:08:37 --- pmm-agent[3094]: WARN[2024-12-11T09:08:37.363+00:00] Failed to send StateChanged request.          component=client
Dec 11 09:08:37 --- pmm-agent[3094]: WARN[2024-12-11T09:08:37.364+00:00] Failed to send StateChanged request.          component=client
Dec 11 09:08:37 --- pmm-agent[3094]: WARN[2024-12-11T09:08:37.377+00:00] Failed to send StateChanged request.          component=client

### Code of Conduct

- [X] I agree to follow Percona Community Code of Conduct

[Cosmicoppai]

does the reconnection attempts are causing the local API to stop working?

[Cosmicoppai]

also when I tried to reconfigured, I got this error

pmm-admin config --server-insecure-tls --server-url=https://user:[email protected] --custom-labels='node_ip=x.x.x.x' --force
Checking local pmm-agent status...
pmm-agent is not running. Please re-run `pmm-agent setup` with --config-file flag.

[BupycHuk]

Hi, seems like you local pmm-agent is not running. and based on your logs seems like it's failing to connect to PMM Server.
Can you try to run pmm-agent setup --server-address=host:port --server-username=user --server-password=password --config-file=/usr/local/percona/pmm2/config/pmm-agent.yml?

[BupycHuk]

another option is to delete /usr/local/percona/pmm2/config/pmm-agent.yml, restart systemd pmm-agent service and try to reconfigure using pmm-admin config.

[Cosmicoppai]

@BupycHuk I did reconfigure the pmm-agent, but it's happening frequently where the systemd service seems fine, but the pmm-agent throws error. Want to understand the cause of it

[BupycHuk]

@Cosmicoppai can you upgrade to the latest version and check again?