"icinga2 object list" does not reflect changes made via API

[oxzi]

Describe the bug

Changes performed via the Icinga 2 API on an object defined from config are not reflected in the icinga2 object list output.

To Reproduce

I am using an IcingaDB object for this example, as this is were I came across this bug.

Starting with a default IcingaDB object as:

$ cat /etc/icinga2/features-enabled/icingadb.conf
object IcingaDB "icingadb" {
  host = "redis-master1"
  //port = 6380
  //password = "xxx"
}

Then, perform a (questionable) change via the API. This will only affect the connection after a restart, see #10102.

$ curl -k -s -S -u root:icinga -H 'Accept: application/json' -X POST 'https://localhost:5665/v1/objects/IcingaDBs/icingadb' -d '{"attrs": {"port": 9999}, "pretty":
true}'

{
    "results": [
        {
            "code": 200,
            "name": "icingadb",
            "status": "Attributes updated.",
            "type": "IcingaDB"
        }
    ]
}

Check the state…

• via icinga2 objects list

$ icinga2 daemon -C --dump-objects
[snip]
$ icinga2 object list --type icingadb | grep port
  * port = 6380

• via and the API.

$ curl -k -s -u root:icinga -H 'Accept: application/json' -H 'X-HTTP-Method-Override: GET' 'https://localhost:5665/v1/objects/icingadbs' -d '{"pretty": true}'
{
    "results": [
        {
            "attrs": {
                "__name": "icingadb",
                [snip]
                "original_attributes": {
                    "port": 6380
                },
                [snip]
                "port": 9999,
        }
    ]
}

While the Icinga 2 API represents the changed port as well as the original_attributes field of the Common Runtime Attributes, the output of icinga2 object list is outdated.

Neither CLI command: Object nor Configuration Troubleshooting, List Configuration Objects document this behavior as a limitation.

Expected behavior

I would expect icinga2 object list to show the updated port attribute.

Screenshots

N/A

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Version used (icinga2 --version): v2.14.0-293-g4f690a6fd
• Operating System and version: Debian GNU/Linux 11
• Enabled features (icinga2 feature list): api checker icingadb notification
• Icinga Web 2 version and modules (System - About): N/A
• Config validation (icinga2 daemon -C): ☑️
• If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.

Additional context

N/A

[yhabteab]

Check the state…

• via icinga2 objects list

$ icinga2 daemon -C --dump-objects
[snip]
$ icinga2 object list --type icingadb | grep port
  * port = 6380

How long did you wait after changing the attribute before dumping the objects with ... --dump-objects? The icinga2 daemon -C --dump-objects command does not dump the in memory state of the objects, but reads it from the individual configuration files. However, the /var/lib/icinga2/modified_attributes.conf is only flushed every 5m, so you must at least wait for the Dumping program state to file ... log entry after the API request, which indicates that the modified attributes and the Icinga 2 state files have actually been flushed.

[oxzi]

How long did you wait after changing the attribute before dumping the objects with ... --dump-objects?

I initially realized this after restarting Icinga 2, as the changed did not take effect during the prior runtime (as stated above). Afterwards, I played a bit with the API and so on, which should have taken longer than five minutes, but I cannot say so with certainty.

To test it, I just gave it another shot.

1. I updated the port again.

   [2024-09-23 11:52:54 +0000] information/HttpServerConnection: Request POST /v1/objects/IcingaDBs/icingadb (from [::ffff:172.24.0.1]:58248, user: root, agent: curl/8.7.1, status: OK) took 0ms.

2. For the moment, both the API and icinga2 object list behave as documented in the initial post.
3. Sipping coffee while waiting :-)
4. Icinga 2 is dumping the program state.

   [2024-09-23 11:57:33 +0000] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'

5. icinga2 object list still shows the outdated port:

   $ icinga2 daemon -C --dump-objects
   [snip]
   
   $ icinga2 object list --type icingadb
   Object 'icingadb' of type 'IcingaDB':
     % declared in '/etc/icinga2/features-enabled/icingadb.conf', lines 1:0-1:25
     * __name = "icingadb"
   [snip]
     * port = 6380
   [snip]
   

   Btw, the modified-attributes.conf reflects my change:

   $ cat /var/lib/icinga2/modified-attributes.conf
   var obj = get_object("IcingaDB", "icingadb")
   if (obj) {
           obj.modify_attribute("port", 9999.000000)
           obj.version = 1727092374.937358
   }
   

Thus, it seem like icinga2 object list ignores the modified-attributes.conf, at least in this case.

[yhabteab]

5. icinga2 object list still shows the outdated port:

Ah, you actually have to dump the objects again here with the --dump-objects flag :).

[oxzi]

5. icinga2 object list still shows the outdated port:

Ah, you actually have to dump the objects again here with the --dump-objects flag :).

Sorry, I pasted the icinga2 object list --type icingadb command twice. The first command was icinga2 daemon -C --dump-objects. I have edited the code block in the previous comment.

[oxzi]

Just had another idea and gave it another shot.

$ curl -k -s -S -u root:icinga -H 'Accept: application/json' -X POST 'https://localhost:5665/v1/objects/IcingaDBs/icingadb' -d '{"attrs": {"port": 9999}, "pretty": true}'
{
    "results": [
        {
            "code": 200,
            "name": "icingadb",
            "status": "Attributes updated.",
            "type": "IcingaDB"
        }
    ]
}

The Icinga 2 log honors this change:

[2024-09-23 12:17:57 +0000] information/HttpServerConnection: Request POST /v1/objects/IcingaDBs/icingadb (from [::ffff:172.24.0.1]:52684, user: root, agent: curl/8.7.1, status: OK) took 0ms.

Having a look at the /var/cache/icinga2/icinga2.debug file:

$ stat /var/cache/icinga2/icinga2.debug
[snip]
Change: 2024-09-23 12:12:09.980045950 +0000
[snip]

$ grep -o '"port":6380' /var/cache/icinga2/icinga2.debug
"port":6380

Now, Icinga 2 has dumped its state again:

[2024-09-23 12:22:33 +0000] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'

Let's verify:

$ icinga2 daemon -C --dump-objects
[snip]

$ stat /var/cache/icinga2/icinga2.debug
[snip]
Change: 2024-09-23 12:23:11.318267722 +0000
[snip]

$ grep -o '"port":6380' /var/cache/icinga2/icinga2.debug
"port":6380
$ grep -o '"port":9999' /var/cache/icinga2/icinga2.debug
$ grep -o ':9999' /var/cache/icinga2/icinga2.debug

So.. the port is not changed after the state dump in this file.

[yhabteab]

It seems to be a bug as the modified attributes are restored while activating the objects and not committing them and the icinga2 daemon -C --dump-objects does only the later.. See #8699

[Al2Klimov]

It seems to be a bug as the modified attributes are restored while activating the objects and not committing them

• I think modified-attributes.conf: store attributes to modify in Internal.modified_attributes #8155 should fix exactly this due to 689f280