Add getLastChange, getLastUpdate, and getPreviousState to GenericItem

[jimtng]

These methods are handy in many situations and isn't possible with mapdb persistence, so previously it would require something like influxdb.

It also solves the issue of needing to wait until persistence write operation is finished (if it was at all set up to persist on change), so this gives us a cheap and reliable / dependable way of getting the last change / last update time.

See discussions in
https://community.openhab.org/t/persistence-on-off-transitions-with-influx/156271/
https://community.openhab.org/t/blockly-persistence-last-changed-returns-no-result-quite-often/157842

[jimtng]

ping @mherwege @rkoshak

[rkoshak]

I like the idea of this information being part of the Item. Of course it won't go back beyond OH startup but something is better than nothing. It certainly handles this use car better than persistence I think given all the different ways perspective can be configured.

How does this interact with restoreOnStartup? If an item is restored and not changed does this return the restored time or null? What makes the most sense? I'm not sure I know (it's close to bed time here so maybe I'll sleep on it).

[jimtng]

How does this interact with restoreOnStartup? If an item is restored and not changed does this return the restored time or null?

It is not aware of persistence restoration, so I guess once persistence restores the state, it will return the time the state was restored.

[spacemanspiff2007]

Imho it should be possible to query these values from the db so restore on startup should work as expected.
Will these fields also be part of the RestAPI / Websockets response?

[jimtng]

Imho it should be possible to query these values from the db so restore on startup should work as expected.

@spacemanspiff2007 would you mind elaborating this please?

Will these fields also be part of the RestAPI / Websockets response?

Good point. I will add them there too.

[spacemanspiff2007]

It's possible to query the last persisted state from persistence and subsequently restore this state as the current state on startup. It's also possible to query the persisted state before that.
Subsequently it should be possible (imho) to get all the values from the persistence service so restore on startup should also work for these fields as expected.

[jimtng]

It's possible to query the last persisted state from persistence and subsequently restore this state as the current state on startup.

Yes, isn't this what restoreOnStartup does?

It's also possible to query the persisted state before that.
Subsequently it should be possible (imho) to get all the values from the persistence service so restore on startup should also work for these fields as expected.

I don't quite understand the above statements.

This PR doesn't interfere with persistence's restoreOnStartup. If I understand @rkoshak's question correctly, it is a question of whether getPreviousState should return (java) null as is the case when there was no previous state being set on the item, or whether it should return UnDefType.NULL because that's the item's state before restoration. Currently it's the latter.

[spacemanspiff2007]

I think it's a misunderstanding.
All I am saying is that it's technically possible to restore these new fields from persistence so imho it makes sense to do so during restoreOnStartup.

[jimtng]

@spacemanspiff2007 I've added it to the REST response

{
  "link": "http://192.168.1.10:8484/rest/items/TestSwitch1",
  "state": "ON",
  "previousState": "NULL",
  "lastUpdate": 1723872081231,
  "lastChange": 1723871965049,
  "editable": false,
  "type": "Switch",
  "name": "TestSwitch1",
  "tags": [],
  "groupNames": [
    "TestSwitches"
  ]
}

[mherwege]

All I am saying is that it's technically possible to restore these new fields from persistence so imho it makes sense to do so during restoreOnStartup.

That depends on the persistence service. And it is not the case for mapDB, the most used persistence service for restoreOnStartup.

I would argue that, when the item state is updated as a consequence of restoreOnStartup, it should not update these lastChange and lastUpdate fields because now is probably the wrong time. But I am not sure this is feasible. I doubt the item knows anything about the source of the update/change.

In general, I think having the extra fields is a good idea. I just feel like we are still not there to give an answer to a simple question: when did my item last change?
If I use this new function (lastChange):

1. Item did changed since startup -> OK, correct answer, and better then what I could get from persistence as persistence could miss the everyChange strategy or could have compressed
2. Item did not change since start -> now, which is wrong
3. If I can figure out the answer is wrong and I have a persistence service with everyChange: call lastChange on the service ->
   a. Current state equal to last state in DB -> correct answer from persistence (ignoring for now impact of compression)
   b. Current state different from last state in DB -> no answer (null)
   c. If no answer, you could return now in the calling rule, although in this case the startup time might be a more appropriate approximation

In an ideal world, mapDB would just store the lastChange time of an item with the state and we could restore that together with the state. I am just not sure it is easily feasible.
Thinking out loud here: As mapDB is kind of a special case which is just used for restoreOnStartup, would it make sense to remove restoreOnStartup from persistence all together and make it part of the core infrastructure, just to take care of this? I would argue you should then persist all items in mapDB all the time, invisible to the user configuration, and use only that special DB in startup to restore item states and their accompanying update dates.

We can keep adding layers to this, but there are always issues and exceptions. I like the idea of keeping lastChange and lastUpdate with the item. But I think the real simplification comes when we have a simpler restore mechanism on startup. Because than, you always get the right answer. Other persistence services then have the focus on data for graphing and calculations, which makes sense in my view.

[jimtng]

OK, I think I now understand what @spacemanspiff2007 was alluding to, thanks to @mherwege's explanation.

Thinking out loud here: As mapDB is kind of a special case which is just used for restoreOnStartup, would it make sense to remove restoreOnStartup from persistence all together and make it part of the core infrastructure, just to take care of this? I would argue you should then persist all items in mapDB all the time, invisible to the user configuration, and use only that special DB in startup to restore item states and their accompanying update dates.

Not a bad idea.

[rkoshak]

In an ideal world, mapDB would just store the lastChange time of an item with the state and we could restore that together with the state. I am just not sure it is easily feasible.

Why would this not be feasible? I beleive all the apis already exists in persistence to get the time from persistence. Once you have that it's just a matter of adding the last change and last update time as a parameter on the Item.

Thinking out loud here: As mapDB is kind of a special case which is just used for restoreOnStartup,

I'm not sure we can make that case that's it's always only used for restoreOnStartup. It's useable if you only care about the most recent change to the Item in a rule (given it's current default strategies). One could change the strategies and it would work for most recent update.

But if these properties are added to the Item and we go down a path where those are made available and accurate on the the Item this use case is handled.

I would argue you should then persist all items in mapDB all the time, invisible to the user configuration, and use only that special DB in startup to restore item states and their accompanying update dates.

Be careful here.

Not all uses want all Items restored.

Not all users want all Items restored to their most recent change or update (e.g they use a rule with .persist to control when the state gets saved).

These use cases are now currently supported. Any move of restoreOnStartup to core would need to handle these use cases too. How the Item's state is saved and which Items are restored needs to be configurable on an Item by Item basis like it is now in persistence.

[spacemanspiff2007]

That depends on the persistence service. And it is not the case for mapDB, the most used persistence service for restoreOnStartup.

It would be trivial to modify mapDB persistence service to save the additional three values. A quick glance at the docs shows it should be possible. If not it's always possible to name mangle additional values with a separator that's not a valid item name (e.g. MyItem#lastUpdate)
The question is - as you correctly pointed out - if that's something that results in a meaningful behavior.

2. Item did not change since start -> now, which is wrong

No - it should return null because it's undefined.

These use cases are now currently supported. Any move of restoreOnStartup to core would need to handle these use cases too. How the Item's state is saved and which Items are restored needs to be configurable on an Item by Item basis like it is now in persistence.

I strongly agree. For many devices that report the state themselves I explicitly do not save the item state so I don't have the wrong state after startup. If the restore makes sense depends strongly on the device, the user and how the rules are written.
I for once was struggling with a rule and strange startup behavior during testing and the root cause was rrd4j restoring states during startup which was automatically configured (rrd4j).

---

Just a hint:
HABApp already provides the last_update and last_change datetimes. I deliberately chose not to make them null to simplify the rules so the users don't have to always check for that. Instead I chose to initialize them with the timestamp when the item gets created.

That would also result in a proper behavior when restoring from persistence.

Time 0 - item creation:
  state: NULL
  previousState: NULL
  lastUpdate: 0,
  lastChange: 0,
...
Time 10 - persistence restore:
  state: "asdf"
  previousState: NULL
  lastUpdate: 10,
  lastChange: 10,

So I am basically suggesting the opposite of what I wrote above 🙈 🤣

[jimtng]

These use cases are now currently supported. Any move of restoreOnStartup to core would need to handle these use cases too. How the Item's state is saved and which Items are restored needs to be configurable on an Item by Item basis like it is now in persistence.

Each item can have a special metadata restoreOnStartup=true. When this is on a group item, persist all its direct members.

[mherwege]

Each item can have a special metadata restoreOnStartup=true. When this is on a group item, persist all its direct members.

I think, what @rkoshak also implied is that one may want to use different rules for persisting values, not just on every change, but e.g. persist it once and always use that same value at startup. And that would mean we also need to support that part of the configuration in core. And then we are back to doing it the way it is done now anyway.

Thinking further about it:

• Changing the restoreOnStartup logic to also restore lastChange, lastUpdate and previousState at startup is fairly easy to do. The method to change is

  openhab-core/bundles/org.openhab.core.persistence/src/main/java/org/openhab/core/persistence/internal/PersistenceManagerImpl.java

  Line 521 in 506ccd4

  private void restoreItemStateIfPossible(Item item) {

  . You may need some extra methods (or an extended setState method that has extra parameters) in GenericItem to also set these fields, although it should probably only be used for restoreOnStartup.
• Many persistence services can already provide the extra info required. MapDB cannot and should be extended to store and retrieve the extra information. I don't know how easy that would be to do. We also seem to be using a very old version of MapDB, so I don't know how much impact that has on feasability.

With the above logic in place, it also would make sense to revisit the persistence extension actions to not return null, but return what is actually in the database, and document it as a breaking change, advising users to use the Item methods for this use case.

[mherwege]

• Many persistence services can already provide the extra info required. MapDB cannot and should be extended to store and retrieve the extra information. I don't know how easy that would be to do. We also seem to be using a very old version of MapDB, so I don't know how much impact that has on feasability.

Looking at MapDB code, it is actually not that difficult. The item state and time are already stored as a JSON string. Adding extra fields should not be a problem.

[rkoshak]

I think, what @rkoshak also implied is that one may want to use different rules for persisting values, not just on every change, but e.g. persist it once and always use that same value at startup. And that would mean we also need to support that part of the configuration in core. And then we are back to doing it the way it is done now anyway.

Since a user needs a rule for calling .persist, if there's also way to turn off the restoreOnStartup for that Item in this new core feature, maybe that would be sufficient as they could create a system started rule to restore the Item's state manually from whatever persistence they are using instead of this new core one. Or maybe we can add a persistence extension to restore the state with the most recent value in the database that could be called from a rule in place of the restoreOnStartup strategy that would be removed.

This new core service doesn't need to do all the use cases. We just need to ensure that all the use cases are covered somehow.

With #4324 I think we will reliably be able to count on persistence being there by runlevel 100. So maybe a system started rule with previousState would be sufficient. I don't know though, I haven't fully explored all the edge cases that can crop up due to timing and such.

2. Item did not change since start -> now, which is wrong

No - it should return null because it's undefined.

Maybe we are thinking too hard about this.

What do we intend these new properties to represent? I see two options:

1. reflect the changes/updates made to the Item
2. refelct the changes/updates to the device

If it's just 1, we don't really need to worry about most of these complexities. If an Item doesn't change after OH startup, getLastChange will be the time when the Item was created and set from restoreOnStartup and getPreviousState would be NULL. If you want the actual device information and not just information about the Item during this run session of OH, then you need to use the existing persistence capabilities. But this also means we don't need a new restoreOnStartup service in core or any of that complexity and the implementation becomes pretty simple.

If we want to make 2 happen then it gets much more complicated, as we can see and we probably need something built into core.

But do we really need to solve 2 here? Maybe 1 is sufficient? I think users who prefer not to use persistence at all would love to have 1.

We also need to thing of second tier impacts like the increase in writes for those running on SD cards and the like if this core restoreOnStartup feature becomes the default behavior.

[jimtng]

So what's the next step?

• Changing the restoreOnStartup logic to also restore lastChange, lastUpdate and previousState at startup is fairly easy to do. The method to change is

  openhab-core/bundles/org.openhab.core.persistence/src/main/java/org/openhab/core/persistence/internal/PersistenceManagerImpl.java

  Line 521 in 506ccd4

  private void restoreItemStateIfPossible(Item item) {

  
  . You may need some extra methods (or an extended setState method that has extra parameters) in GenericItem to also set these fields, although it should probably only be used for restoreOnStartup.

If we are going to proceed in this direction, I would suggest:

• First merge this PR without the extra setState / setters for lastChange, lastUpdate etc.
• If and when the integration with the persistence extensions is added, then add the setters as you suggested. This way, it's clearer how the setters are implemented according to how it's actually used.

[mherwege]

LGTM

[mherwege]

If we are going to proceed in this direction, I would suggest:

• First merge this PR without the extra setState / setters for lastChange, lastUpdate etc.
• If and when the integration with the persistence extensions is added, then add the setters as you suggested. This way, it's clearer how the setters are implemented according to how it's actually used.

I am fine with this.

[spacemanspiff2007]

I think this is the best way, too.
Maybe it makes sense to make persistence of lastChange etc. configurable. This could and should be discussed in the according PR.
Do the the new values also get reported in the corresponding item events?

[openhab-bot]

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/lastupdate-and-null-values/158134/6

[openhab-bot]

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/how-to-monitor-if-item-is-frequently-updated/158260/6

[jimtng]

how do we resolve the access to these methods vs the persistence methods with the same name within rulesDSL?

[mherwege]

how do we resolve the access to these methods vs the persistence methods with the same name within rulesDSL?

Maybe I miss something here, but what gets called would be determined by the imports I would think. As both items and persistence are imported by default, it indeed leads to a conflict which will force changing the rules to have a fully qualified name.

The options then become:

1. Give these new methods another name: this is fully backward compatible.
2. Change the name of the existing persistence actions to getLastPersistedState, getLastPersistedUpdate and getPreviousPersistedState.

In both cases, rules would work without being changed, but 2 would have a slight change in behaviour. I would still prefer 2 because it is more natural not to call persistence if not needed. If one absolutely wants the value from persistence the rule would need to be changed, but it would be very clear what is expected. At the same time, the persistence actions could be modified again to not return null, but the last effective persisted value.
2 also requires adapting the scripting languages to call these changed methods.

[jimtng]

2. Change the name of the existing persistence actions to getLastPersistedState, getLastPersistedUpdate and getPreviousPersistedState.

I think I like this option too!

[andrewfg]

Should you not rather have these methods on the Channels rather than the Items? In that case one would be able to create respective Items with (say) a profile that reads the Channel's getLastUpdated (say) property.

[mherwege]

Should you not rather have these methods on the Channels rather than the Items? In that case one would be able to create respective Items with (say) a profile that reads the Channel's getLastUpdated (say) property.

That would make it impossible to restore these values from persistence at startup. Items have a state, channels only pass on a state. I don´t think we should do this.

[andrewfg]

If you insist on doing it at Item level, then I think you need to find a mechanism whereby the new values from these functions can be accessed in the UI. (e.g. via a profile). Otherwise they are only accessible via rules..

[jimtng]

a mechanism whereby the new values from these functions can be accessed in the UI

They are available through the REST API and it is already a part of this PR.

[andrewfg]

^
Are the data able to be displayed on a widget or a sitemap?

[jimtng]

^ Are the data able to be displayed on a widget or a sitemap?

I don't see why not. It's available in the REST response, along with the current item state

[andrewfg]

don't see why not.

I take that to actually mean that this PR does NOT have a mechanism for that. Or??

I think the Item shall have an attribute that selects if the 'display value' of the item is its actual value (as currently) or the respective value from one of these new methods. This would allow the Item to be configured in OH core, and it's display value in (say) a sitemap would simply depend on the Item config. i.e. it would simply work as a drop in replacement for existing sitemap code.

[florian-h05]

I think the Item shall have an attribute that selects if the 'display value' of the item is its actual value (as currently) or the respective value from one of these new methods.

I would expect to get all values from the Item through the Items endpoint, so the UI (Main UI) could display all values.
The actual state for pages is received through SSE, here and for Sitemaps I agree it would be better to not provide all values but instead only a selected one.

[andrewfg]

for Sitemaps I agree it would be better to .. provide .. only a selected one.

Same for widgets too.

[florian-h05]

Yes, I don’t want to transfer too much over SSE.
Anyway, I see the primary use case of this addition in rules …

[jimtng]

I take that to actually mean that this PR does NOT have a mechanism for that.

No, that would need to be done in the UI itself

[andrewfg]

Coming back to my original point, I believe that these proposed new properties are architecturally speaking attributes of the Channels and NOT attributes of the Items. (Your argument about persistence notwithstanding). And when an Item is created on that Channel, you can select whether it pulls the actual state, or the previous state, or the instant of last change or last update. Whereby that selection is made via a profile, in the normal OH manner. And it requires no downstream changes in MainUI or any other UI.

IMHO the current PR is a kludge; it breaks the OH architectural logic; and (therefore) will require a whole lot of other kludges downstream from this.

[jimtng]

I believe that these proposed new properties are architecturally speaking attributes of the Channels and NOT attributes of the Items.

Items have states. The state can be set by a Thing through a channel, but it can also be set by a rule or through the rest api. A virtual item, one without any channels, can still have a state.

Furthermore, all 4 attributes (of the item) are needed at the same time

• the current state
• the previous state
• last change timestamp
• last update timestamp

Moving these to the channel makes no sense to me and it means that the linked item state can only hold one of the 4 attributes. That would completely miss the point of this PR.

[andrewfg]

Moving these to the channel makes no sense to me and it means that the linked item state can only hold one of the 4 attributes. That would completely miss the point of this PR.

And keeping these in an Item makes no sense to me. It means that the respective Item's values for these properties are not accessible and cannot be displayed in UIs such as site maps or widgets using the normally accepted widget/sitemap design semantics.

[andrewfg]

^
To be specific: how do you propose to do the following??

sitemap 24g label="Munged Generic Item Test" {
        Frame label="Munged Generic Item Test" {
              // ok: standard sitemap semantics
              Text item=Munged_Generic_Item
              // problem: how do you propose to do this ??
              Text item=Munged_Generic_Item.getPreviousState 
              Text item=Munged_Generic_Item.getLastUpdate
              Text item=Munged_Generic_Item.getLastChange
        }
}

[mherwege]

To be specific: how do you propose to do the following??

We don't to propose to do this at all. This PR is not meant to help with UI representation of these properties. This PR is meant to solve a long standing issue with getting these values in rules. As of now, persistence needs to be used for that. But this does not work well and often gives the wrong result, or no result at all. By keeping it with the item state, we don't need to query persistence for this anymore.

I would argue what you would want is another problem to solve, and there may indeed be arguments for that. But doing what you ask in this PR will make it impossible to solve the issue this PR is intending to solve.

[jimtng]

To be specific: how do you propose to do the following??

There are many ways, but the easiest is using label format with scripting

sitemap 24g label="Munged Generic Item Test" {
        Frame label="Munged Generic Item Test" {
              // ok: standard sitemap semantics
              Text item=Munged_Generic_Item
              // problem: how do you propose to do this ??
              Text item=Munged_Generic_Item label="Previous State [RB(|Munged_Generic_Item.previous_state):%s]"
              Text item=Munged_Generic_Item label="Last Update [RB(|Munged_Generic_Item.last_update):%s]"
              Text item=Munged_Generic_Item label="Last Change [RB(|Munged_Generic_Item.last_change):%s]"
        }
}

But as @mherwege said above, this PR has nothing to do with this. You are trying to solve a different problem.

[andrewfg]

You are trying to solve a different problem.

Just to make this clear: I am not trying to solve any problem. I am trying to ensure that your 'solution' here shall not start a chain reaction of creating other new problems elsewhere.

[spacemanspiff2007]

Yes, I don’t want to transfer too much over SSE.

Since HABApp already provides the timestamps, it would be nice if those could be in sync with openHAB.
I'd be interested in receiving these values through SSE and/or websockets.

[jimtng]

I am trying to ensure that your 'solution' here shall not start a chain reaction of creating other new problems elsewhere.

I'm trying to understand what problem this could create.