Docs: add documentation for Backend Metadata and Timedata (#2314)

Adds some basic backend documentation about Metadata and Timedata. Gives some more starter information on the usage of the influxdb and the aggregated influxdb bundle. --------- Co-authored-by: Thomas Sicking <[email protected]> Co-authored-by: Stefan Feilmeier <[email protected]> Co-authored-by: Michael Grill <[email protected]>
OpenEMS · Oct 1, 2023 · 88f8d64 · 88f8d64
1 parent 7e2a6c1
commit 88f8d64
Show file tree

Hide file tree

Showing 13 changed files with 333 additions and 42 deletions.
diff --git a/build.gradle b/build.gradle
@@ -247,42 +247,48 @@ task copyBundleReadmes() {
 		def targetBridge = basePath + "edge/bridge.adoc.d"
 		def targetDeviceService = basePath + "edge/device_service.adoc.d"
 		def targetTimedata = basePath + "edge/timedata.adoc.d"
-
+		def targetBackendService = basePath + "backend/service.adoc.d"
+
 		// initialize target files and directories
-		[targetController, targetScheduler, targetNature, targetBridge, targetDeviceService, targetTimedata].each { target ->
-			delete fileTree(dir: target, include: '**/*.adoc') 
+		[targetController, targetScheduler, targetNature, targetBridge, targetDeviceService, targetTimedata, targetBackendService].each { target ->
+			delete fileTree(dir: target, include: '**/*.adoc')
 			new File(target + "/_include.adoc").write('')
 		}
-	
+
 		subprojects.each { proj ->
 			// in each subproject (= bundle)...
 			proj.file(".").listFiles().each { sourceFile ->
 				// find the 'readme.adoc' file
 				if(sourceFile.getName().equalsIgnoreCase("readme.adoc")) {
 					def bundle = sourceFile.getParentFile().getName()
 					def target = null
-					// evaluate the OpenEMS Component ('Backend' or 'Edge')
+
 					if(bundle.startsWith("io.openems.edge.")) {
-					// evaluate the bundle type (e.g. 'Controller')
-					def edgeBundle = bundle.substring("io.openems.edge.".length())
-					if(edgeBundle.endsWith(".api")) {
-						target = targetNature
-					} else if(edgeBundle.startsWith("controller.")) {
-						target = targetController
-					} else if(edgeBundle.startsWith("scheduler.")) {
-						target = targetScheduler
-					} else if(edgeBundle.startsWith("bridge.")) {
-						target = targetBridge
-					} else if(edgeBundle.startsWith("timedata.")) {
-						target = targetTimedata
-					} else {
-						target = targetDeviceService
-					}
+						def edgeBundle = bundle.substring("io.openems.edge.".length())
+						if(edgeBundle.endsWith(".api")) {
+							target = targetNature
+						} else if(edgeBundle.startsWith("controller.")) {
+							target = targetController
+						} else if(edgeBundle.startsWith("scheduler.")) {
+							target = targetScheduler
+						} else if(edgeBundle.startsWith("bridge.")) {
+							target = targetBridge
+						} else if(edgeBundle.startsWith("timedata.")) {
+							target = targetTimedata
+						} else {
+							target = targetDeviceService
+						}
 
 					} else if(bundle.startsWith("io.openems.backend.")) {
-						// ignore
-						return
-
+						def backendBundle = bundle.substring("io.openems.backend.".length())
+						if(backendBundle.startsWith("timedata.")) {
+							target = targetBackendService
+						} else if(backendBundle.startsWith("metadata.")) {
+							target = targetBackendService
+						} else {
+							return // ignore
+						}
+
 					} else if(bundle.startsWith("io.openems.wrapper")) {
 						// ignore
 						return

diff --git a/doc/modules/ROOT/nav.adoc b/doc/modules/ROOT/nav.adoc
@@ -21,6 +21,9 @@
 * OpenEMS Backend
 ** xref:backend/architecture.adoc[Architecture]
 ** xref:backend/backend-to-backend.adoc[Backend-to-Backend]
+** xref:backend/metadata.adoc[Metadata]
+** xref:backend/timedata.adoc[Timedata]
+** xref:backend/service.adoc[Service]
 ** xref:backend/build.adoc[Build OpenEMS Backend]
 ** xref:backend/deploy.adoc[Deploy OpenEMS Backend]
 * xref:component-communication/index.adoc[Component Communication]

diff --git a/doc/modules/ROOT/pages/backend/metadata.adoc b/doc/modules/ROOT/pages/backend/metadata.adoc
@@ -0,0 +1,37 @@
+= Metadata
+:sectnums:
+:sectnumlevels: 4
+:toc:
+:toclevels: 4
+:experimental:
+:keywords: AsciiDoc
+:source-highlighter: highlight.js
+:icons: font
+:imagesdir: ../../assets/images
+
+The https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.common/src/io/openems/backend/common/metadata/Metadata.java[Metadata] interface defines a service, that coordinates OpenEMS Backend communication with one or more OpenEMS Edges and with the OpenEMS UI.
+It is responsible for identification and authorization of OpenEMS edges and of Users using the OpenEMS UI.
+Beside that the Metadata bundle is responsible for the state of https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.common/src/io/openems/backend/common/metadata/User.java[users] and https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.common/src/io/openems/backend/common/metadata/Edge.java[edges] and related processes.
+
+== Edges
+
+The Metadata service provides access to the https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.common/src/io/openems/backend/common/metadata/Edge.java[Edge] object - the digital twin representation of the OpenEMS Edge instance.
+[NOTE]
+====
+Identification of OpenEMS Edges is done with a unique API key.
+Please take care, that the API key includes a lot of randomness, because it is implicitly used to authorize the Edge.
+====
+
+== Users
+
+The https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.common/src/io/openems/backend/common/metadata/User.java[User] object includes meta information about a user. 
+Furthermore it is used for identification/authorization of the user.
+The metadata service can be used to change various attributes which are associated with a user (e.g. the users default language).
+
+== Processes
+
+The metadata service provides methods to
+
+* add users to an Edge,
+* set and change alerting/notification settings,
+* set up new Edge devices.
diff --git a/doc/modules/ROOT/pages/backend/service.adoc b/doc/modules/ROOT/pages/backend/service.adoc
@@ -0,0 +1,18 @@
+= Backend Services
+:sectnums:
+:sectnumlevels: 4
+:toc:
+:toclevels: 4
+:experimental:
+:keywords: AsciiDoc
+:source-highlighter: highlight.js
+:icons: font
+:imagesdir: ../../assets/images
+
+The following bundles are available in OpenEMS Backend.
+[NOTE]
+====
+This list is not complete.
+====
+
+include::service.adoc.d/_include.adoc[leveloffset=+0]
diff --git a/doc/modules/ROOT/pages/backend/service.adoc.d/.gitignore b/doc/modules/ROOT/pages/backend/service.adoc.d/.gitignore
@@ -0,0 +1 @@
+*.adoc
diff --git a/doc/modules/ROOT/pages/backend/timedata.adoc b/doc/modules/ROOT/pages/backend/timedata.adoc
@@ -0,0 +1,45 @@
+= Timedata
+:sectnums:
+:sectnumlevels: 4
+:toc:
+:toclevels: 4
+:experimental:
+:keywords: AsciiDoc
+:source-highlighter: highlight.js
+:icons: font
+:imagesdir: ../../assets/images
+
+Live and historical data of an OpenEMS Edge are handled by a https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.common/src/io/openems/backend/common/timedata/Timedata.java[Timedata] service.
+It describes basically methods to write and read Edge data to/from a database. There are different kind of timedata providers within OpenEMS (see xref:service.adoc[Service] for concrete implementations).
+
+Within OpenEMS Backend the only component which uses the different timedata services directly is the https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.core/src/io/openems/backend/core/timedatamanager/TimedataManagerImpl.java[TimedataManager].
+It passes Edge relevant data to **all** Timedata service and it reads the data from the **first** Timedata providers which can deliver it. 
+
+[NOTE]
+====
+Following paragraphs describe the new data handlng since OpenEMS version 2023.8.
+====
+
+OpenEMS Edges will send different types of data to the OpenEMS Backend:
+
+* `TimestampedDataNotification` 
+ ** channel values which have changed 
+ ** every 5 minutes a full snapshot of all channel values is sent, including channels which haven't changed over this time period
+
+* `AggregatedDataNotification`
+ ** data is sent in a format that is optimized for fast querying from the database, to allow very fast responses in OpenEMS UI
+ ** aggregated (average or maximum) channel values are sent every 5 minutes
+ ** Backend services handle values differently according to their aggregation type, e.g. 5-minute-average is always stored; maximum is only stored if feasible for fast energy queries (e.g. at the end of a day, depending on a time-zone)
+
+* `ResendDataNotification`
+ ** historic data that is resent after connection problems between Edge and Backend
+ ** this data is always aggregated in the form of the Edge.Timedata service (e.g. RRD4j) 
+
+Splitting the data enables OpenEMS to implement different timedata providers, 
+which handle data differently. This gives more flexibility when scaling OpenEMS.  
+Also due to performance reasons the computation of `AggregatedDataNotification` is done on the Edge side. 
+This helps keeping CPU load on the database server low.
+
+To get a better understanding of the new Edge data concept, have a look at the   
+xref:service.adoc.d/io.openems.backend.timedata.aggregatedinflux.adoc[Aggregated Influx] bundle.
+
diff --git a/io.openems.backend.metadata.file/readme.adoc b/io.openems.backend.metadata.file/readme.adoc
@@ -0,0 +1,29 @@
+= Metadata File
+
+OpenEMS Backend implementation for Metadata. 
+
+Allows you to configure multiple edges by a single JSON file. 
+Using this bundle enables you to easily set up an OpenEMS 
+Backend for testing purposes.
+
+
+== Example configuration file
+
+```
+{
+	edges: {
+		edge0: {
+			comment: "Edge #0",
+			apikey: "edge0",
+			setuppassword: "abcdefgh"
+		}, 
+		edge1: {
+			comment: "Edge #1",
+			apikey: "edge1",
+			setuppassword: "1234567"
+		}
+	}
+}
+```
+
+https://github.com/OpenEMS/openems/tree/develop/io.openems.backend.metadata.file[Source Code icon:github[]]
diff --git a/io.openems.backend.metadata.file/readme.md b/io.openems.backend.metadata.file/readme.md
diff --git a/io.openems.backend.metadata.odoo/readme.adoc b/io.openems.backend.metadata.odoo/readme.adoc
@@ -0,0 +1,17 @@
+= Metadata Odoo
+
+Metadata OpenEMS Backend implementation for https://github.com/odoo/odoo[Odoo]. 
+
+Enables you to manage multiple edges with one backend.	 
+
+
+[NOTE]
+====
+* This bundle works with Odoo 15. For a seamless integration of Odoo 
+and OpenEMS the open-source https://github.com/OpenEMS/odoo-openems[Odoo-OpenEMS] module is needed.
+====
+
+
+
+
+https://github.com/OpenEMS/openems/tree/develop/io.openems.backend.metadata.odoo[Source Code icon:github[]]
diff --git a/io.openems.backend.timedata.aggregatedinflux/readme.adoc b/io.openems.backend.timedata.aggregatedinflux/readme.adoc
@@ -0,0 +1,122 @@
+= Timedata Aggregated InfluxDB
+
+OpenEMS Backend implementation for aggregated InfluxDB access.
+
+[NOTE]
+====
+Bundle is needed for highly optimized environments with
+thousands of connected edges only. If you do not need that functionality you can skip this chapter.
+====
+
+A large number of OpenEMS edges connected to a single backend leads to a performance bottleneck.
+The bottleneck is mainly due to the access to the database. This `Aggregated InfluxDB` bundle in
+combination with the xref:../io.openems.backend.timedata.influx.adoc[Timedata Influx]
+bundle can mitigate this bottleneck. This happens by writing all data as usual via xref:../io.openems.backend.timedata.influx.adoc[Timedata Influx] into
+an Influx database (`DB 1`). Furthermore, all "aggregated data" (see xref:timedata.adoc[edge datatypes]) are written via the `Aggregated InfluxDB` bundle to
+another database (`DB 2`) located on another server.
+Thus `DB 1` holds all data in high resolution, while `DB 2` only holds very few data (5-minute averages for power values
+ or only daily values for energy). Furthermore, a retention policy of e.g. 90 days is applied to `DB 2`.
+Thus `DB 1` provides relatively slow access to a huge amount of data
+and `DB 2` provides fast and responsive access to a relatively small database.
+
+Within OpenEMS backend access to both databases is managed by the
+https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.core/src/io/openems/backend/core/timedatamanager/TimedataManagerImpl.java[TimedataManager].
+The TimedataManager writes edge relevant data to **all** Timedata providers,
+whereas it reads data only from the **first** Timedata provider,
+ which can deliver.
+Assuming correct configuration, TimedataManager first returns historical data
+from `DB 2` (fast and responsive).
+And only in a very few cases it gets the data from `DB 1`
+(probably slower).
+
+
+
+
+[CAUTION]
+====
+The `Timedata.AggregatedInfluxDB` includes a class `AllowedChannels.java`.
+This implementation includes a *hardcoded* list of channels:
+
+* `ALLOWED_AVERAGE_CHANNELS` are used to calculate the 5min average values (e.g. average power values).
+* `ALLOWED_CUMULATED_CHANNELS` are used to calculate daily values (e.g. energy values).
+
+This list must be adopted to a concrete usecase. It strongly depends on
+
+* your strategy to select component-IDs.
+* the components you are using within OpenEMS.
+
+If you detect some widgets within your OpenEMS-UI which hava empty values,
+it may have to do with an incorrect hardcoded list.
+
+====
+
+== Configuration Setup
+
+If you expect your backend to handle thousands of connected edges,
+the following configuration may provide a good start setup:
+
+*Create database and set retention policy:*
+
+Before starting the OpenEMS backend and after intially setting up the influx servers,
+you need to create the databases `influx0` and `aggregated0`
+and some retention policies:
+
+[source,shell]
+----
+
+##### Influx Server 1 #####
+curl -i -XPOST http://127.0.0.1:8082/query --data-urlencode "q=CREATE DATABASE influx0"
+
+##### Influx Server 2 #####
+curl -i -XPOST http://127.0.0.1:8081/query --data-urlencode "q=CREATE DATABASE aggregated0"
+
+curl -i -XPOST http://127.0.0.1:8081/query --data-urlencode "q=CREATE RETENTION POLICY rp_max ON aggregated0 DURATION 90d REPLICATION 1"
+curl -i -XPOST http://127.0.0.1:8081/query --data-urlencode "q=CREATE RETENTION POLICY rp_avg ON aggregated0 DURATION 90d REPLICATION 1"
+
+----
+
+*OpenEMS backend configuration:*
+
+* bundle `Timedata.AggregatedInfluxDB`
+  ** ComponentID: `timedata0`
+  ** QueryLanguage: `INFLUX_QL`
+  ** URL: http://127.0.0.1:8081
+  ** Apikey: `influxuser:influxpassword`
+  ** Bucket: `aggregated0`
+  ** Retention policy for avg values: `rp_avg`
+  ** Retention policy for max values: `rp_max`
+  ** Measurement Avg: `avg`
+  ** Measurement for max values: `Europe/Berlin=max`
+
+* bundle `Timedata.InfluxDB`
+  ** ComponentID: `timedata1`
+  ** QueryLanguage: `INFLUX_QL`
+  ** URL: http://127.0.0.1:8082
+  ** Org: `-`
+  ** Apikey: `influxuser:influxpassword`
+  ** Bucket: `influx0/autogen`
+  ** Measurement: `data`
+
+* bundle `Core Timedata-Manager`
+ ** Timedata-IDs:  `[timedata0, timedata1]`
+
+
+
+
+[NOTE]
+====
+* Note the different Bucket namings, one with retention policy `autogen` and one without.
+* Make sure that your influx databases are located on two different servers
+or that you can easily move one database to another server later.
+* Be sure that the sequence within the `Core Timedata-Manager` is
+configured correctly - the component-ID of the `Timedata.AggregatedInfluxDB`
+comes first.
+* Influx Database must have at least Version 1.8.10.
+* Take care of your edge to backend connections on the edge side. Be sure to understand and select the right `PersistencePriority` for the different datatypes.
+* Remember the hardcoded list in _AllowedChannels.java_
+* Note that you can shift the load of your databases by choosing different retention policies.
+* Be sure you know exactly how Influx handles *reading* and *writing* data when using different retention policies on one database.
+====
+
+
+https://github.com/OpenEMS/openems/tree/develop/io.openems.backend.timedata.influx.aggregatedinflux[Source Code icon:github[]]
diff --git a/io.openems.backend.timedata.dummy/readme.adoc b/io.openems.backend.timedata.dummy/readme.adoc
@@ -0,0 +1,8 @@
+= Timedata Dummy 
+
+OpenEMS Backend dummy implementation for a timeseries database. 
+
+Using this bundle enables you to easily set up an OpenEMS Backend for testing purposes.
+
+
+https://github.com/OpenEMS/openems/tree/develop/io.openems.backend.timedata.influx[Source Code icon:github[]]