Merge pull request #7853 from wazuh/enhancement/edr3144-wazuh-indexer…

…-update-rework-indices

Add Wazuh indexer rework - Indices

source/user-manual/wazuh-indexer/index-life-management.rst

@@ -3,13 +3,15 @@
 .. meta::
    :description: Learn how to define index management policies in this section of the documentation.
 
-Index life management
-=====================
+Index lifecycle management
+==========================
 
-To optimize your cluster performance, you can perform periodic operations, such as index rollovers and deletions.
+Index lifecycle management helps to optimize the Wazuh indexer cluster performance by controlling the lifecycle of an index. You can perform periodic operations, such as index rollovers and deletions. These periodic operations are configured using Index State Management (ISM) policies.
 
 Index State Management (ISM) lets you automate these operational tasks. You can implement lifecycle policies, such as retention policies, for your data using ISM. ISM triggers index operations automatically based on your policies and the changes detected in index age, size, and documents count.
 
+This section discusses some configuration options to manage the index lifecycle for optimization of the Wazuh indexer storage.
+
 Index retention
 ---------------
 
@@ -23,7 +25,7 @@ Creating a retention policy
 Using the Visual editor
 ~~~~~~~~~~~~~~~~~~~~~~~
 
-#. Click on the upper left menu **☰**, go to **Indexer management**, and select **Index Management**. Choose **State management policies** and click **Create policy**. Select **Visual editor** and click **Continue**.
+#. Click on the **upper left menu ☰**, go to **Indexer management**, and select **Index Management**. Choose **State management policies** and click **Create policy**. Select **Visual editor** and click **Continue**.
 
    .. thumbnail:: /images/manual/wazuh-indexer/state-management-policies.png
       :title: State management policies
@@ -45,10 +47,10 @@ Using the Visual editor
       :align: center
       :width: 80%
 
-#. Click **Add template** under **ISM templates** and enter an index pattern such as ``wazuh-alerts-*`` to apply this policy to future alert indices automatically.
+#. Click **Add template** under **ISM templates** and enter an index pattern such as ``wazuh-alerts-*`` to apply this policy to future alert indices automatically.  The priority  is set to the default value of ``1`` and can be set to any other value. The index with higher priority value is treated first.
 #. Click **Add state** to create a state for index deletion. Enter a name such as ``delete_alerts``.
 #. Click **Add action** and select **Delete** in the **Action type**. Click **Add action**. Then click **Save state**.
-#. Click **Add state** again to create an initial state. Enter a name, such as *initial*.
+#. Click **Add state** again to create an initial state. Enter a name, such as ``initial``.
 #. Choose **Add before** from the **Order** tab and select **delete_alerts**.
 #. Click **Add transition** and select **delete_alerts** as the **Destination state**.
 #. Select **Minimum Index Age** in **Condition**. Input the retention value, for example, **90d** for 90 days, in the **Minimum Index Age**.
@@ -57,7 +59,7 @@ Using the Visual editor
 Using the JSON editor
 ~~~~~~~~~~~~~~~~~~~~~
 
-#. Click on the upper left menu **☰**, go to **Indexer management**, and choose **Index Management**. Choose **State management policies** and click **Create policy**. Select **JSON editor** and click **Continue**.
+#. Click on the **upper left menu ☰**, go to **Indexer management**, and choose **Index Management**. Choose **State management policies** and click **Create policy**. Select **JSON editor** and click **Continue**.
 
    .. thumbnail:: /images/manual/wazuh-indexer/configuration-method-json.png
       :title: JSON editor configuration method
@@ -131,7 +133,7 @@ Using the JSON editor
 Applying the retention policy to alerts index
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-#. Choose **Indices** in **Index Management**.
+#. Click on the **upper left menu ☰**, go to **Indexer management**, and choose **Index Management**. Choose **Indices**.
 #. Select the index or indices to attach the policy.
 #. Click **Actions** > **Apply policy**.
 
@@ -141,4 +143,150 @@ Applying the retention policy to alerts index
       :align: center
       :width: 80%
 
-#. Select the policy created in the previous steps from the **Policy ID** menu. Click **Apply**.
+#. Select the policy created in the previous steps from the **Policy ID** menu. Click **Apply**.
+
+Set up hot-warm architecture
+----------------------------
+
+This section shows how to configure indexes to be stored in hot and warm nodes. A hot-warm architecture is made up of hot and warm nodes with the following characteristics:
+
+-  A hot node is typically fast and expensive due to its high computing resources.
+-  A warm node is slower and cheaper due to lower computing resources.
+
+You can design a hot-warm architecture where you first index your data to hot nodes and after a certain period move them to warm nodes. This architecture is suited for you if you have older data that you don't often query. The older data is moved, to be stored on a slower, and less expensive hardware. This architecture helps save money on computing costs.
+
+Rather than increasing the number of hot nodes, you can add warm nodes for data that you don’t access as frequently.
+
+To configure a hot-warm storage architecture, add ``temp`` attributes to the respective nodes.
+
+.. note::
+
+   You can set the attribute name and value to whatever you want as long as it’s consistent for all your hot and warm nodes.
+
+Configure a hot node
+^^^^^^^^^^^^^^^^^^^^
+
+To configure a hot node, add the following configuration to the ``/etc/wazuh-indexer/opensearch.yml`` file:
+
+.. code-block:: yaml
+
+   node.attr.temp: hot
+
+Restart the Wazuh indexer service:
+
+.. code-block:: console
+
+   # systemctl restart wazuh-indexer
+
+Configure a warm node
+^^^^^^^^^^^^^^^^^^^^^
+
+To configure a warm node, add the following configuration to the ``/etc/wazuh-indexer/opensearch.yml`` file:
+
+.. code-block:: yaml
+
+   node.attr.temp: warm
+
+Restart the Wazuh indexer service:
+
+.. code-block:: console
+
+   # systemctl restart wazuh-indexer
+
+Create indexer state management policy
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Perform the following steps on the Wazuh dashboard console.
+
+#. Confirm that the ``temp`` attributes assigned earlier were applied:
+
+   .. code-block:: none
+
+      GET _cat/nodeattrs?v&h=node,attr,value
+
+#. Create an ISM policy to assign indices using the ``wazuh-alerts-4.x-*`` index pattern to hot nodes and move them to warm nodes after a defined time:
+
+   .. code-block:: none
+      :emphasize-lines: 16
+
+      PUT _plugins/_ism/policies/hot_warm
+      {
+          "policy": {
+              "description": "Send shards from hot to warm nodes",
+              "schema_version": 17,
+              "error_notification": null,
+              "default_state": "hot",
+              "states": [
+                  {
+                      "name": "hot",
+                      "actions": [],
+                      "transitions": [
+                          {
+                              "state_name": "warm",
+                              "conditions": {
+                                  "min_index_age": "30d"
+                              }
+                          }
+                      ]
+                  },
+                  {
+                      "name": "warm",
+                      "actions": [
+                          {
+                              "retry": {
+                                  "count": 3,
+                                  "backoff": "exponential",
+                                  "delay": "1m"
+                              },
+                              "replica_count": {
+                                  "number_of_replicas": 0
+                              }
+                          },
+                          {
+                              "retry": {
+                                  "count": 3,
+                                  "backoff": "exponential",
+                                  "delay": "1m"
+                              },
+                              "allocation": {
+                                  "require": {
+                                      "temp": "warm"
+                                  },
+                                  "include": {},
+                                  "exclude": {},
+                                  "wait_for": false
+                              }
+                          }
+                      ],
+                      "transitions": []
+                  }
+              ],
+              "ism_template": [
+                  {
+                      "index_patterns": [
+                          "wazuh-alerts-*"
+                      ],
+                      "priority": 1
+                  }
+              ]
+          }
+      }
+
+   Adjust the ``min_index_age`` from ``30d`` to your preferred number of days to define the minimum number of days to store the indices on a hot node.
+
+Now all future indices created using the ``wazuh-alerts-4.x-*`` index pattern will be allocated to a hot node. After the ``min_index_age`` condition is met, the indices are moved to a warm node and all replicas removed. The removal of the replicas ensures that storage is managed on the warm node since the data will not be queried frequently.
+
+Apply the ISM policy to existing indices
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+#. Choose **Indices** in **Index Management**.
+#. Select the index or indices to attach the policy.
+#. Click **Actions** > **Apply policy**.
+#. Select the ``hot-warm``  policy in Policy ID.
+#. Click **Apply** to add the policy to the selected indices.
+
+   .. thumbnail:: /images/manual/wazuh-indexer/apply-ism-policy-to-indices.png
+      :title: Apply the ISM policy to indices
+      :alt: Apply the ISM policy to indices
+      :align: center
+      :width: 80%

source/user-manual/wazuh-indexer/re-indexing.rst

@@ -6,14 +6,14 @@
 Re-indexing
 ===========
 
-When changes are made to the data schema, it becomes necessary to re-index data to reflect these changes. Existing data may not match the updated schema without re-indexing, leading to data inconsistencies or errors during queries. Re-indexing lets you copy all or a subset of your data from a source index into a destination index.
+When changes are made to the index’s data schema, it becomes necessary to re-index data to reflect these changes. Existing data may not match the updated schema without re-indexing, leading to data inconsistencies or errors during queries. Re-indexing lets you copy all or a subset of your data from a source index into a destination index.
 
 To re-index an existing index, perform the following steps on either the Wazuh dashboard or the Wazuh server.
 
 Wazuh dashboard
 ---------------
 
-#. Click on the upper left menu **☰** and go to **Indexer management** > **Dev Tools**.
+#. Click on the **upper left menu ☰** and go to **Server management** then **Dev Tools**.
 #. Enter the following API call, replacing ``my-source-index`` with the source index pattern and ``my-destination-index`` with the destination index pattern.
 
    .. code-block:: none
@@ -71,7 +71,7 @@ Command line interface
 
 Run the following command on any Wazuh central component that is allowed to authenticate to the Wazuh API. Replace ``<INDEXER_USERNAME>`` and ``<INDEXER_PASSWORD>`` with the indexer username and password:
 
-.. code-block:: console
+.. code-block:: bash
    :emphasize-lines: 4, 7
 
    curl -k -u "<INDEXER_USERNAME>:<INDEXER_PASSWORD>" -XPOST "https://<INDEXER_IP_ADDRESS>:9200/_reindex" -H 'Content-Type: application/json' -d'
@@ -88,7 +88,7 @@ For example:
 
 .. code-block:: console
 
-   curl -k -u "INDEXER_USERNAME:INDEXER_PASSWORD" -XPOST "https://<INDEXER_IP_ADDRESS>:9200/_reindex" -H 'Content-Type: application/json' -d'
+   root@wazuh-server:~$ curl -k -u "INDEXER_USERNAME:INDEXER_PASSWORD" -XPOST "https://<INDEXER_IP_ADDRESS>:9200/_reindex" -H 'Content-Type: application/json' -d'
    {
       "source":{
          "index":"wazuh-alerts-*"
@@ -98,7 +98,7 @@ For example:
       }
    }'
 
-.. code-block:: output
+.. code-block:: none
    :class: output
 
    {"took":18025,"timed_out":false,"total":26854,"updated":26854,"created":0,"deleted":0,"batches":27,"version_conflicts":0,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1.0,"throttled_until_millis":0,"failures":[]}