Skip to content

Commit

Permalink
removing white spaces
Browse files Browse the repository at this point in the history
  • Loading branch information
Snell1224 committed Aug 26, 2024
1 parent 75816ea commit 22225bd
Show file tree
Hide file tree
Showing 192 changed files with 1,933 additions and 1,972 deletions.
6 changes: 3 additions & 3 deletions rtd/README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -33,14 +33,14 @@ Instructions and documentation on how to use ReadTheDocs can be found here:
> git add <files>
> git commit -m "add message"
> git push
Adding A New File
Adding A New File
******************
For any new RST files created, please include them in rtd/docs/src/index.rst under their corresponding sections. All RST files not included in index.rst will not populate on the offical webpage (e.g. readthedocs).

Paper Lock
************
This is for claiming any sections you are working on so there is no overlap.
Please USE paper.lock to indicate if you are editing an existing RST file.
Please USE paper.lock to indicate if you are editing an existing RST file.


28 changes: 14 additions & 14 deletions rtd/docs/source/asf/asf-quickstart.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Create A Simple Analysis
------------------------
To start, please create a folder called ``graf_analysis`` in your home directory and copy the following contents to a python file called ``dsosTemplate.py``:

* This is a python analysis that queries the DSOS database and returns a DataFrame of the ``meminfo`` schema metrics along with the ``timestamp``, ``component_id`` and ``job_id``.
* This is a python analysis that queries the DSOS database and returns a DataFrame of the ``meminfo`` schema metrics along with the ``timestamp``, ``component_id`` and ``job_id``.

dsosTemplate.py:

Expand All @@ -20,7 +20,7 @@ dsosTemplate.py:
class dsosTemplate(Analysis):
def __init__(self, cont, start, end, schema='meminfo', maxDataPoints=4096):
super().__init__(cont, start, end, schema, 1000000)
def get_data(self, metrics, filters=[],params=None):
try:
self.sel = f'select {",".join(metrics)} from {self.schema}'
Expand All @@ -36,37 +36,37 @@ dsosTemplate.py:
a, b, c = sys.exc_info()
print(str(e)+' '+str(c.tb_lineno))
.. note::
If you want to use this analysis module in a Grafana dashboard, you will need to ask your administrator to copy your new analysis module(s) into the directory that Grafana points to. This is because Grafana is setup to look at a specific path directory to query from.
.. note::

If you want to use this analysis module in a Grafana dashboard, you will need to ask your administrator to copy your new analysis module(s) into the directory that Grafana points to. This is because Grafana is setup to look at a specific path directory to query from.

Test Analysis via Terminal Window
----------------------------------
You can easily test your module without the Grafana interface by creating a python script that mimics the Grafana query and formats the returned JSON into a timeseries dataframe or table.
You can easily test your module without the Grafana interface by creating a python script that mimics the Grafana query and formats the returned JSON into a timeseries dataframe or table.

First, create the following file in the same directory as your python analysis (i.e. ``/user/home/graf_analysis/``) and label it ``testDSOSanalysis.py``.
First, create the following file in the same directory as your python analysis (i.e. ``/user/home/graf_analysis/``) and label it ``testDSOSanalysis.py``.

* This python script imitates the Grafana query that calls your analysis module and will return a timeseries DataFrame of the ``Active`` and ``Inactive`` meminfo metrics.

.. code-block :: python
#!/usr/bin/python3
import time,sys
from sosdb import Sos
from grafanaFormatter import DataFormatter
from table_formatter import table_formatter
from time_series_formatter import time_series_formatter
from dsosTemplate import dsosTemplate
sess = Sos.Session("/<DSOS_CONFIG_PATH>/config/dsos.conf")
cont = '<PATH_TO_DATABASE>'
cont = sess.open(cont)
model = dsosTemplate(cont, time.time()-300, time.time(), schema='meminfo', maxDataPoints=4096)
x = model.get_data(['Active','Inactive'], filters=['job_id'], params='')
#fmt = table_formatter(x)
fmt = time_series_formatter(x)
x = fmt.ret_json()
Expand All @@ -84,7 +84,7 @@ First, create the following file in the same directory as your python analysis (
.. note::

All imports are python scripts that need to reside in the same directory as the test analysis module in order for it to run successfully.
All imports are python scripts that need to reside in the same directory as the test analysis module in order for it to run successfully.

Then, run the python script with the current python verion installed. In this case it would be ``python3 <analysisTemplate.py>``

Expand All @@ -103,7 +103,7 @@ Create A New Dashboard
To create a new dashboard, click on the + sign on the left side of the home page and hit dashboard. This will create a blank dashboard with an empty panel in it. Hit the add query button on the panel to begin configuring the query to be sent to an analysis module. 

.. note::

For more information on how to navigate around the Grafana dashboard and what the variables and advanced settings do, please see `Grafana Panel <grafanapanel>`_ and `Grafana Usage <grafanause>`_.

* Next, add your analysis by filling out the required fields shown below:
Expand Down
10 changes: 5 additions & 5 deletions rtd/docs/source/asf/grafanapanel.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Configuring the Query and Visualization

Once you right click on the panel title and select edit, the panel settings will appear. The first tab is for configuring the query. There are 8 fields in the query field defined below:

* Query Type - type of query to perform. The most commonly used in "analysis" which calls an analysis module. "metrics" is used to return raw data without any analysis module.
* Query Type - type of query to perform. The most commonly used in "analysis" which calls an analysis module. "metrics" is used to return raw data without any analysis module.
* Query Format - the type of visualization to be used on the dataset. It is used by Grafana Formatter to properly JSON-ify the data returned from the analysis module. Can be either time_series, table, or heatmap.
* Analysis - required if you choose analysis query type. Specifies the python module to call to transofrm the data.
* Container - the name of the container to be used. This can be either the full path to the container or the Django backend get_container function can be changed to customize for site settings.
Expand All @@ -19,7 +19,7 @@ Once you right click on the panel title and select edit, the panel settings will
* Extra Params - (Optional) pass in an arbitrary string into the analysis module
* Filters - (Optional) include a no-sql like syntax for filtering your query, can be a comma separated list of filters i.e. component_id == 5,job_id > 0

The second tab in the panel settings is for visualization. Graph, Table, and Heatmap are the available visualizations for a query output.
The second tab in the panel settings is for visualization. Graph, Table, and Heatmap are the available visualizations for a query output.

Text, which uses Markdown language, could also be used for Dashboard descriptions or details. If you use a graph visualization, the query Format should be time_series. If you use a table visualization, the query Format should be table.

Expand All @@ -31,7 +31,7 @@ Dashboard Variables and Advanced Settings
-------------------------------------------
.. image:: ../images/grafana/grafanapanel_variables.png

Often we want users to be able to change inputs into the queries, however users cannot change edit queries. What they can edit in Grafana are variables, which are listed at the top of the dashboard. These variables can be referenced with a ``$`` in front of the variable name. For example, we can let the user switch SOS containers they are interested by creating a variable called container and then putting ``$container`` in the container field of the query. To create variables, go to the dashboard settings (gear button at the top right) and go to variables. Here you can create new variables. Common variable types are text boxes, for users to fill in, or queries. We can actually create a pre-populated list of options for certain fields by querying the container. Below are the queryable metrics what information to put in the query field.
Often we want users to be able to change inputs into the queries, however users cannot change edit queries. What they can edit in Grafana are variables, which are listed at the top of the dashboard. These variables can be referenced with a ``$`` in front of the variable name. For example, we can let the user switch SOS containers they are interested by creating a variable called container and then putting ``$container`` in the container field of the query. To create variables, go to the dashboard settings (gear button at the top right) and go to variables. Here you can create new variables. Common variable types are text boxes, for users to fill in, or queries. We can actually create a pre-populated list of options for certain fields by querying the container. Below are the queryable metrics what information to put in the query field.

* Container - select the custom option in the **Type** field and add the name of the container being used to query from in the **custom options** field.
* Schema - ``query=schema&container=<cont_name>``
Expand All @@ -40,8 +40,8 @@ Often we want users to be able to change inputs into the queries, however users
* Component IDs - ``query=components&container=<cont_name>&schema=<schema_name>``
* Jobs - ``query=jobs&container=<cont_name>&schema=<schema_name>``

You can put variables in queries as well. For example, if you already have a $container variable, you can set the schema variable query to be ``query=schema&container=$container``. Then the ``$schema`` variable can be used in other queries.
You can put variables in queries as well. For example, if you already have a $container variable, you can set the schema variable query to be ``query=schema&container=$container``. Then the ``$schema`` variable can be used in other queries.

In the dashboard settings you can also change the dashboard name and folder location and load previously saved versions.
In the dashboard settings you can also change the dashboard name and folder location and load previously saved versions.

Other than the container variable, all other variables bulleted above are set to query in the **Type** field
44 changes: 22 additions & 22 deletions rtd/docs/source/asf/index.rst
Original file line number Diff line number Diff line change
@@ -1,22 +1,22 @@
.. image:: ../images/appsysfusion.png
:width: 300
:height: 125
:align: center

ASF
====
AppSysFusion provides analysis and visualization capabilities aimed at serving insights from HPC monitoring data gathered with LDMS, though could be generalized outside of that scope.
It combines a Grafana front-end with a Django back-end to perform in-query analyses on raw data and return transformed information back to the end user.
By performing in-query analyses, only data of interest to the end-user is operated on rather than the entirety of the dataset for all analyses for all time.
This saves significant computation and storage resources with the penalty of slightly higher query times.
These analyses are modular python scripts that can be easily added or changed to suit evolving needs.
The current implementation is aimed at querying DSOS databases containing LDMS data, though efforts are in progress to abstract this functionality out to other databases and datatypes.

.. toctree::
:maxdepth: 2

asf-quickstart
asf-tutorial
grafanapanel
grafanause
pyanalysis
.. image:: ../images/appsysfusion.png
:width: 300
:height: 125
:align: center

ASF
====
AppSysFusion provides analysis and visualization capabilities aimed at serving insights from HPC monitoring data gathered with LDMS, though could be generalized outside of that scope.
It combines a Grafana front-end with a Django back-end to perform in-query analyses on raw data and return transformed information back to the end user.
By performing in-query analyses, only data of interest to the end-user is operated on rather than the entirety of the dataset for all analyses for all time.
This saves significant computation and storage resources with the penalty of slightly higher query times.
These analyses are modular python scripts that can be easily added or changed to suit evolving needs.
The current implementation is aimed at querying DSOS databases containing LDMS data, though efforts are in progress to abstract this functionality out to other databases and datatypes.

.. toctree::
:maxdepth: 2

asf-quickstart
asf-tutorial
grafanapanel
grafanause
pyanalysis
Loading

0 comments on commit 22225bd

Please sign in to comment.