Merge pull request #29 from BDonnot/bd-dev

Upgrading to 0.5.3

.github/workflows/main.yml

@@ -108,10 +108,24 @@ jobs:
           python -m pip install --upgrade pip
           python -m pip install -r requirements.txt
 
-      - name: Compile SuiteSparse
+      - name: Compile SuiteSparse make
         if: matrix.config.name == 'darwin'
         run: make
 
+      - name: Compile SuiteSparse cmake
+        if: matrix.config.name == 'windows'
+        run: |
+          cd build_cmake
+          python generate_c_files.py
+          mkdir build
+          cd build
+          cmake -DCMAKE_INSTALL_PREFIX=..\built -DCMAKE_BUILD_TYPE=Release ..
+          cmake --build . --config Release
+          cmake --install . --prefix ..\built
+          cd ..
+          cd ..
+          python setup.py build
+
       - name: Build wheel
         run: python setup.py bdist_wheel
 

.gitignore

@@ -188,4 +188,25 @@ lightsim2grid/tests/timer.py
 wheelhouse/*.whl
 benchmarks/ase6495rte.json
 benchmarks/case6495rte.json
-
+test_dir
+build_cmake/build
+build_cmake/built
+build_cmake/SuiteSparse/*/SourceWrappers
+build_cmake/CMakeLists.old.txt
+build_cmake/SuiteSparse/AMD/Include
+build_cmake/SuiteSparse/AMD/Lib
+build_cmake/SuiteSparse/AMD/Source
+build_cmake/SuiteSparse/BTF/Include
+build_cmake/SuiteSparse/BTF/Lib
+build_cmake/SuiteSparse/BTF/Source
+build_cmake/SuiteSparse/COLAMD/Include
+build_cmake/SuiteSparse/COLAMD/Lib
+build_cmake/SuiteSparse/COLAMD/Source
+build_cmake/SuiteSparse/CXSparse/Include
+build_cmake/SuiteSparse/CXSparse/Lib
+build_cmake/SuiteSparse/CXSparse/Source
+build_cmake/SuiteSparse/KLU/Include
+build_cmake/SuiteSparse/KLU/Lib
+build_cmake/SuiteSparse/KLU/Source
+build_cmake/SuiteSparse/SuiteSparse_config/SuiteSparse_config.c
+build_cmake/SuiteSparse/SuiteSparse_config/SuiteSparse_config.h

CHANGELOG.rst

@@ -10,6 +10,13 @@ Change Log
 - distribute pypi package
 - code helm powerflow method
 - interface with gridpack (to enforce q limits for example)
+- have a solver with "NICSLU" (https://github.com/chenxm1986/nicslu) if available on the platform
+- maybe have a look at suitesparse "sliplu" tools ?
+
+[0.5.3] 2021-xx-yy
+-------------------
+- [FIXED] minor issues in the benchmark (some time measurments were wrong)
+- [IMPROVED] trying to make the package available on pypi
 
 [0.5.2] 2021-07-26
 -------------------

README.md

@@ -43,22 +43,33 @@ Suppose you somehow get:
 You can define replace the `newtonpf` function of `pandapower.pandapower.newtonpf` function with the following
 piece of code:
 ```python
-from lighsim2grid.newtonpf import newtonpf
+from lightsim2grid.newtonpf import newtonpf
 V, converged, iterations, J = newtonpf(Ybus, V, Sbus, pv, pq, ppci, options)
 ```
 
 This function uses the KLU algorithm and a c++ implementation of a Newton solver for speed.
 
-## Installation (from source, recommended)
+## Installation (from pypi official repository, recommended)
+
+Since version 0.5.3, lightsim2grid is can be installed like most python packages, with a call to:
+`python -m pip install lightsim2grid`
+
+It includes faster grid2op backend and the `SuiteSparse` faster `KLU` solver, even on windows. This is definitely the 
+easiest method to install lightsim2grid on your system and have it running without too much issues.
+
+Note though that these packages have been compiled on a different platform that the one you are using. You might still
+get some benefit (in terms of performances) to install it from your on your machine.
+
+## Installation (from source, for more advanced user)
 You need to:
-- clone this repository and get the code of Eigen and SparseSuite (mandatory for compilation)
+- clone this repository and get the code of Eigen (mandatory for compilation) and SparseSuite (optional)
 - (optional) compile a piece of SparseSuite
 - install the package
 
 ### Important note
-This package relies on the excellent pybind11 package to integrate c++ code into python easily. 
+This package relies on the excellent `pybind11` package to integrate c++ code into python easily. 
 
-So to install lightsim2grid you need pybind and its requirement, which include a working compiler: for example 
+So to install lightsim2grid you need `pybind11` and its requirement, which include a working compiler: for example 
 (as of writing) 
 gcc (default on ubuntu, version >= 4.8), clang (default on MacOS, version >= 5.0.0) or 
 Microsoft visual studio (Microsoft Visual Studio 2015 Update 3 or newer). 
@@ -83,25 +94,52 @@ git submodule update
 ```
 
 ### (optional) Compilation of SuiteSparse
-SuiteSparse comes with the faster KLU linear solver. Since version 0.3.0 this requirement has been removed. This entails
-that on linux / macos you can still benefit from the faster KLU solver. On windows you will still benefit from the
+SuiteSparse comes with the faster KLU linear solver. 
+
+Since version 0.3.0 this requirement has been removed. This entails
+that on linux / macos you can still benefit from the faster KLU solver. You can still benefit from the
 speed up of lightsim (versus the default PandaPowerBackend) but this speed up will be less than if you manage
 to compile SuiteSparse (see the subsection [Benchmark](#benchmark) for more information).
 
 **NB** in both cases the algorithm to compute the powerflow is exactly the same. It is a 
-Newton Raphson based method. But to carry out this algorithm, one need to solver some linear equations. The only
+Newton-Raphson based method. But to carry out this algorithm, one need to solver some linear equations. The only
 difference in the two version (with KLU and without) is that the linear equation solver is different. Up to the
 double float precision, both results (with and without KLU) should match.
 
-We only detail the compilation on a system using "make" (so most likely GNU-Linux and MacOS). If you manage to
-do this step on Windows, you can continue (and let us know!). If you don't feel comfortable with this, we
+We only detail the compilation on a system using "make" (so most likely GNU-Linux and MacOS). If you don't feel 
+comfortable with this, either you can ignore it, or you have also the possibility to use the
 provided a docker version. See the next section [Installation Using Docker](#installation-using-docker) 
 for more information.
 
+#### (optional) option A. Compilation of SuiteSparse using "make"
+This is the easiest method to compile SuiteSparse on your system but unfortunately it only works on OS where "make" is
+available (*eg* Linux or MacOS) but this will not work on Windows... The compilation on windows is covered in the next
+paragraph 
+[(optional) option B. Compilation of SuiteSparse using "cmake"](#\(optional\)-option-B.-Compilation-of-SuiteSparse-using-"cmake")
+
+Anyway, in this case, it's super easy. Just do:
+
 ```commandline
 # compile static libraries of SparseSuite
 make
 ```
+And you are good to go. Nothing more.
+
+#### (optional) option B. Compilation of SuiteSparse using "cmake"
+This works on most platform including MacOS, Linux and Windows.
+
+It requires to install the free `cmake` program and to do a bit more works than for other system. This is why we
+only recommend to use it on Windows.
+
+The main steps (for windows, somme commands needs to be adapted on linux / macos) are:
+1) `cd build_cmake`
+2) `py generate_c_files.py`
+3) `mkdir build` and cd there: `cd build`
+4) `cmake -DCMAKE_INSTALL_PREFIX=..\built -DCMAKE_BUILD_TYPE=Release ..`
+5) `cmake --build . --config Release`
+6) `cmake --build . --config Release --target install`
+
+For more information, feel free to read the dedicated [README](build_cmake/README.md).
 
 ### 2. Installation of the python package
 Now you simply need to install the lightsim2grid package this way, like any python package:
@@ -128,6 +166,8 @@ python3 benchmark_solvers.py --name l2rpn_neurips_2020_track2_small --no_test --
 ```
 (results may vary depending on the hard drive, the ram etc. and are presented here for illustration only)
 
+(to run these benchmarks, some data will automatically be downloaded, but this requires an internet access)
+
 (we remind that these simulations correspond to simulation on one core of the CPU. Of course it is possible to
 make use of all the available cores, which would increase the number of steps that can be performed)
 

benchmarks/benchmark_solvers.py

@@ -74,6 +74,7 @@ def main(max_ts, env_name_input, test=True,
                                                                      with_type_solver=wst, env_seed=0)
         klu_comp_time = env_lightsim.backend.comp_time
         klu_time_pf = env_lightsim._time_powerflow
+
     if lightsim2grid.SolverType.SparseLU in solver_types:
         env_lightsim.backend.set_solver_type(lightsim2grid.SolverType.SparseLU)
         env_lightsim.backend.set_solver_max_iter(10)
@@ -101,23 +102,23 @@ def main(max_ts, env_name_input, test=True,
     # NOW PRINT THE RESULTS
     env_name = get_env_name_displayed(env_name_input)
     hds = [f"{env_name}", f"grid2op speed (it/s)", f"grid2op powerflow time (ms)", f"solver powerflow time (ms)"]
-    tab = [["PP", int(nb_ts_pp/time_pp),
+    tab = [["PP", f"{nb_ts_pp/time_pp:.2e}",
             f"{1000.*pp_time_pf/nb_ts_pp:.2e}",
             f"{1000.*pp_comp_time/nb_ts_pp:.2e}"]]
     if lightsim2grid.SolverType.GaussSeidel in solver_types and no_gs is False:
-        tab.append(["LS+GS", int(nb_ts_gs/time_gs),
+        tab.append(["LS+GS", f"{nb_ts_gs/time_gs:.2e}",
                     f"{1000.*gs_time_pf/nb_ts_gs:.2e}",
                     f"{1000.*gs_comp_time/nb_ts_gs:.2e}"])
     if lightsim2grid.SolverType.GaussSeidelSynch in solver_types and no_gs_synch is False:
-        tab.append(["LS+GS S", int(nb_ts_gsa/time_gsa),
+        tab.append(["LS+GS S", f"{nb_ts_gsa/time_gsa:.2e}",
                     f"{1000.*gsa_time_pf/nb_ts_gsa:.2e}",
                     f"{1000.*gsa_comp_time/nb_ts_gsa:.2e}"])
     if lightsim2grid.SolverType.SparseLU in solver_types:
-        tab.append(["LS+SLU", int(nb_ts_slu/time_slu),
+        tab.append(["LS+SLU", f"{nb_ts_slu/time_slu:.2e}",
                     f"{1000.*slu_time_pf/nb_ts_slu:.2e}",
                     f"{1000.*slu_comp_time/nb_ts_slu:.2e}"])
     if lightsim2grid.SolverType.KLU in solver_types:
-        tab.append(["LS+KLU", int(nb_ts_klu/time_klu),
+        tab.append(["LS+KLU", f"{nb_ts_klu/time_klu:.2e}",
                     f"{1000.*klu_time_pf/nb_ts_klu:.2e}",
                     f"{1000.*klu_comp_time/nb_ts_klu:.2e}"])
 

benchmarks/utils_benchmark.py

@@ -79,8 +79,14 @@ def run_env(env, max_ts, agent, chron_id=None, keep_forecast=False, with_type_so
     else:
         obs = env.get_obs()
 
+    # don't forget to reset the timers
+    env.backend.comp_time = 0.
+    # it's not 0. because a powerflow is run when the backend is reset, and this is one more powerflow than the
+    # number of steps
+
     if with_type_solver:
         print(f"after reset: {env.backend._grid.get_solver_type()}")
+
     done = False
     reward = env.reward_range[0]
     nb_ts = 0