WIP: added tools command; closes #44 (#60)

* added tools command with placeholders for un/reinstall along with placeholder tests * added missing docs build dependency * updated documentation to reflect tools vs install * refactored some code for DRY, fixed up prior merge with master * fixed broken tests in test_recon_pipeline_shell * existing tests all passing * added tools list command * added tools list command * added tools reinstall * removed lint * fixed reinstall test * fixed install go test * fixed go install test again
epi052 · Jun 28, 2020 · 9d5cac6 · 9d5cac6
1 parent 1ad3adc
commit 9d5cac6
Show file tree

Hide file tree

Showing 12 changed files with 287 additions and 117 deletions.
diff --git a/README.md b/README.md
@@ -58,14 +58,14 @@ pipenv install
 pipenv shell
 ```
 
-After installing the python dependencies, the `recon-pipeline` shell provides its own [install](https://recon-pipeline.readthedocs.io/en/latest/api/commands.html#install) command (seen below).  A simple `install all` will handle all additional installation steps.
+After installing the python dependencies, the `recon-pipeline` shell provides its own [tools](https://recon-pipeline.readthedocs.io/en/latest/api/commands.html#tools) command (seen below).  A simple `tools install all` will handle all additional installation steps.
 
 > Ubuntu Note (and newer kali versions):  You may consider running `sudo -v` prior to running `./recon-pipeline.py`.  `sudo -v` will refresh your creds, and the underlying subprocess calls during installation won't prompt you for your password.  It'll work either way though.
 
-Individual tools may be installed by running `install TOOLNAME` where `TOOLNAME` is one of the known tools that make
+Individual tools may be installed by running `tools install TOOLNAME` where `TOOLNAME` is one of the known tools that make
 up the pipeline.
 
-The installer maintains a (naive) list of installed tools at `~/.local/recon-pipeline/tools/.tool-dict.pkl`.  The installer in no way attempts to be a package manager.  It knows how to execute the steps necessary to install its tools.  Beyond that, it's like Jon Snow, **it knows nothing**.
+The installer maintains a (naive) list of installed tools at `~/.local/recon-pipeline/tools/.tool-dict.pkl`.  The installer in no way attempts to be a package manager.  It knows how to execute the steps necessary to install and remove its tools.  Beyond that, it's like Jon Snow, **it knows nothing**.
 
 [![asciicast](https://asciinema.org/a/318395.svg)](https://asciinema.org/a/318395)
 
@@ -239,7 +239,7 @@ The backbone of this pipeline is spotify's [luigi](https://github.com/spotify/lu
 - Make sure two instances of the same task are not running simultaneously
 - Provide visualization of everything that’s going on
 
-While in the `recon-pipeline` shell, running `install luigi-service` will copy the `luigid.service` file provided in the
+While in the `recon-pipeline` shell, running `tools install luigi-service` will copy the `luigid.service` file provided in the
 repo to its appropriate systemd location and start/enable the service.  The result is that the central scheduler is up
 and running easily.
 

diff --git a/docs/api/commands.rst b/docs/api/commands.rst
@@ -9,23 +9,23 @@ Commands
 
 ``recon-pipeline`` provides a handful of commands:
 
-- ``install``
-- ``scan``
-- ``status``
-- ``database``
-- ``view``
+- :ref:`tools_command`
+- :ref:`scan_command`
+- :ref:`status_command`
+- :ref:`database_command`
+- :ref:`view_command`
 
 All other available commands are inherited from `cmd2 <https://github.com/python-cmd2/cmd2>`_.
 
-.. _install_command:
+.. _tools_command:
 
-install
-#######
+tools
+#####
 
 .. argparse::
     :module: pipeline.recon
-    :func: install_parser
-    :prog: install
+    :func: tools_parser
+    :prog: tools
 
 
 .. _database_command:

diff --git a/docs/overview/installation.rst b/docs/overview/installation.rst
@@ -51,15 +51,15 @@ Both OSs After ``pipenv`` Install
 Everything Else
 ###############
 
-After installing the python dependencies, the recon-pipeline shell provides its own :ref:`install_command` command (seen below).
-A simple ``install all`` will handle all installation steps.  Installation has **only** been tested on **Kali 2019.4 and Ubuntu 18.04/20.04**.
+After installing the python dependencies, the recon-pipeline shell provides its own :ref:`tools_command` command (seen below).
+A simple ``tools install all`` will handle all installation steps.  Installation has **only** been tested on **Kali 2019.4 and Ubuntu 18.04/20.04**.
 
     **Ubuntu Note (and newer kali versions)**: You may consider running ``sudo -v`` prior to running ``./recon-pipeline.py``. ``sudo -v`` will refresh your creds, and the underlying subprocess calls during installation won't prompt you for your password. It'll work either way though.
 
-Individual tools may be installed by running ``install TOOLNAME`` where ``TOOLNAME`` is one of the known tools that make
+Individual tools may be installed by running ``tools install TOOLNAME`` where ``TOOLNAME`` is one of the known tools that make
 up the pipeline.
 
-The installer maintains a (naive) list of installed tools at ``~/.local/recon-pipeline/tools/.tool-dict.pkl``.  The installer in no way attempts to be a package manager.  It knows how to execute the steps necessary to install its tools.  Beyond that, it's
+The installer maintains a (naive) list of installed tools at ``~/.local/recon-pipeline/tools/.tool-dict.pkl``.  The installer in no way attempts to be a package manager.  It knows how to execute the steps necessary to install and remove its tools.  Beyond that, it's
 like Jon Snow, **it knows nothing**.
 
 Tools can also be uninstalled using the ``uninstall all`` command. It is also possible to individually uninstall them in the same manner as shown above.

diff --git a/docs/overview/scheduler.rst b/docs/overview/scheduler.rst
@@ -11,10 +11,8 @@ provides the following two benefits:
 - Make sure two instances of the same task are not running simultaneously
 - Provide :ref:`visualization <visualization-ref-label>` of everything that’s going on
 
-While in the ``recon-pipeline`` shell, running ``install luigi-service`` will copy the ``luigid.service``
+While in the ``recon-pipeline`` shell, running ``tools install luigi-service`` will copy the ``luigid.service``
 file provided in the repo to its appropriate systemd location and start/enable the service. The result is that the
 central scheduler is up and running easily.
 
 The other option is to add ``--local-scheduler`` to your :ref:`scan_command` command from within the ``recon-pipeline`` shell.
-
-
diff --git a/docs/overview/visualization.rst b/docs/overview/visualization.rst
@@ -7,7 +7,7 @@ Setup
 #####
 
 To use the web console, you'll need to :ref:`install the luigid service<install-ref-label>`.  Assuming you've already
-installed ``pipenv`` and created a virtual environment, you can simply run the ``install luigi-service``
+installed ``pipenv`` and created a virtual environment, you can simply run the ``tools install luigi-service``
 from within the pipeline.
 
 Dashboard

diff --git a/pipeline/recon-pipeline.py b/pipeline/recon-pipeline.py
@@ -12,7 +12,9 @@
 import threading
 import subprocess
 import webbrowser
+from enum import IntEnum
 from pathlib import Path
+from typing import List, NewType
 
 DEFAULT_PROMPT = "recon-pipeline> "
 
@@ -62,15 +64,18 @@ def cluge_package_imports(name, package):
 from .recon import (  # noqa: F401,E402
     get_scans,
     scan_parser,
-    install_parser,
-    uninstall_parser,
+    view_parser,
+    tools_parser,
     status_parser,
     database_parser,
-    db_detach_parser,
-    db_list_parser,
     db_attach_parser,
     db_delete_parser,
-    view_parser,
+    db_detach_parser,
+    db_list_parser,
+    tools_list_parser,
+    tools_install_parser,
+    tools_uninstall_parser,
+    tools_reinstall_parser,
     target_results_parser,
     endpoint_results_parser,
     nmap_results_parser,
@@ -81,6 +86,14 @@ def cluge_package_imports(name, package):
 
 from .tools import tools  # noqa: F401,E402
 
+
+class ToolAction(IntEnum):
+    INSTALL = 0
+    UNINSTALL = 1
+
+
+ToolActions = NewType("ToolActions", ToolAction)
+
 # select loop, handles async stdout/stderr processing of subprocesses
 selector = selectors.DefaultSelector()
 
@@ -144,6 +157,10 @@ def _initialize_parsers(self):
         technology_results_parser.set_defaults(func=self.print_webanalyze_results)
         searchsploit_results_parser.set_defaults(func=self.print_searchsploit_results)
         port_results_parser.set_defaults(func=self.print_port_results)
+        tools_install_parser.set_defaults(func=self.tools_install)
+        tools_reinstall_parser.set_defaults(func=self.tools_reinstall)
+        tools_uninstall_parser.set_defaults(func=self.tools_uninstall)
+        tools_list_parser.set_defaults(func=self.tools_list)
 
     def _preloop_hook(self) -> None:
         """ Hook function that runs prior to the cmdloop function starting; starts the selector loop. """
@@ -336,11 +353,46 @@ def _get_dict(self):
 
         return tools
 
-    @cmd2.with_argparser(install_parser)
-    def do_install(self, args):
+    def _finalize_tool_action(self, tool: str, tool_dict: dict, return_values: List[int], action: ToolActions):
+        """ Internal helper to keep DRY
+
+        Args:
+            tool: tool on which the action has been performed
+            tool_dict: tools dictionary to save
+            return_values: accumulated return values of subprocess calls
+            action: ToolAction.INSTALL or ToolAction.UNINSTALL
+        """
+        verb = ["install", "uninstall"][action.value]
+
+        if all(x == 0 for x in return_values):
+            # all return values in retvals are 0, i.e. all exec'd successfully; tool action has succeeded
+
+            self.poutput(style(f"[+] {tool} {verb}ed!", fg="bright_green"))
+
+            tool_dict[tool]["installed"] = True if action == ToolAction.INSTALL else False
+        else:
+            # unsuccessful tool action
+
+            tool_dict[tool]["installed"] = False if action == ToolAction.INSTALL else True
+
+            self.poutput(
+                style(
+                    f"[!!] one (or more) of {tool}'s commands failed and may have not {verb}ed properly; check output from the offending command above...",
+                    fg="bright_red",
+                    bold=True,
+                )
+            )
+
+        # store any tool installs/failures (back) to disk
+        persistent_tool_dict = self.tools_dir / ".tool-dict.pkl"
+
+        pickle.dump(tool_dict, persistent_tool_dict.open("wb"))
+
+    def tools_install(self, args):
         """ Install any/all of the libraries/tools necessary to make the recon-pipeline function. """
 
         tools = self._get_dict()
+
         if args.tool == "all":
             # show all tools have been queued for installation
             [
@@ -350,7 +402,7 @@ def do_install(self, args):
             ]
 
             for tool in tools.keys():
-                self.do_install(tool)
+                self.do_tools(f"install {tool}")
 
             return
 
@@ -366,7 +418,17 @@ def do_install(self, args):
                 )
 
                 # install the dependency before continuing with installation
-                self.do_install(dependency)
+                self.do_tools(f"install {dependency}")
+
+        # this prevents a stale copy of tools when dependency installs alter the state
+        # ex.
+        #   amass (which depends on go) grabs copy of tools (go installed false)
+        #   amass calls install with go as the arg
+        #   go grabs a copy of tools
+        #   go is installed and state is saved (go installed true)
+        #   recursion goes back to amass call (go installed false due to stale tools data)
+        #   amass installs and re-saves go's state as installed=false
+        tools = self._get_dict()
 
         if tools.get(args.tool).get("installed"):
             return self.poutput(style(f"[!] {args.tool} is already installed.", fg="yellow"))
@@ -408,33 +470,12 @@ def do_install(self, args):
 
                 retvals.append(proc.returncode)
 
-        if all(x == 0 for x in retvals):
-            # all return values in retvals are 0, i.e. all exec'd successfully; tool has been installed
+        self._finalize_tool_action(args.tool, tools, retvals, ToolAction.INSTALL)
 
-            self.poutput(style(f"[+] {args.tool} installed!", fg="bright_green"))
-
-            tools[args.tool]["installed"] = True
-        else:
-            # unsuccessful tool install
-
-            tools[args.tool]["installed"] = False
-
-            self.poutput(
-                style(
-                    f"[!!] one (or more) of {args.tool}'s commands failed and may have not installed properly; check output from the offending command above...",
-                    fg="bright_red",
-                    bold=True,
-                )
-            )
-
-        # store any tool installs/failures (back) to disk
-        persistent_tool_dict = self.tools_dir / ".tool-dict.pkl"
-        pickle.dump(tools, persistent_tool_dict.open("wb"))
-
-    @cmd2.with_argparser(uninstall_parser)
-    def do_uninstall(self, args):
+    def tools_uninstall(self, args):
         """ Uninstall any/all of the libraries/tools used by recon-pipeline"""
         tools = self._get_dict()
+
         if args.tool == "all":
             # show all tools have been queued for installation
             [
@@ -444,7 +485,7 @@ def do_uninstall(self, args):
             ]
 
             for tool in tools.keys():
-                self.do_uninstall(tool)
+                self.do_tools(f"uninstall {tool}")
 
             return
 
@@ -454,6 +495,7 @@ def do_uninstall(self, args):
             retvals = list()
 
             self.poutput(style(f"[*] Removing {args.tool}...", fg="bright_yellow"))
+
             if not tools.get(args.tool).get("uninstall_commands"):
                 self.poutput(style(f"[*] {args.tool} removal not needed", fg="bright_yellow"))
                 return
@@ -470,28 +512,28 @@ def do_uninstall(self, args):
 
                 retvals.append(proc.returncode)
 
-        if all(x == 0 for x in retvals):
-            # all return values in retvals are 0, i.e. all exec'd successfully; tool has been uninstalled
+        self._finalize_tool_action(args.tool, tools, retvals, ToolAction.UNINSTALL)
 
-            self.poutput(style(f"[+] {args.tool} removed!", fg="bright_green"))
+    def tools_reinstall(self, args):
+        """ Reinstall a given tool """
+        self.do_tools(f"uninstall {args.tool}")
+        self.do_tools(f"install {args.tool}")
 
-            tools[args.tool]["installed"] = False
-        else:
-            # unsuccessful tool removal
-
-            tools[args.tool]["installed"] = True
+    def tools_list(self, args):
+        """ List status of pipeline tools """
+        for key, value in self._get_dict().items():
+            status = [style(":Missing:", fg="bright_magenta"), style("Installed", fg="bright_green")]
+            self.poutput(style(f"[{status[value.get('installed')]}] - {value.get('path') or key}"))
 
-            self.poutput(
-                style(
-                    f"[!!] one (or more) of {args.tool}'s commands failed and may have not been removed properly; check output from the offending command above...",
-                    fg="bright_red",
-                    bold=True,
-                )
-            )
+    @cmd2.with_argparser(tools_parser)
+    def do_tools(self, args):
+        """ Manage tool actions (install/uninstall/reinstall) """
+        func = getattr(args, "func", None)
 
-        # store any tool installs/failures (back) to disk
-        persistent_tool_dict = self.tools_dir / ".tool-dict.pkl"
-        pickle.dump(tools, persistent_tool_dict.open("wb"))
+        if func is not None:
+            func(args)
+        else:
+            self.do_help("tools")
 
     @cmd2.with_argparser(status_parser)
     def do_status(self, args):
@@ -825,7 +867,7 @@ def print_port_results(self, args):
     def do_view(self, args):
         """ View results of completed scans """
         if self.db_mgr is None:
-            return self.poutput(style("[!] you are not connected to a database", fg="magenta"))
+            return self.poutput(style("[!] you are not connected to a database", fg="bright_magenta"))
 
         func = getattr(args, "func", None)
 

diff --git a/pipeline/recon/__init__.py b/pipeline/recon/__init__.py
@@ -6,16 +6,19 @@
 from .nmap import ThreadedNmapScan, SearchsploitScan
 from .config import top_udp_ports, top_tcp_ports, defaults, web_ports
 from .parsers import (
-    install_parser,
-    uninstall_parser,
     scan_parser,
+    view_parser,
+    tools_parser,
     status_parser,
     database_parser,
     db_attach_parser,
     db_delete_parser,
     db_detach_parser,
     db_list_parser,
-    view_parser,
+    tools_list_parser,
+    tools_install_parser,
+    tools_uninstall_parser,
+    tools_reinstall_parser,
     target_results_parser,
     endpoint_results_parser,
     nmap_results_parser,