Although each component of the Grassroots infrastructure can be built in isolation, it can be much easier to manage all of the components together. That is what this package achieves.
There are various third party packages that Grassroots uses and there is an installation script to simplify this process.
The first stage is to make sure that the required OS packages are installed. On most systems the following packages are already installed, but you can make sure by running:
sudo apt install gcc wget automake unzip flex make git cmake zlib1g-dev g++ libzstd-dev libssl-dev ant
Check to see whether libcurl is installed by running the following command
locate -b "\libcurl.so"
If no file is found it can be installed from one of the following packages:
sudo apt install libcurl4-gnutls-dev
or
sudo apt install libcurl4-nss-dev
or
sudo apt install libcurl4-openssl-dev
You only need to install one of these so simply choose whichever one you prefer.
The other dependencies can be installed using the supplied installation script at build-config/unix/linux/install_dependencies.
The first stage is to edit this file and set the path that you want to install the Grassroots dependencies to. This is specified by the GRASSROOTS_EXTRAS_INSTALL_PATH variable within this file.
Once you have set this to the path that you would like to install to, you can run this script e.g.
unix/linux/install_dependencies
The various libraries annd tools that are part of Grassroots are built from source so you need to create the structure to store them. This is not where the actual files that are generated will be installed to, this is simply a place to generate them. The default layout for the subdirectories containing the Grassroots can be achieved by:
mkdir grassroots
cd grassroots
git clone https://github.com/TGAC/grassroots-build-tools.git build-config
git clone https://github.com/TGAC/grassroots-core.git core
git clone https://github.com/TGAC/grassroots-lucene.git lucene
mkdir clients
mkdir handlers
mkdir libs
mkdir servers
mkdir services
cd servers
git clone https://github.com/TGAC/grassroots-server-apache-httpd.git httpd-server
git clone https://github.com/TGAC/grassroots-jobs-manager-mongodb.git mongodb-jobs-manager
git clone https://github.com/TGAC/grassroots-simple-servers-manager.git simple-servers-manager
which gives the layout shown below
grassroots
|
|--- build-config
|
|--- clients
|
|--- core
|
|--- handlers
|
|--- libs
|
|--- lucene
|
|--- servers
|
|--- httpd-server
|
|--- mongodb-jobs-manager
|
|--- simple-servers-manager
|
|--- services
Where any client, server or service components can be cloned inside the clients, handlers, servers or services directories respectively.
You can view the available repositories of each type by going to one of the links below:
The next stage is to set your preferences which is done in a file called build-config/unix/<PLATFORM NAME>/dependencies.properties. For example if you are running on Linux, then this would be build-config/unix/linux/dependencies.properties, whereas on a mac it would be build-config/unix/mac/dependencies.properties. An example one is provided so you can get an initial version of your preferences file by making a copy of this by doing
cd build-config
cp unix/linux/example-dependencies.properties unix/linux/dependencies.properties
We can now proceed to amending this file, unix/linux/dependencies.properties, to your chosen layout.
The three main variables are:
-
DIR_GRASSROOTS_INSTALL: This is the Grassroots folder where the libraries, services, configuration, etc. will be stored. So to set to this to
/opt/grassroots, the setting would beexport DIR_GRASSROOTS_INSTALL := /opt/grassroots -
DIR_APACHE: This is the path to your Apache Httpd installation. So to set to this to
/opt/apache, the setting would beexport DIR_APACHE := /opt/apache
If however you are using an Apache Httpd that is spread across different directories, such as is often the case when using the OS package manager, then simply leave this blank
```export DIR_APACHE :=```
-
DIR_GRASSROOTS_EXTRAS: This should be set to the value you used for
GRASSROOTS_EXTRAS_INSTALL_PATHwhen you installed the dependencies. So to set to this to/opt/grassroots/extras, the setting would beexport DIR_GRASSROOTS_EXTRAS := /opt/grassroots/extras
If you have iRODS installed and you wish to the Grassroots iRODS functionality such as the search service, etc., then this can be configured here too.
The variables are:
-
IRODS_ENABLED: Set this to 0 to disable the iRODS support and 1 to enable it. For example, to enable the iRODS functionality within Grassroots:
export IRODS_ENABLED := 1 -
IRODS_VERSION: This specifies the major version of the iRODS installation. For recent installations this will be 4.x, so the setting is
export IRODS_VERSION := 4
Grassroots has support for running jobs on high performance computing clusters using DRMAA via LSF, Slurm or HTCondor. When Grassroots is built, you have to specify which system, if any, you wish to use it with. The 3 variables are SLURM_DRMAA_ENABLED, LSF_DRMAA_ENABLED and HTCONDOR_DRMAA_ENABLED and setting them to 0 disables and setting one of them to 1 enables the DRMAA support for that system. So for example, to disable all DRMAA support, the configuration would be:
export SLURM_DRMAA_ENABLED := 0
export LSF_DRMAA_ENABLED := 0
export HTCONDOR_DRMAA_ENABLED := 0
To enable HTCondor support, the configuration would be:
export SLURM_DRMAA_ENABLED := 0
export LSF_DRMAA_ENABLED := 0
export HTCONDOR_DRMAA_ENABLED := 1
Once you have finished setting up the dependencies.properties file, you can then proceed to building Grassroots.
Although any web server could potentially be used as Grassroots server, the default is to use the Apache httpd web server as it has the the benefit of being able to add functionality easily with its modular approach.
On Linux, there are 2 options for installing httpd; using the OS package manager or building it from source
To install httpd using the package manager on Debian and its derivatives such as Ubuntu, Mint, etc., you can run the following commands
sudo apt install apache2
sudo apt install apache2-dev
sudo a2enmod cache socache socache_shmcb
To build the Grassroots system, you need to specify the build subfolder which is unix/<PLATFORM>. So, for example,
to build it for Linux, the folder is unix/linux and for MacOS it is unix/mac. These are specified using the
-C argument for make
For example for Linux, the value is
make -C unix/linux
and for MacOS
make -C unix/mac
The following examples assume you are on Linux but just replace the -C parameter if you are on MacOS
To build the platform, you run
make -C unix/linux all
and the system will proceed to build. To then install it, type
make -C unix/linux install
As well as the standard targets to build the Grassroots system, the makefile has a number of other targets to perform useful operations. These are listed below.
If you wish to get the latest versions of all of the Grassroots repositories that you have checked out, you can run
make -C unix/linux git-pull
to run git pull on all of the Grassroots repositories that you have checked out.
If you are are developing some of the Grassroots repositories and you wish to see which projects have local changes you can run
make -C unix/linux git-check
to list all of the repositories that have changes.
If you are having problems building Grassroots which revolve around missing files, you can check the configuration of which folders Grassroots is using to build itself by typing:
make -C unix/linux show-config