Alfresco Search Services using Alfresco and Apache Solr
Git:
$ git clone https://github.com/Alfresco/SearchServices.gitBuild the project:
$ mvn clean install -DskipTests=trueAll the resources required to run Alfresco Search Services will be available under packaging/target folder.
To run Alfresco Search Services locally, building the ZIP distribution file is required.
$ mvn clean install -DskipTests=trueAfter the project is successfully built, ZIP can be extracted.
$ cd packaging/target
$ unzip alfresco-search-services-*.zip
$ cd alfresco-search-servicesFrom Alfresco Search Services 1.3.0.5, distribution ZIP is released with Mutual Authentication TLS (SSL) by default. So before starting the service, generating secure keys for SSL communication is required. You can find detailed information for this step at Alfresco documentation.
The keystores folder generated by the SSL Tool contains the keystores and truststores for SSL configuration. In the following steps, it's assumed that SSL Tool has been executed from /tmp or C:\tmp folder.
$ tree /tmp/keystores/
keystores/
├── alfresco
│ ├── keystore
│ ├── keystore-passwords.properties
│ ├── ssl-keystore-passwords.properties
│ ├── ssl-truststore-passwords.properties
│ ├── ssl.keystore
│ └── ssl.truststore
├── client
│ └── browser.p12
├── solr
│ ├── ssl-keystore-passwords.properties
│ ├── ssl-truststore-passwords.properties
│ ├── ssl.repo.client.keystore
│ └── ssl.repo.client.truststore
└── zeppelin
├── ssl.repo.client.keystore
└── ssl.repo.client.truststoreSOLR SSL configuration files are available in /tmp/keystores/solr folder.
These files must be copied to rerank configuration folder.
$ cp /tmp/keystores/solr/* solrhome/templates/rerank/conf
If you are running from a Linux or Mac OS X machine, add following lines to solr.in.sh file.
SOLR_SSL_KEY_STORE=/tmp/keystores/solr/ssl.repo.client.keystore
SOLR_SSL_KEY_STORE_PASSWORD=keystore
SOLR_SSL_KEY_STORE_TYPE=JCEKS
SOLR_SSL_TRUST_STORE=/tmp/keystores/solr/ssl.repo.client.truststore
SOLR_SSL_TRUST_STORE_PASSWORD=truststore
SOLR_SSL_TRUST_STORE_TYPE=JCEKS
SOLR_SSL_NEED_CLIENT_AUTH=true
SOLR_SSL_WANT_CLIENT_AUTH=false
If you are running from a Windows machine, add following lines to solr.in.cmd file.
set SOLR_SSL_KEY_STORE=C:\tmp\keystores\solr\ssl.repo.client.keystore
SOLR_SSL_KEY_STORE_PASSWORD=keystore
SOLR_SSL_KEY_STORE_TYPE=JCEKS
SOLR_SSL_TRUST_STORE=C:\tmp\keystores\solr\ssl.repo.client.truststore
SOLR_SSL_TRUST_STORE_PASSWORD=truststore
SOLR_SSL_TRUST_STORE_TYPE=JCEKS
SOLR_SSL_NEED_CLIENT_AUTH=true
SOLR_SSL_WANT_CLIENT_AUTH=false
Once this settings are ready, start SOLR service from command line:
$ ./solr/bin/solr start -a "-Dcreate.alfresco.defaults=alfresco,archive \
-Dsolr.ssl.checkPeerName=false \
-Dsolr.allow.unsafe.resourceloading=true" -f
SOLR will create Alfresco cores (alfresco and archive) when starting, and configuration from rerank template will be copied to each core.
If you also started an ACS instance running in https://localhost:8443/alfresco with the keystores provided by the SSL Tool (keystores/alfresco folder), then the index will be populated.
SOLR Web Console will be available at:
Note Client certificate browser.p12, generated by the SSL Tool, is required to be installed in your browser in order to access to this Web Console. See "Installing Browser certificate" section below.
By default Alfresco Search Services runs on port 8983, but this can be set by supplying e.g. -p 8083 to the "solr start" command.
To set up remote debugging (on port 5005) start Alfresco Search Services with the following command and then connect using your IDE:
$ ./solr/bin/solr start -a "-Dcreate.alfresco.defaults=alfresco,archive \
-Dsolr.ssl.checkPeerName=false \
-Dsolr.allow.unsafe.resourceloading=true \
-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005" -fDBID based sharding can be set up from the command line. For example a core containing shards 0, 1, 6 and 7 from an index with twelve shards can be set up by starting an instance of Alfresco Search Services with a command like:
./bin/solr start -a "-Dcreate.alfresco.defaults=alfresco,archive -Dnum.shards=12 -Dshard.ids=0,1,6,7"Further instances should be set up to contain the other shards, and it is possible to adjust the distribution and replication of shards to achieve the desired index performance and redundancy.
To stop Alfresco Search Services:
$ ./solr/bin/solr stopUsing Plain HTTP
If you want to use Plain HTTP for SOLR instead of Mutual Auth TLS (SSL), use following steps.
$ mvn clean install -DskipTests=trueAfter the project is successfully built, ZIP can be extracted.
$ cd packaging/target
$ unzip alfresco-search-services-*.zip
$ cd alfresco-search-servicesChange default Alfresco Communication protocol to none, and set alfresco.allowUnauthenticatedSolrEndpoint to true:
$ sed -i 's/alfresco.secureComms=https/alfresco.secureComms=none\nalfresco.allowUnauthenticatedSolrEndpoint=true/' solrhome/templates/rerank/conf/solrcore.propertiesNote Above line is written in GNU sed, you can use gsed from Mac OS X or just edit the file with a Text Editor.
Start SOLR service from command line:
$ ./solr/bin/solr start -a "-Dcreate.alfresco.defaults=alfresco,archive" -f
SOLR will create Alfresco cores (alfresco and archive) when starting, and configuration from rerank template will be copied to each core and if you also started an ACS instance running in http://localhost:8080/alfresco then the index will be populated.
SOLR Web Console will be available at:
In order to access to SOLR Web Console, available by default at https://localhost:8983/solr, browser certificate must be installed in your machine.
For Windows systems, client\browser.p12 file must be imported as new private certificate to Windows Certificates application.
For Mac OS X systems, client/browser.p12 file must be imported to Keychain Access application.
Also setting the right options in these application to trust in this certificate is required.
Once the certificate is installed, the following message should be showed by your browser when accessing to Solr Web Console:
Your connection is not private
Attackers might be trying to steal your information from localhost (for example, passwords, messages or credit cards). Learn more
NET::ERR_CERT_AUTHORITY_INVALID
As the certificate has been generated for localhost, this warning is expected. Just click on Advanced >> Proceed and use your browser certificate to access Solr Web Console.
Once the project has been built, the Docker image can be also built:
$ cd packaging/target/docker-resources/
$ docker build -t searchservices:develop .Search Services Docker image is configured with Mutual Authentication TLS (SSL) by default.
Building Docker Image from Windows
When building Search Services or Insight Engine Docker Images from Windows, some steps need to be added to default building process.
Clone the repository preserving Linux line endings
$ git clone [email protected]:search_discovery/insightengine.git --config core.autocrlf=input
Alternatively you can use global settings before cloning the repository
$ git config --global core.autocrlf input
$ git config --list
core.autocrlf=input
$ git clone https://git.alfresco.com/search_discovery/insightengine.git
Build the Maven project
$ mvn clean package -DskipTests
Modify default Dockerfile
$ cd insight-engine/packaging/target/docker-resources/
Replace the last lines (from EXPOSE to CMD) of Dockerfile file in this folder with the following ones:
EXPOSE 8983
RUN chown -R solr:solr $DIST_DIR
RUN chmod +x $DIST_DIR/solr/bin/*
RUN set -x \
&& yum install -y dos2unix \
&& yum clean all
RUN dos2unix $DIST_DIR/solr.in.sh
USER ${USERNAME}
CMD $DIST_DIR/solr/bin/search_config_setup.sh "$DIST_DIR/solr/bin/solr start -f"
And build the Docker Image.
$ docker build -t insightengine:develop .
Configuration
The "-e" argument can be used to pass an environment variable:
$ docker run -e SOLR_JAVA_MEM="-Xms4g -Xmx4g" -p 8983:8983 searchservices:developTo pass several environment variables (e.g. SOLR_ALFRESCO_HOST, SOLR_ALFRESCO_PORT, SOLR_SOLR_HOST, SOLR_SOLR_PORT, SOLR_CREATE_ALFRESCO_DEFAULTS, SOLR_HEAP, etc.), just include the "-e" argument as times as required:
$ docker run -e SOLR_ALFRESCO_HOST=localhost -e SOLR_ALFRESCO_PORT=8080 -p 8983:8983 searchservices:developThe following environment variables are supported:
| Name | Format | Description |
|---|---|---|
| SOLR_OPTS | "-Dparam=value ..." | Options to pass when starting the Java process. |
| SOLR_HEAP | Memory amount (e.g. 2g) | The Java heap assigned to Solr. |
| SOLR_JAVA_MEM | "-Xms... -Xmx..." | The exact memory settings for Solr. Note that SOLR_HEAP takes precedence over this. |
| MAX_SOLR_RAM_PERCENTAGE | Integer | The percentage of available memory to assign to Solr. Note that SOLR_HEAP and SOLR_JAVA_MEM take precedence over this. |
| SEARCH_LOG_LEVEL | ERROR, WARN, INFO, DEBUG or TRACE | The root logger level. |
| ENABLE_SPELLCHECK | true or false | Whether spellchecking is enabled or not. |
| DISABLE_CASCADE_TRACKING | true or false | Whether cascade tracking is enabled or not. Disabling cascade tracking will improve performance, but result in some feature loss (e.g. path queries). |
| SOLR_SSL_... | --- | These variables are also used to configure SSL. See below. |
| ALFRESCO_SECURE_COMMS | secret or https | This property instructs Solr if it should enable Shared Secret authentication or mTLS authentication with HTTPS. See below. |
Using Mutual Auth TLS (SSL)
This Docker image is exposing as VOLUME the folder /opt/alfresco-search-services/keystores, that can be used to mount keystores folder from host.
By default Docker image is using SSL, but an environment variable ALFRESCO_SECURE_COMMS=https can be also passed to the Docker container to declare explicitly the SSL mode.
Additionally, SOLR Jetty server must be configured to start in SSL Mode using SOLR_SSL_* environment variables and Search Services must be configured by using Java environment variables starting with alfresco.encryption.ssl.*
Following command will start Search Services with SSL using keystores located at /tmp/keystores/solr. Note that the internal folders are relative to /opt/alfresco-search-services/keystores, as this is the Docker container folder exposed to hold the keystores.
$ docker run -p 8983:8983 \
-v /tmp/keystores/solr:/opt/alfresco-search-services/keystores \
-e SOLR_CREATE_ALFRESCO_DEFAULTS=alfresco,archive \
-e SOLR_SSL_KEY_STORE=/opt/alfresco-search-services/keystores/ssl.repo.client.keystore \
-e SOLR_SSL_KEY_STORE_PASSWORD=keystore \
-e SOLR_SSL_KEY_STORE_TYPE=JCEKS \
-e SOLR_SSL_TRUST_STORE=/opt/alfresco-search-services/keystores/ssl.repo.client.truststore \
-e SOLR_SSL_TRUST_STORE_PASSWORD=truststore \
-e SOLR_SSL_TRUST_STORE_TYPE=JCEKS \
-e SOLR_SSL_NEED_CLIENT_AUTH=true \
-e SOLR_OPTS="-Dsolr.ssl.checkPeerName=false \
-Dsolr.allow.unsafe.resourceloading=true" \
searchservices:developSOLR Web Console will be available at:
Note You must install the browser.p12 certificate in your browser in order to access to this URL.
Using Shared Secret Authentication
An alternative is to use a shared secret in order to secure repo <-> solr communication. You just need to set ALFRESCO_SECURE_COMMS=secret AND JAVA_TOOL_OPTIONS="-Dalfresco.secureComms.secret=my_super_secret_secret".
By default, the SOLR Web Console will be available at:
but you can also start the Jetty server in SSL mode as explained above, in that case the SOLR Web Console will be available at:
Note You must install the browser.p12 certificate in your browser in order to access to this URL.
In both cases, when trying to access the SOLR Web Console you will have to provide the X-Alfresco-Search-Secret header in the request, specifying as its value the same value that was used for the -Dalfresco.secureComms.secret property.
You can do so natively on Safari through the Dev Tools > Local Overrides feature, or with a browser extension on Google Chrome/Firefox/Opera/Edge: ModHeader.
Using Shared Secret Authentication
By default Docker image is using SSL, so it's required to add an environment variable ALFRESCO_SECURE_COMMS=secret AND JAVA_TOOL_OPTIONS="-Dalfresco.secureComms.secret=my_super_secret_secret" to use SOLR with Shared Secret authentication.
To run the docker image:
$ docker run -p 8983:8983 -e ALFRESCO_SECURE_COMMS=secret -e SOLR_CREATE_ALFRESCO_DEFAULTS=alfresco,archive -e JAVA_TOOL_OPTIONS="-Dalfresco.secureComms.secret=my_super_secret_secret" searchservices:developSOLR Web Console will be available at:
You will have to provide the X-Alfresco-Search-Secret header in the request, specifying as its value the same value that was used for the -Dalfresco.secureComms.secret property.
Enabling YourKit Java Profiler
This Docker Image includes YourKit Java Profiler server service. In order to enable this service, so the SOLR JVM can be inspected with the YourKit local program, additional configuration is required to set the YourKit agentpath. Mapping the exposed profiling port (10001 by default) is also required.
Sample Docker command line invocation
$ docker run -p 8983:8983 -p 10001:10001 \
-e SOLR_CREATE_ALFRESCO_DEFAULTS=alfresco,archive \
-e SOLR_OPTS="-agentpath:/usr/local/YourKit-JavaProfiler-2019.8/bin/linux-x86-64/libyjpagent.so=port=10001,listen=all" \
searchservices:developSample Docker Compose service settings
solr6:
image: searchservices:develop
mem_limit: 2500m
environment:
# Solr needs to know how to register itself with Alfresco
SOLR_ALFRESCO_HOST: "alfresco"
SOLR_ALFRESCO_PORT: "8080"
# Alfresco needs to know how to call solr
SOLR_SOLR_HOST: "solr6"
SOLR_SOLR_PORT: "8983"
# HTTP settings
ALFRESCO_SECURE_COMMS: "secret"
#Create the default alfresco and archive cores
SOLR_CREATE_ALFRESCO_DEFAULTS: "alfresco,archive"
SOLR_JAVA_MEM: "-Xms2g -Xmx2g"
SOLR_OPTS: "
-agentpath:/usr/local/YourKit-JavaProfiler-2019.8/bin/linux-x86-64/libyjpagent.so=port=10001,listen=all
"
JAVA_TOOL_OPTIONS: "
-Dalfresco.secureComms.secret=my_super_secret_secret
"
ports:
- 8083:8983 #Browser port
- 10001:10001 #YourKit portPublic Docker repository
This Docker Image is available at Alfresco Docker Hub:
https://hub.docker.com/r/alfresco/alfresco-search-services
To use the public image instead of the local one (searchservices:develop) just use alfresco/alfresco-search-services:1.3.x.x labels.
To enable slave node specify environment value REPLICATION_TYPE=slave, by default Master config is enabled and slave is disabled.
During deployment time whenever Search Services or Insight Engine image starts, it will execute the script search_config_setup.sh which will configure the slave config setup based on the value specified in the script.
To run the docker image:
$ docker run -p 8984:8983 -e REPLICATION_TYPE=slave -e ALFRESCO_SECURE_COMMS=secret -e SOLR_CREATE_ALFRESCO_DEFAULTS=alfresco,archive -e JAVA_TOOL_OPTIONS="-Dalfresco.secureComms.secret=my_super_secret_secret" searchservices:developSolr-slave End point: http://localhost:8984/solr
To generate your own Docker-compose file please follow generator-alfresco-docker-compose
Sample configuration in a Docker Compose file using Shared Secret Authentication to communicate with Alfresco Repository.
solr6:
image: searchservices:develop
mem_limit: 2500m
environment:
# Solr needs to know how to register itself with Alfresco
SOLR_ALFRESCO_HOST: "alfresco"
SOLR_ALFRESCO_PORT: "8080"
# Alfresco needs to know how to call solr
SOLR_SOLR_HOST: "solr6"
SOLR_SOLR_PORT: "8983"
# HTTP settings
ALFRESCO_SECURE_COMMS: "secret"
#Create the default alfresco and archive cores
SOLR_CREATE_ALFRESCO_DEFAULTS: "alfresco,archive"
SOLR_JAVA_MEM: "-Xms2g -Xmx2g"
JAVA_TOOL_OPTIONS: "
-Dalfresco.secureComms.secret=my_super_secret_secret
"
ports:
- 8083:8983 #Browser port
SOLR Web Console will be available at:
You will have to provide the X-Alfresco-Search-Secret header in the request, specifying as its value the same value that was used for the -Dalfresco.secureComms.secret property.
Sample configuration in a Docker Compose file using Mutual Auth TLS (SSL) protocol to communicate with Alfresco Repository.
solr6:
image: searchservices:develop
mem_limit: 2500m
environment:
# Solr needs to know how to register itself with Alfresco
SOLR_ALFRESCO_HOST: "alfresco"
SOLR_ALFRESCO_PORT: "8443"
# Alfresco needs to know how to call solr
SOLR_SOLR_HOST: "solr6"
SOLR_SOLR_PORT: "8983"
# SSL settings
ALFRESCO_SECURE_COMMS: "https"
SOLR_SSL_TRUST_STORE: "/opt/alfresco-search-services/keystore/ssl.repo.client.truststore"
SOLR_SSL_TRUST_STORE_PASSWORD: "truststore"
SOLR_SSL_TRUST_STORE_TYPE: "JCEKS"
SOLR_SSL_KEY_STORE: "/opt/alfresco-search-services/keystore/ssl.repo.client.keystore"
SOLR_SSL_KEY_STORE_PASSWORD: "keystore"
SOLR_SSL_KEY_STORE_TYPE: "JCEKS"
SOLR_SSL_NEED_CLIENT_AUTH: "true"
#Create the default alfresco and archive cores
SOLR_CREATE_ALFRESCO_DEFAULTS: "alfresco,archive"
SOLR_JAVA_MEM: "-Xms2g -Xmx2g"
SOLR_OPTS: "
-Dsolr.ssl.checkPeerName=false
-Dsolr.allow.unsafe.resourceloading=true
"
ports:
- 8083:8983 #Browser port
volumes:
- ./keystores/solr:/opt/alfresco-search-services/keystores
SOLR Web Console will be available at:
Samples for development use only
docker-compose files can be used to start up Search Services with Alfresco and Share. There are two docker-composes files available. Depending on the version you want to start either change to 5.x or 6.x. E.g.
cd packaging/target/docker-resources/6.x
docker compose up##Docker Master-Slave setup We have seperate docker compose file for slave. To setup Master slave setup
docker compose -f docker-compose.yml -f ./master-slave/docker-compose.slave.yml up
The slave running behind the nginx load balancer under 8084, so we can spin up multiple slaves with the same port. To deploy multiple slaves
docker compose -f docker-compose.yml -f ./master-slave/docker-compose.slave.yml up --scale search_slave=2
This will start up Alfresco, Postgres, Share and SearchServices. You can access the applications using the following URLs:
- Alfresco: http://localhost:8081/alfresco
- Share: http://localhost:8082/share
- Solr: http://localhost:8083/solr
- Solr-slave: http://localhost:8084/solr
Note: Once deployed and all services up and running, goto http://localhost:8081/alfresco/s/enterprise/admin/admin-searchservice?m=admin-console.success. Scroll down and click save button. This is the bug which currently working on SEARCH-2085
If you start version 5.x instead you can also access the API Explorer:
- API Explorer: http://localhost:8084/api-explorer
Copyright (C) 2005 - 2017 Alfresco Software Limited
This file is part of the Alfresco software. If the software was purchased under a paid Alfresco license, the terms of the paid license agreement will prevail. Otherwise, the software is provided under the following open source license terms:
Alfresco is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
Alfresco is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public License along with Alfresco. If not, see http://www.gnu.org/licenses/.