How to install Manticoresearch with Docker and Postgresql?

[jjoloka]

How to install Manticoresearch with Docker and Postgresql indexer (pg table public.[a_block_element])?

docker-compose.yml

version: '2.2'
services:
  manticore:
    container_name: manticore
    image: manticoresearch/manticore
    environment:
      - EXTRA=1
    restart: always
    ports:
      - 127.0.0.1:9306:9306
      - 127.0.0.1:9308:9308
    ulimits:
      nproc: 65535
      nofile:
         soft: 65535
         hard: 65535
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./manticore/data:/var/lib/manticore/data
      - ./manticore/etc:/etc/manticoresearch
      - ./manticore/logs:/var/log/manticore

Error:
manticore | [Tue Feb 6 12:21:42.319 2024] [1] using config file '/etc/manticoresearch/manticore.conf' (254 chars)...
manticore | [Tue Feb 6 12:21:42.319 2024] [1] FATAL: 'searchd' config section not found in '/etc/manticoresearch/manticore.conf'

/home/ubuntu/manticore

/home/ubuntu/manticore/etc/manticore.conf

source text_src {
  type = pgsql
  sql_host = 192.168.0.86
  sql_user = postgres
  sql_pass = qazQASD!
  sql_db = api_db
  sql_query = SELECT id, name from a_block_element
}

index text_idx {
  type = plain
  source = text_src
  path = home/ubuntu/data/full_search
}

[jjoloka]

Last config:

source text_src {
  type = pgsql
  sql_host = 192.168.0.86
  sql_user = postgres
  sql_pass = qazQASD!
  sql_db = api_db
  sql_query = SELECT id, name from a_block_element
}

index text_idx {
        type = plain
        source = text_src
        path = /home/ubuntu/manticore/data
}

searchd {
    listen = 127.0.0.1:9312
    listen = 127.0.0.1:9308:http
    pid_file = /var/run/manticore/searchd.pid
}     

Error:
manticore | WARNING: Unable to load header... Error failed to open /home/ubuntu/manticore/data.sph: No such file or directory
manticore | WARNING: table 'text_idx': prealloc: failed to open /home/ubuntu/manticore/data.sph: No such file or directory - NOT SERVING

[sanikolaev]

No such file or directory - NOT SERVING is ok. Build the table, rotate it and the searchd will start serving it. Read https://github.com/manticoresoftware/docker/blob/bc48debe5c1ce3b8551f521f646a2921f4fade59/README.md#building-plain-tables and https://github.com/manticoresoftware/docker/blob/bc48debe5c1ce3b8551f521f646a2921f4fade59/README.md#configuring-manticore-search-with-docker (about gosu manticore indexer) on how to build plain tables in docker.

[jjoloka]

CREATE_PLAIN_TABLES=1 - can be added to docker-compose.yml?

[sanikolaev]

Sure, why not?

[jjoloka]

We change docker-compose.yml, and get an error

docker-compose.yml:

version: '2.2'
  
services:
  manticore:
    container_name: manticore
    image: manticoresearch/manticore
    environment:
      - EXTRA=1
      - CREATE_PLAIN_TABLES=1
    restart: always
    ports:
      - 127.0.0.1:9306:9306
      - 127.0.0.1:9308:9308
    ulimits:
      nproc: 65535
      nofile:
         soft: 65535
         hard: 65535
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./manticore/data:/var/lib/manticore/data
      - ./manticore/etc:/etc/manticoresearch
      - ./manticore/logs:/var/log/manticore

What are we doing wrong?

[sanikolaev]

How do I reproduce it?

[jjoloka]

How do I reproduce it?

1. Create docker-compose.yml in ~, with this content:

version: '2.2'
  
services:
  manticore:
    container_name: manticore
    image: manticoresearch/manticore
    environment:
      - EXTRA=1
      - CREATE_PLAIN_TABLES=1
    restart: always
    ports:
      - 127.0.0.1:9306:9306
      - 127.0.0.1:9308:9308
    ulimits:
      nproc: 65535
      nofile:
         soft: 65535
         hard: 65535
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./manticore/data:/var/lib/manticore/data
      - ./manticore/etc:/etc/manticoresearch
      - ./manticore/logs:/var/log/manticore

2. install Postgresql, create user/password (postgres/qazQASD!), create DB (api_db), create table (a_block_element) with structure ([id] - sequence not null incriment, [name] - varchar, [text] - jsonb), add test data.

3. Create manticore.conf In ~/manticore/etc/manticore.conf

source text_src {
        type = pgsql
        sql_user = postgres
        sql_pass = qazQASD!
        sql_db = api_db
        sql_port = 5432
        sql_query = SELECT id, name from a_block_element
}

index text_idx {
        type = plain
        source = text_src
        path = /home/ubuntu/manticore/data
}

searchd {
    listen = 127.0.0.1:9312
    listen = 127.0.0.1:9306:mysql
    listen = 127.0.0.1:9308:http
    pid_file = /var/run/manticore/searchd.pid
}

4. Start docker-compose
   stop docker compose: $ docker-compose down
   remove the container: $ docker system prune -a
   start docker compose: $ docker-compose up

Current permissions:

[sanikolaev]

It looks like the main problem was in:

      - ./manticore/etc:/etc/manticoresearch

since it removes /etc/manticoresearch/manticore.conf.sh internally which is required.

If I do:

      - ./manticore/etc/manticore.conf:/etc/manticoresearch/manticore.conf

instead it works fine. Here's the whole test case:

Dir structure:

snikolaev@dev2:~/1801$ find .
.
./manticore
./manticore/etc
./manticore/etc/manticore.conf
./manticore/data
./manticore/logs
./docker-compose.yml
./init.sql

Postgres init file:

snikolaev@dev2:~/1801$ cat init.sql
\c api_db;

CREATE TABLE a_block_element (
  id SERIAL PRIMARY KEY,
  name VARCHAR NOT NULL,
  text JSONB
);

INSERT INTO a_block_element (name, text) VALUES
('Element 1', '{"key1": "value1"}'),
('Element 2', '{"key2": "value2"}');

docker-compose yaml:

snikolaev@dev2:~/1801$ cat docker-compose.yml
version: '2.2'

services:
  manticore:
    container_name: manticore
    image: manticoresearch/manticore
    environment:
      - EXTRA=1
      - CREATE_PLAIN_TABLES=1
    restart: always
    ports:
      - 127.0.0.1:39306:9306
      - 127.0.0.1:39308:9308
    ulimits:
      nproc: 65535
      nofile:
         soft: 65535
         hard: 65535
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./manticore/data:/var/lib/manticore/
      - ./manticore/logs:/var/log/manticore
      - ./manticore/etc/manticore.conf:/etc/manticoresearch/manticore.conf

  postgres:
    image: postgres:latest
    container_name: postgres
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: qazQASD!
      POSTGRES_DB: api_db
    ports:
      - "5432:5432"
    volumes:
      - ./init.sql:/docker-entrypoint-initdb.d/init.sql

Manticore config:

snikolaev@dev2:~/1801$ cat ./manticore/etc/manticore.conf
source text_src {
        type = pgsql
        sql_host = postgres
        sql_user = postgres
        sql_pass = qazQASD!
        sql_db = api_db
        sql_port = 5432
        sql_query = SELECT id, name from a_block_element
}

index text_idx {
        type = plain
        source = text_src
        path = /var/lib/manticore/data
}

searchd {
    listen = 127.0.0.1:9312
    listen = 127.0.0.1:9306:mysql
    listen = 127.0.0.1:9308:http
    pid_file = /var/run/manticore/searchd.pid
}

Starting postgres:

snikolaev@dev2:~/1801$ docker-compose up postgres
Creating network "1801_default" with the default driver
Creating postgres ... done
Attaching to postgres
postgres     | The files belonging to this database system will be owned by user "postgres".
...
postgres     | 2024-06-20 04:50:56.058 UTC [1] LOG:  database system is ready to accept connections

Starting Manticore:

In another tab:

snikolaev@dev2:~/1801$ docker-compose up manticore
Creating manticore ... done
Attaching to manticore
manticore    | Install extra packages
manticore    |
manticore    |      0K .......... .......... .......... .......... ..........  0% 70.7M 0s
manticore    |     50K .......... .......... .......... .......... ..........  0% 86.3M 0s
...
manticore    | indexing table 'text_idx'...
manticore    | collected 2 docs, 0.0 MB
manticore    | creating secondary index
manticore    | creating lookup: 0.0 Kdocs, 100.0% done
manticore    | sorted 0.0 Mhits, 100.0% done
manticore    | total 2 docs, 18 bytes
manticore    | total 0.078 sec, 229 bytes/sec, 25.50 docs/sec
manticore    | total 3 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
manticore    | total 15 writes, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
manticore    | [Thu Jun 20 04:51:17.197 2024] [1] using config file '/etc/manticoresearch/manticore.conf.sh' (490 chars)...
manticore    | starting daemon version '6.3.0 1811a9efb@24052209 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)' ...
manticore    | listening on 127.0.0.1:9312 for sphinx and http(s)
manticore    | listening on 127.0.0.1:9306 for mysql
manticore    | listening on 127.0.0.1:9308 for sphinx and http(s)
manticore    | Manticore 6.3.0 1811a9efb@24052209 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)
manticore    | Copyright (c) 2001-2016, Andrew Aksyonoff
manticore    | Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
manticore    | Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)
manticore    |
manticore    | precaching table 'text_idx'
manticore    | prereading 1 tables
manticore    | preread 1 tables in 0.001 sec
manticore    | accepting connections
manticore    | [BUDDY] started v2.3.10 '/usr/share/manticore/modules/manticore-buddy/bin/manticore-buddy --listen=http://127.0.0.1:9312 --bind=127.0.0.1  --threads=32 --skip=manticoresoftware/buddy-plugin-sharding --skip=manticoresoftware/buddy-plugin-queue' at http://127.0.0.1:33525
manticore    | [BUDDY] Loaded plugins:
manticore    | [BUDDY]   core: empty-string, backup, emulate-elastic, create, insert, alias, select, show, cli-table, plugin, test, alter-distributed-table, alter-rename-table, modify-table, knn, replace
manticore    | [BUDDY]   local:
manticore    | [BUDDY]   extra:

Restarting Manticore:

^CGracefully stopping... (press Ctrl+C again to force)
Stopping manticore ... done

snikolaev@dev2:~/1801$ docker-compose up manticore
Starting manticore ... done
Attaching to manticore
manticore    | Manticore 6.3.0 1811a9efb@24052209 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)
manticore    | Manticore 6.3.0 1811a9efb@24052209 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)
manticore    | Copyright (c) 2001-2016, Andrew Aksyonoff
manticore    | Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
manticore    | Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)
manticore    |
manticore    | using config file '/etc/manticoresearch/manticore.conf'...
manticore    | indexing table 'text_idx'...
manticore    | collected 2 docs, 0.0 MB
manticore    | creating secondary index
manticore    | creating lookup: 0.0 Kdocs, 100.0% done
manticore    | sorted 0.0 Mhits, 100.0% done
manticore    | total 2 docs, 18 bytes
manticore    | total 0.108 sec, 165 bytes/sec, 18.36 docs/sec
manticore    | total 3 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
manticore    | total 15 writes, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
manticore    | [Thu Jun 20 04:52:10.058 2024] [1] using config file '/etc/manticoresearch/manticore.conf.sh' (490 chars)...
manticore    | starting daemon version '6.3.0 1811a9efb@24052209 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)' ...
manticore    | listening on 127.0.0.1:9312 for sphinx and http(s)
manticore    | listening on 127.0.0.1:9306 for mysql
manticore    | listening on 127.0.0.1:9308 for sphinx and http(s)
manticore    | Manticore 6.3.0 1811a9efb@24052209 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)
manticore    | Copyright (c) 2001-2016, Andrew Aksyonoff
manticore    | Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
manticore    | Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)
manticore    |
manticore    | precaching table 'text_idx'
manticore    | prereading 1 tables
manticore    | preread 1 tables in 0.000 sec
manticore    | accepting connections
manticore    | [BUDDY] started v2.3.10 '/usr/share/manticore/modules/manticore-buddy/bin/manticore-buddy --listen=http://127.0.0.1:9312 --bind=127.0.0.1  --threads=32 --skip=manticoresoftware/buddy-plugin-sharding --skip=manticoresoftware/buddy-plugin-queue' at http://127.0.0.1:36731
manticore    | [BUDDY] Loaded plugins:
manticore    | [BUDDY]   core: empty-string, backup, emulate-elastic, create, insert, alias, select, show, cli-table, plugin, test, alter-distributed-table, alter-rename-table, modify-table, knn, replace
manticore    | [BUDDY]   local:
manticore    | [BUDDY]   extra:

Testing:

snikolaev@dev2:~/1801$ docker-compose exec manticore mysql -P9306 -h0
mysql> select * from text_idx;
+------+-----------+
| id   | name      |
+------+-----------+
|    1 | Element 1 |
|    2 | Element 2 |
+------+-----------+