ConnectionError: Initial connection to Elasticsearch failed

[khang-nguyen2907]

Hi, because of some reasons, I cannot install Elasticsearch for my computer, and I cannot use localhost:9200 to connect to ElasticsearchDocumentStore, I try to use ElasticsearchDocumentStore on Colab with following code:

I also follow the tutorial with ElasticsearchDocumentStore for setting up the environment on Colab. When I create my deployment on Elasticsearch page, I see the version is 8.4.1, then I change the version downloaded on Colab is the same with the version showing in my deployment, I already tried lower version and it failed, too.

from haystack.utils import launch_es
launch_es()

# In Colab / No Docker environments: Start Elasticsearch from source
! wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.4.1-linux-x86_64.tar.gz -q
! tar -xzf elasticsearch-8.4.1-linux-x86_64.tar.gz
! chown -R daemon:daemon elasticsearch-8.4.1

import os
from subprocess import Popen, PIPE, STDOUT

es_server = Popen(
    ["elasticsearch-8.4.1/bin/elasticsearch"], stdout=PIPE, stderr=STDOUT, preexec_fn=lambda: os.setuid(1)  # as daemon
)
# wait until ES has started
! sleep 30

from haystack.document_stores import ElasticsearchDocumentStore
import certifi
cw = certifi.where()
document_store = ElasticsearchDocumentStore(host="https://wikipedia-document.es.us-central1.gcp.cloud.es.io", port=9243, scheme='https',
                                             ca_certs=cw , 
                                             username="elastic", password="<pwd got from creating deployment>", 
                                             index="document", embedding_field="embedding", embedding_dim=768)

The link passing to the host argument I get from copying Elasticsearch endpoint url on https://cloud.elastic.co/deployments/xxx

I also open the link I passed to the host argument and login with the username and password, it returns information in json format like Elasticsearch is running. However, when I run the code, the error appeared as:

ConnectionError: Initial connection to Elasticsearch failed. Make sure you run an Elasticsearch instance at `[{'host': 'https://wikipedia-document.es.us-central1.gcp.cloud.es.io', 'port': 9243}]` and that it has finished the initial ramp up (can take > 30s). site:stackoverflow.com

I do not know how to solve this problem, I am new to Haystack and Elasticsearch. Moreover, most of solutions are on localhost, which I do not use. I want to use ElasticsearchDocumentStore on Colab to store the data in the database for using later, so that is the reason why I want to connect Elasticsearch cloud when initializing DocumentStore. Could you help me to figure out how to solve this please? I would be appreciated it, thank you.

[vblagoje]

Hey @khang-nguyen2907 I am confused about your post - why are launching a local instance of the ES server and then trying to connect ElasticsearchDocumentStore to a remote es somewhere on the cloud?

[danielbichuetti]

@khang-nguyen2907 You should remove the https:// from the host parameter.

Also, you can get one year of free OpenSearch Services on AWS. It should be good for testing in your scenario. Running on Colab, you will lose any data when it resets.