An alternative code up and running to use ELASTIC CLOUD on GCP from the really cool Haystack !

[RodolpheCalvet]

Hi everyone, wie geht's? Thank you for your amazing work the Haystack team!!

It takes a discussion for this because I couldn't PR, working from another project I couldn't figure it out sorry.
The idea is to share with you this recent setup, to ENABLE ELASTIC CONNEXION not only on aws but also on GCP, one of the three cloud offers supported by Elastic (looks like they don't have their own..).

It comes into two parts :

• the code to try, if ever of any interest for anyone, ie copy/paste in the elasticsearch.py file. After several tries, it looks like the cheapest modification to the repo's code, hopefully it's fully working and these changes don't extend beyond the file. The "aws4auth" parameter could be changed for something more "cloud generic", then the branching be done to build the right client (Azure next?),

• the full client-side code I used and tested (in Jupyter and PyCharm), that successfully wrote the remote elastic DB.

Looks like it also seemlessly overcomes the transitory procedure proposed by Elastic [https://www.elastic.co/guide/en/elasticsearch/client/python-api/master/migration.html#migration-compat-mode] to ease the move towards Elastic 8+, as will be the case for the team.

************** Suggested PR : From line 219 to 233 in ../haystack/document_stores/elasticsearch.py *********************

    # ---> Core suggestion of this PR : refactor "aws4auth" w. ->~~ "elasticCloud", then let the passed bundle
    # be treated/classified to build the suitable client..

    elif aws4auth:
        # aws elasticsearch with IAM
        # see https://elasticsearch-py.readthedocs.io/en/v7.12.0/index.html?highlight=http_auth#running-on-aws-with-iam
        if len(aws4auth) == 1:    # case for single element aws auth. solution (is it single?)
            logger.warning(
                "aws4auth passed to the ElasticsearchDocumentStore on AWS. The username will be ignored and aws4auth will be used for authentication."
            )
            client = Elasticsearch(
                hosts=hosts,
                http_auth=aws4auth,
                connection_class=RequestsHttpConnection,
                use_ssl=True,
                verify_certs=True,
                timeout=timeout,
            )
        elif len(aws4auth) == 3:    # case for three element tuple w. GCP credentials
            logger.warning(
                "aws4auth passed to the ElasticsearchDocumentStore on GCP."
            )
            client = Elasticsearch(
                cloud_id=aws4auth[2],
                http_auth=aws4auth[:2]
            )

    elif username: etc......................................ok

**************************** Code to run in nb, eg copy/paste in the middle of tutorial 13 ****************************

from haystack.document_stores import ElasticsearchDocumentStore
from elasticsearch import Elasticsearch
from python.config import ELASTIC_PASSWORD, ELASTIC_USER, ELASTIC_CLOUD_ID


# Create the client instance

# NOK : In haystack repo, the keywords are intercepted in a given order, that doesnt enable connexion on elastic cloud.
# In a commercial offer and as a whole if I'm not mistaken, we must choose bet. Goo, Aws and Azure. My first try was :
#client = Elasticsearch(host='https://013561904739491f90ae6857808c83b6.us-central1.gcp.cloud.es.io',
#                                            port=443, username=ELASTIC_USER, password=ELASTIC_PASSWORD,
#                                           api_key_id=ELASTIC_API_KEY_ID , api_key=ELASTIC_API_KEY)

# OK : check client call on elastic > 8 cloud (google gcp)
client = Elasticsearch(
    cloud_id=ELASTIC_CLOUD_ID,
    http_auth=("elastic", ELASTIC_PASSWORD)
)
# Successful response!
client.info()
# {'name': 'instance-0000000000', 'cluster_name': ...}

# As suggested in elasticsearch doc. Although finally writing in the cloud worked without it, after suggested changes...
import os
os.environ["ELASTIC_CLIENT_APIVERSIONING"] = "true"

# Sample text to write, from an awesome tutorial on genQA..
text1 = "Python is an interpreted, high-level, general-purpose programming language. Created by Guido van Rossum and first released in 1991, Python's design philosophy emphasizes code readability with its notable use of significant whitespace."
text2 = "Princess Arya Stark is the third child and second daughter of Lord Eddard Stark and his wife, Lady Catelyn Stark. She is the sister of the incumbent Westerosi monarchs, Sansa, Queen in the North, and Brandon, King of the Andals and the First Men. After narrowly escaping the persecution of House Stark by House Lannister, Arya is trained as a Faceless Man at the House of Black and White in Braavos, using her abilities to avenge her family. Upon her return to Westeros, she exacts retribution for the Red Wedding by exterminating the Frey male line."
text3 = "Dry Cleaning are an English post-punk band who formed in South London in 2018.[3] The band is composed of vocalist Florence Shaw, guitarist Tom Dowse, bassist Lewis Maynard and drummer Nick Buxton. They are noted for their use of spoken word primarily in lieu of sung vocals, as well as their unconventional lyrics. Their musical stylings have been compared to Wire, Magazine and Joy Division.[4] The band released their debut single, 'Magic of Meghan' in 2019. Shaw wrote the song after going through a break-up and moving out of her former partner's apartment the same day that Meghan Markle and Prince Harry announced they were engaged.[5] This was followed by the release of two EPs that year: Sweet Princess in August and Boundary Road Snacks and Drinks in October. The band were included as part of the NME 100 of 2020,[6] as well as DIY magazine's Class of 2020.[7] The band signed to 4AD in late 2020 and shared a new single, 'Scratchcard Lanyard'.[8] In February 2021, the band shared details of their debut studio album, New Long Leg. They also shared the single 'Strong Feelings'.[9] The album, which was produced by John Parish, was released on 2 April 2021.[10]"

docs = [{"content": text1}, {"content": text2}, {"content": text3}]

# Initialize document store and write in the documents

# NOK :
#document_store = ElasticsearchDocumentStore(host='https://013561904739491f90ae6857808c83b6.us-central1.gcp.cloud.es.io',
#                                            port=443, username=ELASTIC_USER, password=ELASTIC_PASSWORD,
#                                           api_key_id=ELASTIC_API_KEY_ID , api_key=ELASTIC_API_KEY)

# OK :
# The keyWord aws4auth is kept for the hack, although full elasticSearch here,
# working with given PR for ../haystack/document_stores/elasticsearch.py
document_store = ElasticsearchDocumentStore(host='https://013561904739491f90ae6857808c83b6.us-central1.gcp.cloud.es.io',
                                            port=443, aws4auth=(ELASTIC_USER, ELASTIC_PASSWORD, ELASTIC_CLOUD_ID ))

# A basic procedure for generating the standard http "Basic Authorization" (usualy done eg inside the python "requests" pkg)
# cf https://stackoverflow.com/questions/18139093/base64-authentication-python

import base64
userpass = ELASTIC_USER + ':' + ELASTIC_PASSWORD
encoded_u = base64.b64encode(userpass.encode()).decode()
headers = {"Authorization" : "Basic %s" % encoded_u}

document_store.write_documents(docs, headers=headers)

# The write_documents returns None, making it return the http response could be cool, w/o it we check in elastic console/ok

******************************************************************************************************************************

Happy coding guys :)

[julian-risch]

@RodolpheCalvet Thanks for sharing and great to hear that you needed to change only very few lines of Haystack code to get it working for you with Elastic Cloud on GCP!

[RodolpheCalvet]

Great team, great city! :)

[overmode]

This would be very useful, do you think that a PR will actually be created and merged ?

[scottroot]

Hi, thanks for sharing this. Does this change only work if running ES8 on ElasticCloud? I have ES8 running in AWS EC2 right now and haven't been able to get this working, still getting the same error as before.

[RodolpheCalvet]

Hi there! So the code in my imports from haystack etc.. (reqs seen w. conda list) is: - elasticsearch 7.17.9, - farm-haystack 1.15.0rc0, - protobuf 3.19.6 And my elastic cloud is separate from my Google app engine, not a package offer from Google, w. the Elastic cloud instance running having happened to be elastic8.6.2. Works fine! Cheers Le jeu. 22 juin 2023 à 8:25 AM, Scott ***@***.***> a écrit :

…

Hi, thanks for sharing this. Does this change only work if running ES8 on ElasticCloud? I have ES8 running in AWS EC2 right now and haven't been able to get this working, still getting the same error as before. — Reply to this email directly, view it on GitHub <#4547 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARDYHCTVOIJU5VTUKDWYESDXMPQN3ANCNFSM6AAAAAAWMCHQRM> . You are receiving this because you were mentioned.Message ID: ***@***.***>