
Connecting to an OpenSearch (ES) service running in AWS using Python is painful. Most examples I find online either don't work or are outdated, leaving me constantly fixing the same issues. To save time and frustration, here’s a collection of working code snippets, up-to-date as of December 2024.
- Connect using the opensearch-py library (OpenSearch ElasticSearch)
-
Connect using the elasticsearch library (ElasticSearch only)
- elasticsearch >= 8
- elasticsearch < 8
Connect using the opensearch-py library (OpenSearch ElasticSearch)
This is my preferred way of connecting to an ES instance managed by AWS. It works for both ElasticSearch and OpenSearch clusters, and the authentication can take advantage of AWS profiles.
Install opensearch-py and boto3 (for authentication):
1 | pip install opensearch-py boto3
|
Copy after login
Copy after login
At the time of writing, this installs opensearch-py==2.8.0 and boto3==1.35.81.
Now, you can create a client using the following:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | import boto3
from opensearchpy import (
AWSV4SignerAuth,
OpenSearch,
RequestsHttpConnection,
)
es_host = "search-my-aws-esdomain-5k2baneoyj4vywjseocultv2au.eu-central-1.es.amazonaws.com"
aws_access_key = "AKIAXCUEGTAF3CV7GYKA"
aws_secret_key = "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"
region = "eu-central-1"
# Note: you can also use boto3.Session(profile_name= "my-profile" ) or other ways
session = boto3.Session(
aws_access_key_id=aws_access_key,
aws_secret_access_key=aws_secret_key,
region_name=region,
)
client = OpenSearch(
hosts=[{ "host" : es_host, "port" : 443}],
http_auth=AWSV4SignerAuth(session.get_credentials(), region, "es" ),
connection_class=RequestsHttpConnection,
use_ssl=True,
)
|
Copy after login
Copy after login
Note that boto3.Session supports various ways of creating a session: using a profile, environment variables, and more. I will let you check it out!
Once you have it, check the connection using:
1 2 | client.ping() # should return True
client.info() # use this to get a proper error message if ping fails
|
Copy after login
To check indices:
1 2 3 4 5 6 | # List all indices
client.cat.indices()
client.indices.get( "*" )
# Check the existence of an indice
client.indices.exists( "my-index" )
|
Copy after login
Connect using the elasticsearch library (ElasticSearch only)
? This only works for ElasticSearch clusters! Connecting to an OpenSearch cluster raises
UnsupportedProductError: The client noticed that the server is not Elasticsearch and we do not support this unknown product
elasticsearch >= 8
Most snippets are still referencing RequestsHttpConnection, a class that was removed in elasticsearch 8.X. If you were googling for the error cannot import name 'RequestsHttpConnection' from 'elasticsearch’, you are at the right place!
Install elasticsearch (this should install elastic-transport as well), and requests_aws4auth . The latter, based on requests, is required to handle authentication to AWS:
1 | pip install elasticsearch requests-aws4auth
|
Copy after login
At the time of writing, this installs elastic-transport==8.15.1, elasticsearch==8.17.0 and requests-aws4auth==1.3.1.
Now, you can create a client using the following:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | from elastic_transport import RequestsHttpNode
from elasticsearch import Elasticsearch
from requests_aws4auth import AWS4Auth
es_endpoint = "search-my-aws-esdomain-5k2baneoyj4vywjseocultv2au.eu-central-1.es.amazonaws.com"
aws_access_key = "AKIAXCUEGTAF3CV7GYKA"
aws_secret_key = "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"
region = "eu-central-1"
es = Elasticsearch(
f "https://{es_host}" ,
http_auth=AWS4Auth(
aws_access_key,
aws_secret_key,
region,
"es" ,
),
verify_certs=True,
node_class=RequestsHttpNode,
)
|
Copy after login
Once you have it, check the connection using:
1 2 | es.ping() # should return True
es.info() # use this to get a proper error message if ping fails
|
Copy after login
elasticsearch < 8
If you are still on an old version of elasticsearch:
1 | pip install "elasticsearch<8" requests-aws4auth
|
Copy after login
Currently elasticsearch==7.17.12, requests-aws4auth==1.3.1.
Now, you can create a client using the following:
1 | pip install opensearch-py boto3
|
Copy after login
Copy after login
Check the connection:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | import boto3
from opensearchpy import (
AWSV4SignerAuth,
OpenSearch,
RequestsHttpConnection,
)
es_host = "search-my-aws-esdomain-5k2baneoyj4vywjseocultv2au.eu-central-1.es.amazonaws.com"
aws_access_key = "AKIAXCUEGTAF3CV7GYKA"
aws_secret_key = "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"
region = "eu-central-1"
# Note: you can also use boto3.Session(profile_name= "my-profile" ) or other ways
session = boto3.Session(
aws_access_key_id=aws_access_key,
aws_secret_access_key=aws_secret_key,
region_name=region,
)
client = OpenSearch(
hosts=[{ "host" : es_host, "port" : 443}],
http_auth=AWSV4SignerAuth(session.get_credentials(), region, "es" ),
connection_class=RequestsHttpConnection,
use_ssl=True,
)
|
Copy after login
Copy after login
The above is the detailed content of How to connect to AWS OpenSearch or Elasticsearch clusters using python. For more information, please follow other related articles on the PHP Chinese website!