Get started with Vector Search in Azure Cosmos DB

This tutorial demonstrates how to quickly implement vector search in Azure Cosmos DB for NoSQL using a simple movie dataset. The application is available in Python, TypeScript, .NET, and Java, providing step-by-step instructions for setup, data loading, and similarity search queries.

Vector databases excel at storing and managing vector embeddings—high-dimensional mathematical representations of data. Each dimension reflects a data feature, potentially numbering in the tens of thousands. A vector's location in this space signifies its characteristics. This technique vectorizes various data types, including words, phrases, documents, images, and audio, enabling applications like similarity search, multi-modal search, recommendation engines, and large language models (LLMs).

Prerequisites:

• An Azure subscription (or a free Azure account, or the free tier of Azure Cosmos DB for NoSQL).
• An Azure Cosmos DB for NoSQL account.
• An Azure OpenAI Service resource with the text-embedding-ada-002 embedding model deployed (accessible via the Azure AI Foundry portal). This model provides text embeddings.
• The necessary programming language environment (Maven for Java).

Configuring the Vector Database in Azure Cosmos DB for NoSQL:

1. Enable the feature: This is a one-time step. Explicitly enable vector indexing and search within Azure Cosmos DB.

   

2. Create database and container: Create a database (e.g., movies_db) and a container (e.g., movies) with a partition key of /id.

3. Create policies: Configure a vector embedding policy and an indexing policy for the container. For this example, use the settings shown below (manual configuration via the Azure portal is used here, though programmatic methods are also available).

   

   Index Type Note: The example uses the diskANN index type with a dimension of 1536, matching the text-embedding-ada-002 model. While adaptable, changing the index type necessitates adjusting the embedding model to match the new dimension.

Loading Data into Azure Cosmos DB:

A sample movies.json file provides movie data. The process involves:

1. Reading movie information from the JSON file.
2. Generating vector embeddings for movie descriptions using the Azure OpenAI Service.
3. Inserting the complete data (title, description, and embeddings) into the Azure Cosmos DB container.

Set the following environment variables before proceeding:

export COSMOS_DB_CONNECTION_STRING=""
export DATABASE_NAME=""
export CONTAINER_NAME=""
export AZURE_OPENAI_ENDPOINT=""
export AZURE_OPENAI_KEY=""
export AZURE_OPENAI_VERSION="2024-10-21"
export EMBEDDINGS_MODEL="text-embedding-ada-002"

Clone the repository:

git clone https://github.com/abhirockzz/cosmosdb-vector-search-python-typescript-java-dotnet
cd cosmosdb-vector-search-python-typescript-java-dotnet

Language-specific instructions for data loading are provided below. Each method uses the environment variables defined above. Successful execution will output messages indicating data insertion into Cosmos DB.

Data Loading Instructions (Abbreviated):

• Python: cd python; python3 -m venv .venv; source .venv/bin/activate; pip install -r requirements.txt; python load.py
• TypeScript: cd typescript; npm install; npm run build; npm run load
• Java: cd java; mvn clean install; java -jar target/cosmosdb-java-vector-search-1.0-SNAPSHOT.jar load
• .NET: cd dotnet; dotnet restore; dotnet run load

Verifying Data in Azure Cosmos DB:

Confirm data insertion using the Azure portal or a Visual Studio Code extension.

Vector/Similarity Search:

The search component uses the VectorDistance function to find similar movies based on a search criterion (e.g., "comedy"). The process is:

1. Generate a vector embedding for the search criterion.
2. Use VectorDistance to compare it with existing embeddings.

The query:

SELECT TOP @num_results c.id, c.description, VectorDistance(c.embeddings, @embedding) AS similarityScore FROM c ORDER BY VectorDistance(c.embeddings, @embedding)

Language-specific instructions (assuming environment variables are set and data is loaded):

Search Instructions (Abbreviated):

• Python: python search.py "inspiring" 3
• TypeScript: npm run search "inspiring" 3
• Java: java -jar target/cosmosdb-java-vector-search-1.0-SNAPSHOT.jar search "inspiring" 3
• .NET: dotnet run search "inspiring" 3

Closing Notes:

Experiment with different vector index types (flat, quantizedFlat), distance metrics (cosine, Euclidean, dot product), and embedding models (text-embedding-3-large, text-embedding-3-small). Azure Cosmos DB for MongoDB vCore also supports vector search.

The above is the detailed content of Get started with Vector Search in Azure Cosmos DB. For more information, please follow other related articles on the PHP Chinese website!