How Can You Calculate Cosine Similarity Between Two Sentences Without External Libraries?

Calculating Cosine Similarity Between Two Sentence Strings Without External Libraries

In the context of natural language processing, calculating cosine similarity is essential for measuring the textual similarity between documents. While external libraries like tf-idf-cosine can facilitate this task, it's also possible to compute cosine similarity manually without relying on such dependencies.

Cosine Similarity Overview

Cosine similarity quantifies the angle between two vectors representing the texts. A higher cosine similarity indicates a smaller angle, implying a greater similarity between the texts. It is calculated using the dot product of the normalized vectors divided by their magnitudes.

Implementing Cosine Similarity Manually

To calculate cosine similarity manually, we define the following steps:

1. Tokenization: Split the sentences into individual words.
2. Vectorization: Create a counter for each word in each sentence, which represents its frequency (term frequency).
3. Normalization: Normalize the vectors by dividing each element by the square root of the sum of its elements' squares (L2 norm).
4. Cosine Calculation: Compute the dot product of the normalized vectors and divide it by their magnitudes.

Code Implementation

Below is a Python implementation of the manual cosine similarity calculation:

<code class="python">import math
import re
from collections import Counter

WORD = re.compile(r"\w+")

def get_cosine(vec1, vec2):
    intersection = set(vec1.keys()) & set(vec2.keys())
    numerator = sum([vec1[x] * vec2[x] for x in intersection])
    
    sum1 = sum([vec1[x] ** 2 for x in vec1])
    sum2 = sum([vec2[x] ** 2 for x in vec2])
    denominator = math.sqrt(sum1) * math.sqrt(sum2)
    
    if not denominator:
        return 0.0
    else:
        return numerator / denominator

def text_to_vector(text):
    words = WORD.findall(text)
    return Counter(words)

text1 = "This is a foo bar sentence ."
text2 = "This sentence is similar to a foo bar sentence ."

vector1 = text_to_vector(text1)
vector2 = text_to_vector(text2)

cosine = get_cosine(vector1, vector2)

print("Cosine:", cosine)</code>

Result

This code executes as follows:

Cosine: 0.861640436855

This value indicates a high cosine similarity between the two sentences, confirming that they are textually similar.

Further Considerations

While this manual approach provides a basic implementation, it can be enhanced by:

• Incorporating stemming or lemmatization for improved word normalization.
• Implementing a more sophisticated tokenization scheme.
• Adding weightings like TF-IDF for more accurate similarity calculations.

The above is the detailed content of How Can You Calculate Cosine Similarity Between Two Sentences Without External Libraries?. For more information, please follow other related articles on the PHP Chinese website!