Python英文文本相似度匹配

本文将从多个方面探讨Python中英文文本相似度匹配的方法和应用。

一、基于词向量的文本相似度匹配

1、词向量表示

import numpy as np
from gensim.models import Word2Vec

# 构建词向量模型
sentences = [['I', 'love', 'Python'], ['Python', 'is', 'great']]
model = Word2Vec(sentences, min_count=1)

# 获取某个单词的词向量
vector = model['Python']

2、文本相似度计算

from scipy.spatial.distance import cosine

def calculate_similarity(s1, s2, model):
    # 分词
    tokens1 = s1.split()
    tokens2 = s2.split()
    
    # 计算句子向量
    vector1 = np.mean([model[token] for token in tokens1], axis=0)
    vector2 = np.mean([model[token] for token in tokens2], axis=0)
    
    # 计算余弦相似度
    similarity = 1 - cosine(vector1, vector2)
    
    return similarity

s1 = "I love Python"
s2 = "Python is great"
similarity_score = calculate_similarity(s1, s2, model)

二、基于TF-IDF的文本相似度匹配

1、文本向量化

from sklearn.feature_extraction.text import TfidfVectorizer

corpus = ["I love dogs", "I hate cats", "Dogs are cute"]
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(corpus)
feature_names = vectorizer.get_feature_names()

2、文本相似度计算

from sklearn.metrics.pairwise import cosine_similarity

def calculate_similarity(s1, s2, vectorizer):
    # 向量化句子
    vector1 = vectorizer.transform([s1])
    vector2 = vectorizer.transform([s2])
    
    # 计算余弦相似度
    similarity = cosine_similarity(vector1, vector2)[0][0]
    
    return similarity

s1 = "I love dogs"
s2 = "Dogs are cute"
similarity_score = calculate_similarity(s1, s2, vectorizer)

三、基于BERT的文本相似度匹配

1、使用预训练的BERT模型

!pip install transformers

from transformers import BertTokenizer, BertModel

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')

2、文本相似度计算

from transformers import BertTokenizer, BertModel
import torch

def calculate_similarity(s1, s2, tokenizer, model):
    # 分词
    tokens1 = tokenizer.tokenize(s1)
    tokens2 = tokenizer.tokenize(s2)
    
    # 添加特殊标记
    tokens = ['[CLS]'] + tokens1 + ['[SEP]'] + tokens2 + ['[SEP]']
    
    # 将标记转换为对应的id
    input_ids = tokenizer.convert_tokens_to_ids(tokens)
    input_ids = torch.tensor([input_ids])
    
    # 获取句子的BERT表示
    outputs = model(input_ids)
    sentence_embeddings = outputs[0][:, 0, :]
    
    # 计算余弦相似度
    similarity = torch.nn.functional.cosine_similarity(sentence_embeddings[0], sentence_embeddings[1]).item()
    
    return similarity

s1 = "I love dogs"
s2 = "Dogs are cute"
similarity_score = calculate_similarity(s1, s2, tokenizer, model)

四、小结

本文介绍了Python中基于词向量、TF-IDF和BERT的文本相似度匹配方法。这些方法可以应用于文本分类、相似文档查找等任务中，具有很高的实用价值。