Sentence transformers

"""
This is a simple application for sentence embeddings: semantic search

We have a corpus with various sentences. Then, for a given query sentence,
we want to find the most similar sentence in this corpus.

This script outputs for various queries the top 5 most similar sentences in the corpus.
"""
from sentence_transformers import SentenceTransformer, util
import torch

embedder = SentenceTransformer('all-MiniLM-L6-v2')

# Corpus with example sentences
corpus = ['A man is eating food.',
          'A man is eating a piece of bread.',
          'The girl is carrying a baby.',
          'A man is riding a horse.',
          'A woman is playing violin.',
          'Two men pushed carts through the woods.',
          'A man is riding a white horse on an enclosed ground.',
          'A monkey is playing drums.',
          'A cheetah is running behind its prey.'
          ]
corpus_embeddings = embedder.encode(corpus, convert_to_tensor=True)

# Query sentences:
queries = ['A man is eating pasta.', 'Someone in a gorilla costume is playing a set of drums.', 'A cheetah chases prey on across a field.']


# Find the closest 5 sentences of the corpus for each query sentence based on cosine similarity
top_k = min(5, len(corpus))
for query in queries:
    query_embedding = embedder.encode(query, convert_to_tensor=True)

    # We use cosine-similarity and torch.topk to find the highest 5 scores
    cos_scores = util.cos_sim(query_embedding, corpus_embeddings)[0]
    top_results = torch.topk(cos_scores, k=top_k)

    print("\n\n======================\n\n")
    print("Query:", query)
    print("\nTop 5 most similar sentences in corpus:")

    for score, idx in zip(top_results[0], top_results[1]):
        print(corpus[idx], "(Score: {:.4f})".format(score))

    """
    # Alternatively, we can also use util.semantic_search to perform cosine similarty + topk
    hits = util.semantic_search(query_embedding, corpus_embeddings, top_k=5)
    hits = hits[0]      #Get the hits for the first query
    for hit in hits:
        print(corpus[hit['corpus_id']], "(Score: {:.4f})".format(hit['score']))
    """

Source: sbert.net

Similar codes

0 vote159 views1 answer
Create matrix with complex python
Python
8 votes151 views1 answer
Remove strings from a list python if they're short
Python
31 votes171 views3 answers
How to adda vaslues to data frame
Python
0 vote159 views3 answers
Find rules of decision tree python
Python
0 vote153 views1 answer
ValueError: Graph disconnected: cannot obtain value for tensor Tensor("input_3_1:0", shape=(None, None, 71), dtype=float32) at layer "input_3". The following previous layers were accessed without issue: []
Python
1 vote179 views1 answer
How to make an object set once python
Python
1 vote165 views1 answer
Make column nullable django
Python
2 votes164 views3 answers
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Python
0 vote178 views1 answer
Get return value of contract call in brownie
Python
58 votes219 views6 answers
String list to int list python
Python