Detecting Similarities in Vectorized Documents

One particularly useful technique when working with vectorized documents is similarity detection—in simpler terms, identifying when an author may have copied or heavily borrowed from another source. Whether it's a literal copy/paste from another document or paraphrased content, semantic embeddings allow us to catch it.

How Similar is "Too Similar"?

The degree of similarity between two text chunks is expressed as a number—typically, the closer the number is to -1, the more similar the texts are. What's great is that to perform this kind of search, you don't need AI at all. A simple SQL SELECT will do the trick.

Here's an example:

SELECT a.cluster AS cluster_a,
    a.file AS file_a,
    a.page AS page_a,
    a.position AS position_a,
    b.cluster AS cluster_b,
    b.file AS file_b,
    b.page AS page_b,
    b.position AS position_b,
    a.embedding <#> b.embedding AS similarity,
    a.text_chunk AS text_chunk_a,
    b.text_chunk AS text_chunk_b
FROM embeddings a, embeddings b
WHERE a.id < b.id
ORDER BY similarity ASC
LIMIT 20;

This query surfaces the top 20 most similar text pairs in your dataset, based on vector distance.

Example of the this SELECT operation applied to the documents mentioned in the previous blog post. 

When Does Similarity Matter?

The importance of these findings depends on the nature of your documents. For example:

• In financial reports or structured tables, a high number of similar passages might be perfectly normal.

• In academic papers or theses, it could be a red flag for plagiarism.

Zooming In: Similarity Within Clusters

You can refine your approach further by restricting similarity checks to within the same cluster. Here's how to count how many high-similarity text pairs exist per cluster:

SELECT
    a.cluster,
    COUNT(*) AS count_above_threshold
FROM embeddings a
JOIN embeddings b ON a.id < b.id AND a.cluster = b.cluster
WHERE a.embedding <#> b.embedding < -0.98 -- set your threshold GROUP BY a.cluster
ORDER BY count_above_threshold DESC;

Example of the this SELECT operation applied to the documents mentioned in the previous blog post.

This helps you identify clusters with unusually high internal similarity—potentially indicating redundancy or heavy reuse of content.

No AI Needed

These examples highlight how you can work with vectorized text data without invoking AI models for every query. It's a fast, transparent, and surprisingly powerful approach.

If you have ideas, use cases, or questions, I'd love to hear from you.

Happy text mining! 🧠📊

Would you like me to add visuals or code snippets for sharing on LinkedIn too?