univo

Univo is a powerful tool for identifying duplicates in Python data, designed for efficiency and flexibility. It supports various forms of data—including strings, records, and nested objects—allowing users to implement fast, local duplicate detection without the need for a server. Univo primarily employs fuzzy matching alongside local embeddings and configurable field-level rules to maintain clean records in datasets.

Key Features:

• Wide Range of Data Types: Effectively compares not just strings, but also complex data structures like dictionaries and lists, ensuring field-by-field examination.
• Customizable Match Criteria: Users can define schemas that specify how and which fields to compare, enabling tailored rules to exclude certain matches based on field differences.
• Fast and Lightweight: Designed for speed, Univo enables the deduplication of large datasets—in tests, 100,000 names were processed in just 4.5 seconds. The entire package exists in a compact size (approximately 2 MiB), facilitating quick installation with no heavy dependencies, enhancing its practicality for data cleaning.

Local Processing:

Univo operates entirely offline; therefore, sensitive records never leave the local environment, and there is no need for API keys or external servers. This enhances data security and processing speed.

Example Usage:

Here is how to effectively use Univo to find duplicate names:

from univo import Finder

names = ["Northwind Traders", "Contoso", "Northwnd Traders", "NORTHWIND TRADERS", "Fabrikam"]
for match in Finder().find(names, min_score=0.8).matches:
    print(names[match.left_id], "~", names[match.right_id], round(match.score, 2))

This example demonstrates how Univo can return duplicates with their respective matching scores, highlighting its capability to assist in real-time duplicate checks.

Robust Schema Configuration:

Univo allows for powerful schema configurations, enabling users to dictate exact field comparisons and the reasoning behind each match:

from univo import Exact, Finder, Schema, Text

schema = Schema(
    id_field="id",
    fields={"name": Text(), "address.city": Text(), "vat": Exact(conflict="reject")},
)
finder = Finder(schema=schema)
finder.add([
    {"id": 1, "name": "Northwind Traders SAS", "address": {"city": "LYON"}, "vat": "FR401"},
    {"id": 2, "name": "Northwind Traders", "address": {"city": "Lyon"}, "vat": "FR999"},
])
query = {"id": 3, "name": "Northwind Traders", "address": {"city": "Lyon"}, "vat": "FR401"}
for match in finder.search(query).matches:
    print(match.right_id, round(match.score, 2), match.reasons)

Here, the matching process provides clarity on why certain records are identified as duplicates based on various criteria, ensuring thorough data validation.

Semantic Comparisons:

Univo also leverages semantic understanding for enhanced matching capabilities. Users can employ models like Model2Vec for meaningful comparisons among text entries:

from univo import Finder, Schema, Text

finder = Finder(
    schema=Schema(fields={"description": Text(mode="semantic")}),
    model="minishlab/potion-base-8M"  # Utilizing a pre-trained model
)
texts = ["online clothing store", "shop selling clothes on the internet", "steel factory"]
report = finder.find([{ "description": text} for text in texts], min_score=0.6)
for match in report.matches:
    print(texts[match.left_id], "~", texts[match.right_id], round(match.score, 2))

In this context, Univo demonstrates the ability to recognize semantically similar entries, providing users with crucial insights into their data.

Performance Comparisons:

Univo is purpose-built for its niche and consistently outperforms other available tools in terms of speed, accuracy, and simplicity. For instance, in benchmark tests against tools like SemHash, Univo showed:

• Faster deduplication times with optimized memory usage
• More controlled matching rules for structured records, reducing erroneous matches significantly

Documentation and Support:

For comprehensive guidance on using Univo effectively, the following resources are available:

• Quickstart Guide
• Reference Documentation
• Performance Benchmarks

Univo represents a significant advancement in data integrity and record management, making it an ideal choice for developers and data scientists looking for a robust solution to duplicate detection.