Archivist

Archivist helps manage data quality across large, heterogeneous datasets. It provides validation rules, automatic correction suggestions, and a REST API for integration into existing workflows.

• Perceptual hashing for duplicate detection in large image/text datasets.
• Trigram search for fuzzy text matching and suggestions.
• KD Tree implementation for spatial data queries and validation.

Data Validation Pipeline

Archivist processes data through multiple validation stages:

graph TD
    Input[Data Source] --> Check1[Schema Check]
    Check1 --> Check2[Type Check]
    Check2 --> Check3[Fuzzy Match]
    Check3 --> Check4[Spatial Check]
    Check4 --> Output[Normalized Data]

REST API

Comprehensive endpoints for programmatic access:

POST /validate
Content-Type: application/json

{
  "dataset": "users",
  "records": [...]
}

Features

Perceptual Hashing

• Detects near-duplicates in image datasets
• Configurable similarity thresholds
• Batch processing for large collections

Fuzzy Text Matching

• Trigram search for suggestions
• Levenshtein distance calculations
• Auto-correction recommendations

Spatial Validation

• KD Tree for efficient queries
• Geospatial constraint checking
• Proximity-based validation

Scalability

• Incremental processing: Handle datasets larger than memory
• Distributed execution: Parallel validation across machines
• Streaming support: Process data as streams