deduplication AI Agent Skills

nemo-curator

21.8k

davila7

GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.

memory-fabric

97

yonatangross

Knowledge graph memory orchestration - entity extraction, query parsing, deduplication, and cross-reference boosting. Use when designing memory orchestration.

Notification Spam Detector

71

PramodDutta

Test notification systems for spam behavior including duplicate alerts, missing throttling, incorrect delivery channels, and notification preference violations

photo-content-recognition-curation-expert

43

curiositech

Expert in photo content recognition, intelligent curation, and quality filtering. Specializes in face/animal/place recognition, perceptual hashing for de-duplication, screenshot/meme detection, burst photo selection, and quick indexing strategies. Activate on 'face recognition', 'face clustering', 'perceptual hash', 'near-duplicate', 'burst photo', 'screenshot detection', 'photo curation', 'photo indexing', 'NSFW detection', 'pet recognition', 'DINOHash', 'HDBSCAN faces'. NOT for GPS-based location clustering (use event-detection-temporal-intelligence-expert), color palette extraction (use color-theory-palette-harmony-expert), semantic image-text matching (use clip-aware-embeddings), or video analysis/frame extraction.

Skills Collection Manager

9

jackspace

A comprehensive toolkit for managing large Claude Code skill collections, including bulk downloading from GitHub, organizing into categories, detecting and removing duplicates, consolidating skills, and maintaining clean skill repositories with 100+ skills.

building-vulnerability-dashboard-with-defectdojo

2

mukul975

Deploy DefectDojo as a centralized vulnerability management dashboard with scanner integrations, deduplication, metrics tracking, and Jira ticketing workflows.