semtools

Perform semantic (meaning-based) search across code and documents using embedding-based similarity matching.

Purpose

The semtools skill provides access to Semtools, a high-performance Rust-based CLI for semantic search and document processing. Unlike traditional text search (ripgrep) which matches exact strings, or structural search (ast-grep) which matches syntax patterns, semtools understands semantic meaning through embeddings.

Key capabilities:

1. Semantic Search: Find code/text by meaning, not just keywords
2. Workspace Management: Index large codebases for fast repeated searches
3. Document Parsing: Convert PDFs, DOCX, PPTX to searchable text (requires API key)

Semtools excels at discovery - finding relevant code when you don't know the exact keywords, function names, or syntax patterns.

When to Use This Skill

Use the semtools skill when you need meaning-based search:

Semantic Code Discovery:

• Finding code that implements a concept ("error handling", "data validation")
• Discovering similar functionality across different modules
• Locating examples of a pattern when you don't know exact names
• Understanding what code does without reading everything

Documentation & Knowledge:

• Searching documentation by concept, not keywords
• Finding related discussions in comments or docs
• Discovering similar issues or solutions
• Analyzing technical documents (PDFs, reports)

Use Cases:

• "Find all authentication-related code" (without knowing function names)
• "Show me error handling patterns" (regardless of specific error types)
• "Find code similar to this implementation" (semantic similarity)
• "Search research papers for 'distributed consensus'" (document search)

Choose semtools over file-search (ripgrep/ast-grep) when:

• You know the concept but not the keywords
• Exact string matching misses relevant results
• You want semantically similar code, not exact matches
• Searching across languages or mixed content

Still use file-search when:

• You know exact keywords, function names, or patterns
• You need structural code matching (ast-grep)
• Speed is critical (ripgrep is faster for exact matches)
• You're searching for specific symbols or references

Available Commands

Semtools provides three CLI commands you can use via execute_command:

• search - Semantic search across code and text files
• workspace - Manage workspaces for caching embeddings
• parse - Convert documents (PDF, DOCX, PPTX) to searchable text

All commands work out-of-the-box in your execution environment. Document parsing requires the LLAMA_CLOUD_API_KEY environment variable to be set.

Core Operations

1. Semantic Search (search)

Find files and code sections by semantic meaning:

# Basic semantic search
search "authentication logic" src/

# Search with more context (5 lines before/after)
search "error handling" --n-lines 5 src/

# Get more results (default: 3)
search "database queries" --top-k 10 src/

# Control similarity threshold (0.0-1.0, lower = more lenient)
search "API endpoints" --max-distance 0.4 src/

Parameters:

• --n-lines N: Show N lines of context around matches (default: 3)
• --top-k K: Return top K most similar matches (default: 3)
• --max-distance D: Maximum embedding distance (0.0-1.0, default: 0.3)
• -i: Case-insensitive matching

Output format:

Match 1 (similarity: 0.12)
File: src/auth/handlers.py
Lines: 42-47
----
def authenticate_user(username: str, password: str) -> Optional[User]:
    """Authenticate user credentials against database."""
    user = get_user_by_username(username)
    if user and verify_password(password, user.password_hash):
        return user
    return None
----

Match 2 (similarity: 0.18)
File: src/middleware/auth.py
...

2. Workspace Management (workspace)

For large codebases, create workspaces to cache embeddings and enable fast repeated searches:

# Create/activate workspace
workspace use my-project

# Set workspace via environment variable
export SEMTOOLS_WORKSPACE=my-project

# Index files in workspace (workspace auto-detected from env var)
search "query" src/

# Check workspace status
workspace status

# Clean up old workspaces
workspace prune

Benefits:

• Fast repeated searches: Embeddings cached, no re-computation
• Large codebases: IVF_PQ indexing for scalability
• Session persistence: Maintain context across multiple searches

When to use workspaces:

• Searching the same codebase multiple times
• Very large projects (1000+ files)
• Interactive exploration sessions
• CI/CD pipelines with repeated searches

3. Document Parsing (parse) ⚠️ Requires API Key

Convert documents to searchable markdown (requires LlamaParse API key):

# Parse PDFs to markdown
parse research_papers/*.pdf

# Parse Word documents
parse reports/*.docx

# Parse presentations
parse slides/*.pptx

# Parse and pipe to search
parse docs/*.pdf | xargs search "neural networks"

Supported formats:

• PDF (.pdf)
• Word (.docx)
• PowerPoint (.pptx)

Configuration:

# Via environment variable
export LLAMA_CLOUD_API_KEY="llx-..."

# Via config file
cat > ~/.parse_config.json << EOF
{
  "api_key": "llx-...",
  "max_concurrent_requests": 10,
  "timeout_seconds": 3600
}
EOF

Important: Document parsing is optional. Semantic search works without it.

Workflow Patterns

Pattern 1: Concept Discovery

When you know what you're looking for conceptually but not by name:

# Step 1: Broad semantic search
search "rate limiting implementation" src/

# Step 2: Review results, refine query
search "throttle requests per user" src/ --top-k 10

# Step 3: Use ripgrep for exact follow-up
rg "RateLimiter" --type py src/

Pattern 2: Similar Code Finder

When you want to find code similar to a reference implementation:

# Step 1: Extract key concepts from reference code
# [Read example_auth.py and identify key concepts]

# Step 2: Search for similar implementations
search "user authentication with JWT tokens" src/

# Step 3: Compare implementations
# [Review semantic matches to find similar approaches]

Pattern 3: Documentation Search

When researching concepts in documentation or comments:

# Search code comments semantically
search "thread safety guarantees" src/ --n-lines 10

# Search markdown documentation
search "deployment best practices" docs/

# Combined search
search "performance optimization" --top-k 20

Pattern 4: Cross-Language Search

When searching for concepts across different languages:

# Semantic search works across languages
search "connection pooling" src/

# May find:
# - Java: "ConnectionPool manager"
# - Python: "database connection reuse"
# - Go: "pool of persistent connections"
# All semantically related despite different terminology

Pattern 5: Document Analysis (with API key)

When analyzing PDFs or documents:

# Step 1: Parse documents to markdown
parse research/*.pdf > papers.md

# Step 2: Search converted content
search "transformer architecture" papers.md

# Step 3: Combine with code search
search "attention mechanism implementation" src/

Integration with file-search

Semtools and file-search (ripgrep/ast-grep) are complementary tools. Use them together for comprehensive search:

Search Strategy Matrix

| You Know | Use First | Then Use | Why | |----------|-----------|----------|-----| | Exact keywords | ripgrep | search | Fast exact match, then find similar | | Concept only | search | ripgrep | Find relevant code, then search specifics | | Function name | ripgrep | search | Find definition, then find similar usage | | Code pattern | ast-grep | search | Find structure, then find similar logic | | Approximate idea | search | ripgrep + ast-grep | Discover, then drill down |

Layered Search Approach

# Layer 1: Semantic discovery (what's related?)
search "user session management" --top-k 10

# Layer 2: Exact text search (what's the implementation?)
rg "SessionManager|session_store" --type py

# Layer 3: Structural search (how is it used?)
sg --pattern 'session.$METHOD($$$)' --lang python

# Layer 4: Reference tracking (where is it called?)
# [Use serena skill for symbol-level tracking]

Best Practices

1. Start Broad, Then Narrow

Use semantic search for discovery, then narrow with exact search:

# GOOD: Broad semantic discovery first
search "authentication" src/ --top-k 10
# [Review results to learn terminology]
rg "authenticate|verify_credentials" --type py src/

# AVOID: Starting too narrow and missing variations
rg "authenticate" --type py  # Misses "verify_credentials", "check_auth", etc.

2. Adjust Similarity Threshold

Tune --max-distance based on results:

# Too many irrelevant results? Decrease distance (more strict)
search "query" --max-distance 0.2

# Missing relevant results? Increase distance (more lenient)
search "query" --max-distance 0.5

# Default (0.3) works well for most cases
search "query"

3. Use Workspaces for Repeated Searches

For interactive exploration, always use workspaces:

# GOOD: Create workspace once, search many times
export SEMTOOLS_WORKSPACE=my-analysis
search "concept1" src/
search "concept2" src/
search "concept3" src/

# INEFFICIENT: Re-compute embeddings every time
search "concept1" src/
search "concept2" src/

4. Combine with Context Tools

Get more context around semantic matches:

# Find semantically similar code
search "retry logic" src/ --n-lines 2

# Get more context with ripgrep
rg -C 10 "retry" src/specific_file.py

# Or read the full file
cat src/specific_file.py

5. Phrase Queries Conceptually

Write queries as concepts, not exact keywords:

# GOOD: Conceptual queries
search "handling network timeouts"
search "user input validation"
search "concurrent data access"

# LESS EFFECTIVE: Exact keyword queries (use ripgrep instead)
search "timeout"  # Use: rg "timeout"
search "validate"  # Use: rg "validate"

Understanding Semantic Distance

Semtools uses embedding vectors to measure semantic similarity:

• Distance 0.0: Identical meaning
• Distance 0.1-0.2: Very similar (synonyms, paraphrases)
• Distance 0.2-0.3: Related concepts (default threshold)
• Distance 0.3-0.4: Loosely related
• Distance 0.5+: Weakly related or unrelated

Practical guidelines:

# Strict matching (only close matches)
--max-distance 0.2

# Balanced matching (default, recommended)
--max-distance 0.3

# Lenient matching (exploratory search)
--max-distance 0.4

# Very lenient (may include false positives)
--max-distance 0.5

Local vs. Cloud Embeddings

Semantic Search (Local):

• Uses local embeddings (model2vec, potion-multilingual-128M)
• No API calls or cloud dependencies
• Fast, private, no cost
• Works offline

Document Parsing (Cloud):

• Uses LlamaParse API (cloud-based)
• Requires API key and internet connection
• Processes PDFs, DOCX, PPTX
• Usage-based pricing (check LlamaIndex pricing)

Privacy consideration: Semantic search is 100% local. Only document parsing sends data to LlamaParse API.

Performance Considerations

Speed Characteristics

Without workspace:

• First search: ~2-5 seconds (embedding computation)
• Subsequent searches: ~2-5 seconds each (re-compute embeddings)

With workspace (cached embeddings):

• First search: ~2-5 seconds (builds index)
• Subsequent searches: ~0.1-0.5 seconds (cached)
• Large codebases: IVF_PQ indexing for scalability

Comparison:

• ripgrep: 0.01-0.1 seconds (fastest, exact match)
• ast-grep: 0.1-0.5 seconds (fast, structural)
• semtools (cached): 0.1-0.5 seconds (fast, semantic)
• semtools (uncached): 2-5 seconds (slower, semantic)

Optimization Tips

# 1. Use workspaces for repeated searches
export SEMTOOLS_WORKSPACE=my-project

# 2. Limit search scope to relevant directories
search "query" src/ --not tests/

# 3. Use --top-k to control result count
search "query" --top-k 5

# 4. Pipe to head for quick preview
search "query" | head -50

Unix Pipeline Integration

Semtools is designed for Unix-style composition:

# Find and parse PDFs, then search
find docs/ -name "*.pdf" | xargs parse | xargs search "topic"

# Search and filter with grep
search "authentication" src/ | grep -i "jwt"

# Count matches
search "error handling" src/ | grep "Match" | wc -l

# Combine with other tools
search "API" src/ | xargs -I {} rg -l "REST" {}

Limitations

When NOT to Use Semtools

1. Exact keyword search: Use ripgrep for known keywords

   # WRONG TOOL: Semantic search for exact function name
   search "authenticate_user"
   
   # RIGHT TOOL: Use ripgrep for exact matches
   rg "authenticate_user" --type py
   

2. Structural code patterns: Use ast-grep for syntax matching

   # WRONG TOOL: Semantic search for code structure
   search "class with constructor"
   
   # RIGHT TOOL: Use ast-grep for structure
   sg --pattern 'class $NAME { constructor($$$) { $$$ } }'
   

3. Symbol references: Use serena for LSP-based tracking

   # WRONG TOOL: Semantic search for all usages
   search "MyClass usage"
   
   # RIGHT TOOL: Use serena for precise references
   serena find_referencing_symbols --name 'MyClass'
   

4. Small codebases: Overhead not worth it for <100 files

   • ripgrep is faster and simpler for small projects

Known Edge Cases

• Ambiguous queries: Vague concepts return broad results
• Technical jargon: Domain-specific terms may have lower accuracy
• Short code snippets: Limited context reduces embedding quality
• Mixed languages: Embeddings tuned for English (multilingual model used)
• Generated code: Repetitive patterns may cluster together

Troubleshooting

No Semantic Matches Found

If semantic search returns zero results:

1. Verify files exist: Use ripgrep to confirm content

   rg "concept" src/
   

2. Increase similarity threshold: Be more lenient

   search "query" --max-distance 0.5
   

3. Rephrase query: Try different terminology

   search "user authentication"
   search "verify user credentials"
   search "login validation"
   

4. Check file types: Ensure searching correct extensions

   search "query" src/*.py  # Target specific types
   

Too Many Irrelevant Results

If semantic search returns too much noise:

1. Decrease similarity threshold: Be more strict

   search "query" --max-distance 0.2
   

2. Limit result count: Review top matches only

   search "query" --top-k 3
   

3. Narrow directory scope: Search specific paths

   search "query" src/specific_module/
   

4. Refine query: Add more specific concepts

   # Vague
   search "data"
   
   # Specific
   search "data validation with regex patterns"
   

Document Parsing Fails

If parse fails:

1. Verify API key is set:

   echo $LLAMA_CLOUD_API_KEY
   

2. Check file format: Ensure supported format (PDF, DOCX, PPTX)

   file document.pdf  # Verify file type
   

3. Check file size: Large files may timeout

   du -h document.pdf  # Check size
   

4. Review parse config: Adjust timeouts if needed

   cat ~/.parse_config.json
   

Workspace Issues

If workspace commands fail:

# Check workspace status
workspace status

# Prune corrupted workspaces
workspace prune

# Recreate workspace
rm -rf ~/.semtools/workspaces/my-workspace
export SEMTOOLS_WORKSPACE=my-workspace

Resources

• Semtools GitHub
• LlamaParse Documentation
• model2vec Embeddings
• Semantic Search Concepts