testing-prpm-cli

Overview

Test PRPM CLI commands against a local registry by building the package, setting the registry URL, and invoking the CLI directly. Includes comprehensive cross-format conversion testing patterns and contract testing to verify documented behavior matches implementation.

When to Use

• Testing new CLI commands or features
• Debugging CLI behavior
• Verifying CLI changes before committing
• Testing against local registry data
• Validating cross-format conversions
• Testing new format converters
• Contract testing: verifying documented behavior matches implementation

Quick Reference

| Step | Command | |------|---------| | Build CLI | npm run build --workspace=packages/cli | | Start local registry | npm run dev --workspace=packages/registry | | Set local registry | export PRPM_REGISTRY_URL=http://127.0.0.1:3111 | | Run CLI directly | node /Users/khaliqgant/Projects/prpm/app/packages/cli/dist/index.js <command> | | Run via npm link | prpm <command> (after linking) |

Workflow

1. Build the CLI (Required First Step)

Always rebuild before testing to ensure you're testing current code:

npm run build --workspace=packages/cli

2. Start Local Registry

Start the registry server in background:

npm run dev --workspace=packages/registry &
sleep 3
lsof -i :3111  # Verify it's running

3. Configure Local Registry

Point CLI to local registry instead of production:

export PRPM_REGISTRY_URL=http://127.0.0.1:3111

Verify it's set:

echo $PRPM_REGISTRY_URL

4. Run CLI Commands

Option A: Direct invocation (recommended for testing)

node /Users/khaliqgant/Projects/prpm/app/packages/cli/dist/index.js search typescript
node /Users/khaliqgant/Projects/prpm/app/packages/cli/dist/index.js install some-package

Option B: npm link (for interactive testing)

cd packages/cli
npm link
prpm search typescript

Comprehensive Conversion Testing

Use Self-Improving Skill to Find Test Packages

Before testing conversions, use the self-improving skill to download a diverse set of packages:

# Search for packages of different types
node $CLI search "claude" --limit 10
node $CLI search "cursor" --limit 10
node $CLI search "agent" --limit 10
node $CLI search "skill" --limit 10

Create Test Directory and Install Diverse Packages

mkdir -p /tmp/prpm-conversion-tests
cd /tmp/prpm-conversion-tests
export PRPM_REGISTRY_URL=http://127.0.0.1:3111
CLI="/Users/khaliqgant/Projects/prpm/app/packages/cli/dist/index.js"

# Install packages of different subtypes
node $CLI install @prpm/agent-builder-skill --as claude      # Skill
node $CLI install @prpm/creating-cursor-rules --as cursor    # Cursor Rule
node $CLI install @camoneart/context-engineering-agent --as claude  # Agent

Supported Formats (CLI_SUPPORTED_FORMATS)

Test conversions across ALL supported formats:

| Format | Description | |--------|-------------| | cursor | Cursor IDE rules (.mdc) | | claude | Claude Code (skills, agents, commands) | | windsurf | Windsurf rules | | continue | Continue rules | | copilot | GitHub Copilot instructions | | kiro | Kiro steering files | | agents.md | Agents.md format | | gemini | Gemini CLI extensions | | ruler | Ruler format | | zed | Zed editor extensions | | opencode | OpenCode rules | | aider | Aider conventions | | trae | Trae rules | | replit | Replit agent rules | | zencoder | ZenCoder rules | | droid | Factory/Droid rules |

Conversion Test Matrix

Run comprehensive conversion tests:

cd /tmp/prpm-conversion-tests
mkdir -p /tmp/conversions
CLI="/Users/khaliqgant/Projects/prpm/app/packages/cli/dist/index.js"

# Claude Skill → All formats
for format in cursor windsurf kiro gemini zed continue copilot opencode aider trae replit zencoder droid; do
  node $CLI convert .claude/skills/*/SKILL.md --to $format -o /tmp/conversions/skill-to-$format.md 2>&1
done

# Cursor Rule → Multiple formats
for format in claude windsurf gemini zed; do
  node $CLI convert .cursor/rules/*.mdc --to $format -o /tmp/conversions/cursor-to-$format.md 2>&1
done

# Claude Agent → Multiple formats
for format in cursor gemini windsurf; do
  node $CLI convert .claude/agents/*.md --to $format -o /tmp/conversions/agent-to-$format.md 2>&1
done

Round-Trip Testing

Verify content preservation through round-trip conversions:

# Claude → Cursor → Claude
node $CLI convert .claude/skills/*/SKILL.md --to cursor -o /tmp/conversions/step1-cursor.mdc
node $CLI convert /tmp/conversions/step1-cursor.mdc --to claude -o /tmp/conversions/step2-claude.md

# Compare file sizes (expect some reduction but not dramatic)
wc -c .claude/skills/*/SKILL.md /tmp/conversions/step1-cursor.mdc /tmp/conversions/step2-claude.md

Validation Checklist

For each conversion, verify:

• [ ] Command succeeds - Exit code 0, no errors
• [ ] Output file created - File exists at specified path
• [ ] Content preserved - Core markdown structure intact
• [ ] Format-specific frontmatter - Correct fields for target format
• [ ] File size reasonable - Not truncated (compare with source)

Expected File Sizes

| Conversion Type | Expected Size Ratio | |----------------|---------------------| | Claude → Cursor | ~95-100% | | Claude → Windsurf | ~95-100% | | Claude → Gemini | ~95-100% | | Claude → OpenCode | May be smaller (format limits) | | Claude → Droid | May be smaller (format limits) | | Round-trip | ~50-70% (metadata loss expected) |

Test Report Template

Document results in this format:

## Conversion Test Report

### Test Setup
- Registry: Local (port 3111)
- Test packages: [list installed packages]

### Results

| Source | Target | Status | Output Size |
|--------|--------|--------|-------------|
| Claude Skill | Gemini | Pass/Fail | X bytes |
| ... | ... | ... | ... |

### Observations
- [Note any issues, truncations, or unexpected behavior]

Contract Testing (CRITICAL)

Contract testing ensures documented behavior matches implementation. This is the most important type of testing for features with configurable behavior.

Why Contract Testing Matters

The eager/lazy loading bug is a case study: documentation described a precedence chain (CLI > file > package > default), but implementation only handled CLI flags. Tests passed because they only tested the CLI flag path. Contract testing would have caught this.

Contract Testing Principles

1. Test EVERY documented behavior path, not just the happy path
2. Test precedence chains completely - if docs say "A > B > C > default", test all 4 cases
3. Test behavior WITHOUT flags - verify defaults and configuration-driven behavior
4. Test WITH and WITHOUT explicit settings - don't assume flag presence

Contract Testing Checklist

For ANY feature with documented behavior:

• [ ] Read the documentation first - what does it claim to do?
• [ ] List all behavior paths - every if/else/default mentioned
• [ ] Create test for each path - one test per documented behavior
• [ ] Test WITHOUT user flags - verify package/config defaults work
• [ ] Test precedence - verify higher-priority overrides lower
• [ ] Verify error messages match - documented errors should occur

Example: Eager/Lazy Loading Contract Tests

Documentation states: "Precedence: CLI flag > package-level > default (lazy)"

Required tests:

# Setup test directory
mkdir -p /tmp/prpm-contract-tests
cd /tmp/prpm-contract-tests
export PRPM_REGISTRY_URL=http://127.0.0.1:3111
CLI="/Users/khaliqgant/Projects/prpm/app/packages/cli/dist/index.js"

# Test 1: CLI --eager flag (highest priority)
rm -rf .openskills AGENTS.md
node $CLI install @prpm/some-skill --as agents.md --eager
# VERIFY: AGENTS.md contains activation="eager" or priority="0"
grep -q 'activation="eager"\|priority="0"' AGENTS.md && echo "PASS: CLI --eager works" || echo "FAIL: CLI --eager"

# Test 2: CLI --lazy flag overrides package eager
rm -rf .openskills AGENTS.md
# Install a package that has eager:true in prpm.json with --lazy flag
node $CLI install @prpm/eager-package --as agents.md --lazy
# VERIFY: Should be lazy despite package setting
grep -q 'activation="lazy"\|priority="1"' AGENTS.md && echo "PASS: CLI --lazy overrides" || echo "FAIL: CLI --lazy"

# Test 3: Package-level eager (NO CLI flag) - THIS IS THE BUG THAT WAS MISSED
rm -rf .openskills AGENTS.md
# Install a package that has eager:true in its prpm.json WITHOUT --eager flag
node $CLI install @prpm/eager-package --as agents.md
# VERIFY: Should be eager based on package setting
grep -q 'activation="eager"\|priority="0"' AGENTS.md && echo "PASS: Package eager works" || echo "FAIL: Package eager"

# Test 4: Default (lazy) when no flags and no package setting
rm -rf .openskills AGENTS.md
node $CLI install @prpm/normal-skill --as agents.md
# VERIFY: Should be lazy by default
grep -q 'activation="lazy"\|priority="1"' AGENTS.md && echo "PASS: Default lazy works" || echo "FAIL: Default"

Contract Test Template

For any new feature, create tests in this format:

## Contract Tests for [Feature Name]

### Documentation Claims:
1. [Claim 1 from docs]
2. [Claim 2 from docs]
3. [Precedence/default behavior from docs]

### Test Cases:

| # | Documented Behavior | Test Setup | Expected Result | Pass/Fail |
|---|--------------------|-----------| ----------------|-----------|
| 1 | [Claim 1] | [How to test] | [What to verify] | |
| 2 | [Claim 2] | [How to test] | [What to verify] | |
| 3 | Default behavior | [No flags/config] | [Default result] | |

### Results:
- All tests must pass before feature is considered complete
- Document any deviations between docs and implementation

Anti-Patterns to Avoid

| Anti-Pattern | Why It Fails | Correct Approach | |--------------|--------------|------------------| | Only testing with flags | Misses config/default paths | Test without flags first | | Assuming documentation is implementation | Docs may describe intent, not reality | Verify each claim with test | | Testing happy path only | Misses precedence bugs | Test all documented paths | | Skipping default behavior test | Defaults often broken | Always test "no config" case | | Not reading docs before testing | Miss documented behaviors | Read docs, list claims, test each |

Common Mistakes

| Mistake | Symptom | Fix | |---------|---------|-----| | Forgetting to build | Old behavior, changes not reflected | Run npm run build --workspace=packages/cli | | Missing registry env | Commands hit production registry | Set PRPM_REGISTRY_URL before running | | Stale npm link | Wrong version running | Re-run npm link after rebuilding | | Local registry not running | Connection refused errors | Start registry: npm run dev --workspace=packages/registry | | Testing single format only | Miss format-specific bugs | Test ALL formats in CLI_SUPPORTED_FORMATS | | No round-trip testing | Miss content loss bugs | Always verify round-trip preservation | | No contract testing | Docs ≠ implementation | Test EVERY documented behavior | | Only testing with CLI flags | Miss config/default bugs | Test without flags to verify defaults |

One-Liner Setup

npm run build --workspace=packages/cli && export PRPM_REGISTRY_URL=http://127.0.0.1:3111

Then test commands:

node packages/cli/dist/index.js --help

Unit Test Commands

Run converter unit tests:

# All converter tests
npm run test --workspace=packages/converters

# Specific test files
npm run test --workspace=packages/converters -- --testPathPattern="file-references"
npm run test --workspace=packages/converters -- --testPathPattern="security"
npm run test --workspace=packages/converters -- --testPathPattern="cross-format"
npm run test --workspace=packages/converters -- --testPathPattern="zed"