multi-model-validation

Version: 3.3.0 Purpose: Patterns for running multiple AI models in parallel via Claudish proxy with context-aware preferences, dynamic model discovery, session-based workspaces, and performance statistics Status: Production Ready

Overview

Multi-model validation is the practice of running multiple AI models (Grok, Gemini, GPT-5, DeepSeek, etc.) in parallel to validate code, designs, or implementations from different perspectives. This achieves:

• 3-5x speedup via parallel execution (15 minutes → 5 minutes)
• Consensus-based prioritization (issues flagged by all models are CRITICAL)
• Diverse perspectives (different models catch different issues)
• Cost transparency (know before you spend)
• Free model discovery (NEW v3.0) - find high-quality free models from trusted providers
• Performance tracking - identify slow/failing models for future exclusion
• Data-driven recommendations - optimize model shortlist based on historical performance

Key Innovations:

1. Context-Aware Preferences (NEW v3.3.0) - Automatically use saved model preferences per task type (debug/research/coding/review) from .claude/multimodel-team.json
2. Dynamic Model Discovery (v3.0) - Use claudish --top-models and claudish --free to get current available models with pricing
3. Session-Based Workspaces (v3.0) - Each validation session gets a unique directory to prevent conflicts
4. 4-Message Pattern - Ensures true parallel execution by using only Task tool calls in a single message
5. Pattern 7-8 - Statistics collection and data-driven model recommendations

This skill is extracted from the /review command and generalized for use in any multi-model workflow.

---

⚠️ MANDATORY: Learn and Reuse User Preferences

Model preferences are learned per context and reused automatically.

• First time a context is used → ASK user → SAVE to that context
• Next time same context → USE saved models automatically (no asking)
• User explicitly says "change models" or "different models" → ASK and UPDATE

# FIRST STEP - Read preferences file
cat .claude/multimodel-team.json 2>/dev/null

Flow:

1. Detect context from task keywords
   - "debug", "error", "bug", "fix" → debug
   - "research", "analyze", "investigate" → research
   - "implement", "build", "create", "code" → coding
   - "review", "audit", "check" → review

2. Check if contextPreferences[context] exists and is non-empty

   IF EXISTS (has models saved):
   → Use those models directly
   → DO NOT ask user
   → Proceed with validation

   IF EMPTY/MISSING (first time for this context):
   → Run: claudish --top-models
   → Ask user to select models (AskUserQuestion)
   → Save to contextPreferences[context]
   → Proceed with validation

3. User override triggers (explicit request to change):
   - "use different models"
   - "change models"
   - "update model preferences"
   → Ask user to select new models
   → Update contextPreferences[context]

Example - Learning Flow:

# First debug task ever:
Task: "Debug this authentication error"
→ Context: debug
→ contextPreferences.debug is empty
→ ASK: "Which models for debug tasks?"
→ User selects: grok, glm, minimax
→ SAVE to contextPreferences.debug
→ Run with those models

# Second debug task:
Task: "Debug the API timeout"
→ Context: debug
→ contextPreferences.debug = ["grok", "glm", "minimax"]
→ USE directly (no asking)
→ Run with saved models

# User wants to change:
Task: "Debug this error, use different models"
→ Detected: "different models" override trigger
→ ASK: "Which models for debug tasks?"
→ User selects: gemini, gpt-5-codex
→ UPDATE contextPreferences.debug
→ Run with new models

---

Related Skills

CRITICAL: Tracking Protocol Required

Before using any patterns in this skill, ensure you have completed the pre-launch setup from orchestration:model-tracking-protocol.

Launching models without tracking setup = INCOMPLETE validation.

Cross-References:

• orchestration:model-tracking-protocol - MANDATORY tracking templates and protocols (NEW in v0.6.0)
  • Pre-launch checklist (8 required items)
  • Tracking table templates
  • Failure documentation format
  • Results presentation template
• orchestration:quality-gates - Approval gates and severity classification
• orchestration:task-orchestration - Progress tracking during execution
• orchestration:error-recovery - Handling failures and retries

Skill Integration:

This skill (multi-model-validation) defines execution patterns (how to run models in parallel). The model-tracking-protocol skill defines tracking infrastructure (how to collect and present results).

Use both together:

skills: orchestration:multi-model-validation, orchestration:model-tracking-protocol

---

Core Patterns

Pattern 0: Session Setup and Model Discovery (NEW v3.0)

Purpose: Create isolated session workspace and discover available models dynamically.

Why Session-Based Workspaces:

Using a fixed directory like ai-docs/reviews/ causes problems:

• ❌ Multiple sessions overwrite each other's files
• ❌ Stale data from previous sessions pollutes results
• ❌ Hard to track which files belong to which session

Instead, create a unique session directory for each validation:

# Generate unique session ID
TARGET_SLUG=$(echo "${TASK_NAME:-review}" | tr '[:upper:] ' '[:lower:]-' | sed 's/[^a-z0-9-]//g' | head -c20)
SESSION_ID="review-${TARGET_SLUG}-$(date +%Y%m%d-%H%M%S)-$(head -c 4 /dev/urandom | xxd -p)"
SESSION_DIR="ai-docs/sessions/${SESSION_ID}"

# Create session workspace
mkdir -p "$SESSION_DIR"

echo "Session: $SESSION_ID"
echo "Directory: $SESSION_DIR"

# Example output:
# Session: review-auth-impl-20251212-143052-a3f2
# Directory: ai-docs/sessions/review-auth-impl-20251212-143052-a3f2

Benefits:

• ✅ Each session is isolated (no cross-contamination)
• ✅ Traceable - can associate files with a specific session
• ✅ Session ID can be used for tracking in statistics
• ✅ Parallel sessions don't conflict
• ✅ Aligned with dev:feature session pattern
• ✅ Committed to git for audit trail (unlike /tmp/)

⚠️ Do NOT use /tmp/ for session directories. Files in /tmp/ are not traceable, not committable, and parallel runs will overwrite each other.

---

Dynamic Model Discovery:

NEVER hardcode model lists. Models change frequently - new ones appear, old ones deprecate, pricing updates. Instead, use claudish to get current available models:

# Get top paid models (best value for money)
claudish --top-models

# Example output:
#   google/gemini-3-pro-preview    Google     $7.00/1M   1048K   🔧 🧠 👁️
#   openai/gpt-5.2-codex           Openai     $5.63/1M   400K    🔧 🧠 👁️
#   x-ai/grok-code-fast-1          X-ai       $0.85/1M   256K    🔧 🧠
#   minimax/minimax-m2.5           Minimax    $0.64/1M   262K    🔧 🧠
#   z-ai/glm-4.7                   Z-ai       $1.07/1M   202K    🔧 🧠
#   qwen/qwen3-vl-235b-a22b-ins... Qwen       $0.70/1M   262K    🔧    👁️

# Get free models from trusted providers
claudish --free

# Example output:
#   google/gemini-2.0-flash-exp:free  Google     FREE      1049K   ✓ · ✓
#   mistralai/devstral-2512:free      Mistralai  FREE      262K    ✓ · ·
#   qwen/qwen3-coder:free             Qwen       FREE      262K    ✓ · ·
#   qwen/qwen3-235b-a22b:free         Qwen       FREE      131K    ✓ ✓ ·
#   openai/gpt-oss-120b:free          Openai     FREE      131K    ✓ ✓ ·

Recommended Free Models for Code Review:

| Model | Provider | Context | Capabilities | Why Good | |-------|----------|---------|--------------|----------| | qwen/qwen3-coder:free | Qwen | 262K | Tools ✓ | Coding-specialized, large context | | mistralai/devstral-2512:free | Mistral | 262K | Tools ✓ | Dev-focused, excellent for code | | qwen/qwen3-235b-a22b:free | Qwen | 131K | Tools ✓ Reasoning ✓ | Massive 235B model, reasoning |

Model Selection Flow (Learn and Reuse):

1. Read Preferences File
   → cat .claude/multimodel-team.json
   → If file NOT exists → create empty one

2. Detect Task Context
   → Parse task for keywords (case-insensitive):
     - "debug", "error", "bug", "fix", "trace", "issue" → debug
     - "research", "investigate", "analyze", "explore", "find" → research
     - "implement", "build", "create", "code", "develop", "feature" → coding
     - "review", "audit", "check", "validate", "verify" → review
   → If no keywords match → context = "default"

3. Check for Override Triggers in User Message
   → "use different models", "change models", "update preferences"
   → If found → force_ask = true

4. Load or Learn Models
   → models = contextPreferences[context]

   IF models exist AND NOT force_ask:
     → USE models directly (no asking)
     → Go to step 6

   IF models empty OR force_ask:
     → Run: claudish --top-models
     → AskUserQuestion with multiSelect
     → Save user selection to contextPreferences[context]
     → Go to step 6

5. Save Updated Preferences
   → Write .claude/multimodel-team.json
   → Update lastUpdated timestamp

6. Execute with Models
   → Launch parallel validation
   → No further confirmation needed

Context Keywords:

| Context | Keywords | |---------|----------| | debug | debug, error, bug, fix, trace, issue | | research | research, investigate, analyze, explore, find | | coding | implement, build, create, code, develop, feature | | review | review, audit, check, validate, verify |

Override Triggers (force re-selection):

• "use different models"
• "change models"
• "update model preferences"
• "select new models"

⚠️ Prefix Collision Awareness

CRITICAL: When using claudish, be aware of model ID prefix routing.

Claudish routes to different backends based on model ID prefix:

| Prefix | Backend | Required Key | |--------|---------|--------------| | (none) | OpenRouter | OPENROUTER_API_KEY | | g/ gemini/ | Google Gemini API | GEMINI_API_KEY | | oai/ | OpenAI Direct API | OPENAI_API_KEY | | mmax/ mm/ | MiniMax Direct API | MINIMAX_API_KEY | | kimi/ moonshot/ | Kimi Direct API | KIMI_API_KEY | | glm/ zhipu/ | GLM Direct API | GLM_API_KEY | | ollama/ | Ollama (local) | None | | lmstudio/ | LM Studio (local) | None | | vllm/ | vLLM (local) | None | | mlx/ | MLX (local) | None |

Collision-Free Models (safe for OpenRouter):

• x-ai/grok-code-fast-1 ✅
• google/gemini-* ✅ (use g/ for Gemini Direct)
• deepseek/deepseek-chat ✅
• minimax/* ✅ (use mmax/ for MiniMax Direct)
• qwen/qwen3-coder:free ✅
• mistralai/devstral-2512:free ✅
• moonshotai/* ✅ (use kimi/ for Kimi Direct)
• z-ai/glm-* ✅ (use glm/ for GLM Direct)
• openai/* ✅ (use oai/ for OpenAI Direct)
• anthropic/claude-* ✅

Direct API prefixes for cost savings: | OpenRouter Model | Direct API Prefix | API Key Required | |------------------|-------------------|------------------| | openai/gpt-* | oai/gpt-* | OPENAI_API_KEY | | google/gemini-* | g/gemini-* | GEMINI_API_KEY | | minimax/* | mmax/* | MINIMAX_API_KEY | | moonshotai/* | kimi/* | KIMI_API_KEY | | z-ai/glm-* | glm/* | GLM_API_KEY |

Rule: OpenRouter models work without prefix. Use direct API prefixes for cost savings when you have the corresponding API key.

Interactive Model Selection (AskUserQuestion with multiSelect):

CRITICAL: Use AskUserQuestion tool with multiSelect: true to let users choose models interactively. This provides a better UX than just showing recommendations.

// Use AskUserQuestion to let user select models
AskUserQuestion({
  questions: [{
    question: "Which external models should validate your code? (Internal Claude reviewer always included)",
    header: "Models",
    multiSelect: true,
    options: [
      // Top paid (from claudish --top-models + historical data)
      {
        label: "x-ai/grok-code-fast-1 ⚡",
        description: "$0.85/1M | Quality: 87% | Avg: 42s | Fast + accurate"
      },
      {
        label: "google/gemini-3-pro-preview",
        description: "$7.00/1M | Quality: 91% | Avg: 55s | High accuracy"
      },
      // Free models (from claudish --free)
      {
        label: "qwen/qwen3-coder:free 🆓",
        description: "FREE | Quality: 82% | 262K context | Coding-specialized"
      },
      {
        label: "mistralai/devstral-2512:free 🆓",
        description: "FREE | 262K context | Dev-focused, new model"
      }
    ]
  }]
})

Remember Selection for Session:

Store the user's model selection in the session directory so it persists throughout the validation:

# After user selects models, save to session
save_session_models() {
  local session_dir="$1"
  shift
  local models=("$@")

  # Always include internal reviewer
  echo "claude-embedded" > "$session_dir/selected-models.txt"

  # Add user-selected models
  for model in "${models[@]}"; do
    echo "$model" >> "$session_dir/selected-models.txt"
  done

  echo "Session models saved to $session_dir/selected-models.txt"
}

# Load session models for subsequent operations
load_session_models() {
  local session_dir="$1"
  cat "$session_dir/selected-models.txt"
}

# Usage:
# After AskUserQuestion returns selected models
save_session_models "$SESSION_DIR" "x-ai/grok-code-fast-1" "qwen/qwen3-coder:free"

# Later in the session, retrieve the selection
MODELS=$(load_session_models "$SESSION_DIR")

Session Model Memory Structure:

$SESSION_DIR/
├── selected-models.txt    # User's model selection (persists for session)
├── code-context.md        # Code being reviewed
├── claude-review.md       # Internal review
├── grok-review.md         # External review (if selected)
├── qwen-coder-review.md   # External review (if selected)
└── consolidated-review.md # Final consolidated review

Why Remember the Selection:

1. Re-runs: If validation needs to be re-run, use same models
2. Consistency: All phases of validation use identical model set
3. Audit trail: Know which models produced which results
4. Cost tracking: Accurate cost attribution per session

Always Include Internal Reviewer:

BEST PRACTICE: Always run internal Claude reviewer alongside external models.

Why?
✓ FREE (embedded Claude, no API costs)
✓ Fast baseline (usually fastest)
✓ Provides comparison point
✓ Works even if ALL external models fail
✓ Consistent behavior (same model every time)

The internal reviewer should NEVER be optional - it's your safety net.

---

Pattern 1: The 4-Message Pattern (MANDATORY)

This pattern is CRITICAL for achieving true parallel execution with multiple AI models.

Why This Pattern Exists:

Claude Code executes tools sequentially by default when different tool types are mixed in the same message. To achieve true parallelism, you MUST:

1. Use ONLY one tool type per message
2. Ensure all Task calls are in a single message
3. Separate preparation (Bash) from execution (Task) from presentation

The Pattern:

Message 1: Preparation (Bash Only)
  - Create workspace directories
  - Validate inputs (check if claudish installed)
  - Write context files (code to review, design reference, etc.)
  - NO Task calls
  - NO Tasks calls

Message 2: Parallel Execution (Task Only)
  - Launch ALL AI models in SINGLE message
  - ONLY Task tool calls
  - Separate each Task with --- delimiter
  - Each Task is independent (no dependencies)
  - All execute simultaneously

Message 3: Auto-Consolidation (Task Only)
  - Automatically triggered when N ≥ 2 models complete
  - Launch consolidation agent
  - Pass all review file paths
  - Apply consensus analysis

Message 4: Present Results
  - Show user prioritized issues
  - Include consensus levels (unanimous, strong, majority)
  - Link to detailed reports
  - Cost summary (if applicable)

Example: 5-Model Parallel Code Review

Message 1: Preparation (Session Setup + Model Discovery)
  # Create unique session workspace
  Bash: SESSION_ID="review-$(date +%Y%m%d-%H%M%S)-$(head -c 4 /dev/urandom | xxd -p)"
  Bash: SESSION_DIR="ai-docs/sessions/${SESSION_ID}" && mkdir -p "$SESSION_DIR"
  Bash: git diff > "$SESSION_DIR/code-context.md"

  # Discover available models
  Bash: claudish --top-models  # See paid options
  Bash: claudish --free        # See free options

  # User selects models via AskUserQuestion (see Pattern 0)

Message 2: Parallel Execution (ONLY Task calls - single message)
  Task: senior-code-reviewer
    Prompt: "Review $SESSION_DIR/code-context.md for security issues.
             Write detailed review to $SESSION_DIR/claude-review.md
             Return only brief summary."
  ---
  Bash: claudish --model x-ai/grok-code-fast-1 --stdin --quiet
    < $SESSION_DIR/review-prompt.md > $SESSION_DIR/grok-review.md 2>$SESSION_DIR/grok-stderr.log
  ---
  Bash: claudish --model qwen/qwen3-coder:free --stdin --quiet
    < $SESSION_DIR/review-prompt.md > $SESSION_DIR/qwen-coder-review.md 2>$SESSION_DIR/qwen-stderr.log
  ---
  Bash: claudish --model openai/gpt-5.1-codex --stdin --quiet
    < $SESSION_DIR/review-prompt.md > $SESSION_DIR/gpt5-review.md 2>$SESSION_DIR/gpt5-stderr.log
  ---
  Bash: claudish --model mistralai/devstral-2512:free --stdin --quiet
    < $SESSION_DIR/review-prompt.md > $SESSION_DIR/devstral-review.md 2>$SESSION_DIR/devstral-stderr.log

  All 5 models execute simultaneously (5x parallelism!)

Message 3: Auto-Consolidation
  (Automatically triggered - don't wait for user to request)

  Task: senior-code-reviewer
    Prompt: "Consolidate 5 code reviews from:
             - $SESSION_DIR/claude-review.md
             - $SESSION_DIR/grok-review.md
             - $SESSION_DIR/qwen-coder-review.md
             - $SESSION_DIR/gpt5-review.md
             - $SESSION_DIR/devstral-review.md

             Apply consensus analysis:
             - Issues flagged by ALL 5 → UNANIMOUS (VERY HIGH confidence)
             - Issues flagged by 4 → STRONG (HIGH confidence)
             - Issues flagged by 3 → MAJORITY (MEDIUM confidence)
             - Issues flagged by 1-2 → DIVERGENT (LOW confidence)

             Prioritize by consensus level and severity.
             Write to $SESSION_DIR/consolidated-review.md"

Message 4: Present Results + Update Statistics
  # Track performance for each model (see Pattern 7)
  track_model_performance "claude-embedded" "success" 32 8 95
  track_model_performance "x-ai/grok-code-fast-1" "success" 45 6 87
  track_model_performance "qwen/qwen3-coder:free" "success" 52 5 82
  track_model_performance "openai/gpt-5.1-codex" "success" 68 7 89
  track_model_performance "mistralai/devstral-2512:free" "success" 48 5 84

  # Record session summary
  record_session_stats 5 5 0 68 245 3.6

  "Multi-model code review complete! 5 AI models analyzed your code.
   Session: $SESSION_ID

   Top 5 Issues (Prioritized by Consensus):
   1. [UNANIMOUS] Missing input validation on POST /api/users
   2. [UNANIMOUS] SQL injection risk in search endpoint
   3. [STRONG] Weak password hashing (bcrypt rounds too low)
   4. [MAJORITY] Missing rate limiting on authentication endpoints
   5. [MAJORITY] Insufficient error handling in payment flow

   Model Performance (this session):
   | Model                          | Time | Issues | Quality | Cost   |
   |--------------------------------|------|--------|---------|--------|
   | claude-embedded                | 32s  | 8      | 95%     | FREE   |
   | x-ai/grok-code-fast-1          | 45s  | 6      | 87%     | $0.002 |
   | qwen/qwen3-coder:free          | 52s  | 5      | 82%     | FREE   |
   | openai/gpt-5.1-codex        | 68s  | 7      | 89%     | $0.015 |
   | mistralai/devstral-2512:free   | 48s  | 5      | 84%     | FREE   |

   Parallel Speedup: 3.6x (245s sequential → 68s parallel)

   See $SESSION_DIR/consolidated-review.md for complete analysis.
   Performance logged to ai-docs/llm-performance.json"

Performance Impact:

• Sequential execution: 5 models × 3 min = 15 minutes
• Parallel execution: max(model times) ≈ 5 minutes
• Speedup: 3x with perfect parallelism

---

Pattern 2: Parallel Execution Architecture

Single Message, Multiple Tasks:

The key to parallel execution is putting ALL Task calls in a single message with the --- delimiter:

✅ CORRECT - Parallel Execution:

Task: agent1
  Prompt: "Task 1 instructions"
---
Task: agent2
  Prompt: "Task 2 instructions"
---
Task: agent3
  Prompt: "Task 3 instructions"

All 3 execute simultaneously.

Anti-Pattern: Sequential Execution

❌ WRONG - Sequential Execution:

Message 1:
  Task: agent1

Message 2:
  Task: agent2

Message 3:
  Task: agent3

Each task waits for previous to complete (3x slower).

Independent Tasks Requirement:

Each Task must be independent (no dependencies):

✅ CORRECT - Independent:
  Task: review code for security
  Task: review code for performance
  Task: review code for style

  All can run simultaneously (same input, different perspectives).

❌ WRONG - Dependent:
  Task: implement feature
  Task: write tests for feature (depends on implementation)
  Task: review implementation (depends on tests)

  Must run sequentially (each needs previous output).

Unique Output Files:

Each Task MUST write to a unique output file within the session directory:

✅ CORRECT - Unique Files in Session Directory:
  Task: reviewer1 → $SESSION_DIR/claude-review.md
  Task: reviewer2 → $SESSION_DIR/grok-review.md
  Task: reviewer3 → $SESSION_DIR/qwen-coder-review.md

❌ WRONG - Shared File:
  Task: reviewer1 → $SESSION_DIR/review.md
  Task: reviewer2 → $SESSION_DIR/review.md (overwrites reviewer1!)
  Task: reviewer3 → $SESSION_DIR/review.md (overwrites reviewer2!)

❌ WRONG - Fixed Directory (not session-based):
  Task: reviewer1 → ai-docs/reviews/claude-review.md  # May conflict with other sessions!

Wait for All Before Consolidation:

Do NOT consolidate until ALL tasks complete:

✅ CORRECT - Wait for All:
  Launch: Task1, Task2, Task3, Task4 (parallel)
  Wait: All 4 complete
  Check: results.filter(r => r.status === 'fulfilled').length
  If >= 2: Proceed with consolidation
  If < 2: Offer retry or abort

❌ WRONG - Premature Consolidation:
  Launch: Task1, Task2, Task3, Task4
  After 30s: Task1, Task2 done
  Consolidate: Only Task1 + Task2 (Task3, Task4 still running!)

---

Pattern 3: External Model Invocation via Bash+claudish

How External Models Are Invoked:

External AI models are invoked deterministically via Bash+claudish CLI. The orchestrator calls claudish directly — no LLM delegation needed. This is 100% reliable.

# Pattern: Bash tool with

<!-- Content truncated for initial SEO render. Open the source file tab for the full file. -->