swe-bench-lite

This skill provides a streamlined way to run the SWE-bench Lite benchmark with pre-configured defaults.

What This Skill Does

This skill runs a quick SWE-bench Lite evaluation with:

• 5 sample tasks (configurable)
• Verbose output for visibility
• Results saved to results.json
• Report saved to report.md

Prerequisites Check

Before running, verify:

1. Docker is running:

   docker ps
   

2. API key is set:

   echo $ANTHROPIC_API_KEY
   

3. Config file exists:

   • Check for mcpbr.yaml in the current directory
   • If missing, run mcpbr init to generate it

Default Command

The default command for SWE-bench Lite:

mcpbr run -c mcpbr.yaml --dataset SWE-bench/SWE-bench_Lite -n 5 -v -o results.json -r report.md

Customization Options

Users can customize the run by modifying:

• Sample size: Change -n 5 to any number (or remove for full dataset)
• Config file: Change -c mcpbr.yaml to point to a different config
• Verbosity: Use -vv for very verbose output
• Output files: Change results.json and report.md to different paths

Example Variations

Minimal quick test (1 task)

mcpbr run -c mcpbr.yaml -n 1 -v

Full evaluation (all ~300 tasks)

mcpbr run -c mcpbr.yaml --dataset SWE-bench/SWE-bench_Lite -v -o results.json

MCP-only (skip baseline)

mcpbr run -c mcpbr.yaml -n 5 -M -v -o results.json

Specific tasks

mcpbr run -c mcpbr.yaml -t astropy__astropy-12907 -t django__django-11099 -v

Expected Runtime & Cost

For 5 tasks with default settings:

• Runtime: 15-30 minutes (depends on task complexity)
• Cost: $2-5 (depends on task complexity and model used)

What to Do If It Fails

1. Docker not running: Start Docker Desktop
2. API key missing: Set with export ANTHROPIC_API_KEY="sk-ant-..."
3. Config missing: Run mcpbr init to generate default config
4. Config invalid: Check that {workdir} placeholder is in the args array
5. MCP server fails: Test the server command independently

After the Run

Once complete, you'll have:

• results.json: Full evaluation data with metrics, token usage, and per-task results
• report.md: Human-readable summary with resolution rates and comparisons
• Console output: Real-time progress and summary table

Review the results to see how your MCP server performed compared to the baseline!

Pro Tips

• Start with -n 1 to verify everything works before running larger evaluations
• Use --log-dir logs/ to save detailed per-task logs for debugging
• Compare multiple runs by changing the MCP server config between runs
• Use --baseline-results baseline.json to detect regressions between versions