content-scout

YouTube monitoring → frame extraction → classification → transcription → daily brief.

⚠️ Success Criteria — VERIFY BEFORE REPORTING DONE

The pipeline is complete ONLY when the final JSON output contains "verdict": "COMPLETE".

After the pipeline finishes, verify:

python3 -c "
import json
log = json.load(open('content-vault/processing-log.json'))
print(f'Processed: {len(log[\"processedVideoIds\"])} videos')
print(f'Recent IDs: {log[\"processedVideoIds\"][-3:]}')
"

Required checks:

1. Pipeline JSON output shows "verdict": "COMPLETE"
2. content-vault/processing-log.json includes the video ID in processedVideoIds
3. content-vault/daily/YYYY-MM-DD/ contains the daily brief markdown

If any check fails, do NOT report success. Report which steps failed and why.

Pipeline Steps

Run the full pipeline or individual steps. The orchestrator is scripts/run_pipeline.py.

# Full daily pipeline
python3 scripts/run_pipeline.py --date 2026-02-24

# Single video
python3 scripts/run_pipeline.py --video-url "https://youtube.com/watch?v=VIDEO_ID"

# Resume from a specific step
python3 scripts/run_pipeline.py --date 2026-02-24 --resume-from classify

Step Order

1. select — select_videos.py — pick videos from monitored channels (config/channels.json)
2. download — download.py — download via yt-dlp (720p max, delays between downloads)
3. extract — extract_frames.py — extract frames at configured interval, deduplicate via perceptual hash
4. transcribe — transcribe.py — transcribe audio via OpenAI Whisper API
5. window — window_transcripts.py — split transcript into 30s windows
6. classify — classify_annotate.py — LLM classifies each frame (CHART_VISUAL, SLIDE, SCREEN, GRAPHIC, TALKING_HEAD)
7. compress — compress_and_store.py — compress kept frames to webp, store in content-vault
8. tags — build_tag_index.py — build searchable tag index from metadata
9. log — update_log.py — update processing log
10. notion — notion_sync.py — sync to Notion databases (optional, needs NOTION_TOKEN)
11. brief — generate_brief.py — generate daily markdown brief with takeaways
12. archive / cleanup — archive transcripts, clean tmp files

Configuration

All config lives in config/:

• channels.json — monitored YouTube channels with priority scores
• keywords.json — keyword relevance scoring
• watchlist.json — additional video URLs to process
• settings.json — pipeline parameters (frame interval, thresholds, limits, cron schedule)

Working Directory

Set CONTENT_SCOUT_WORKDIR to control where output goes. Defaults to cwd. Config files (config/) always resolve from the skill installation directory.

export CONTENT_SCOUT_WORKDIR=/path/to/project
python3 scripts/run_pipeline.py --date 2026-02-24

Output structure (created in workdir, NOT in the skill directory):

content-vault/           # Output directory (created automatically)
├── daily/YYYY-MM-DD/    # Frames and briefs per day
├── channels/            # Per-channel metadata
├── tags/                # Tag index
└── processing-log.json  # Run history
tmp/                     # Working directory (cleaned up after pipeline)
├── audio/               # Extracted audio files
├── frames/              # Raw extracted frames
└── transcripts/         # Whisper output

Environment Variables

| Variable | Required | Purpose | |----------|----------|---------| | OPENAI_API_KEY | For transcription | Whisper API transcription | | NOTION_TOKEN | For Notion sync | Notion API integration | | ANTHROPIC_API_KEY | For classification | Frame classification via Claude |

Key Utilities (_common.py)

Shared utilities used by all scripts: resolve_path, load_json, save_json, ensure_dir, setup_logging, run_command, ROOT_DIR, ContentScoutError, normalize_slug, utcnow, utc_today_str, parse_upload_date.

ROOT_DIR resolves to the skill's parent directory at runtime (two levels up from scripts/). All path resolution is relative to ROOT_DIR.

Dependencies

yt-dlp
ffmpeg (system)
Pillow
imagehash
python-slugify
openai (optional — transcription)
anthropic (optional — classification)
notion-client (optional — Notion sync)