Video analysis and editing with FFmpeg and Whisper. This skill should be used when video files are shared (.mov, .mp4, .avi, etc.) or when you encounter "cannot read binary files" errors for video files, when users request video analysis or summarization, or when users ask to edit videos (clip, merge, split).
Comprehensive video processing skill combining editing capabilities with multi-modal analysis (visual frames + audio transcription).
Activate this skill when:
What this means for users:
When you share a video file, Claude will automatically recognize it and offer to analyze it properly using the video-toolkit, rather than attempting to read the binary file directly.
Required Dependencies:
FFmpeg - Video processing and frame extraction
scripts/install_dependencies.shffmpeg -versionPython 3.8+ with virtual environment
scripts/install_dependencies.shOpenAI Whisper - Speech transcription (local, no API key required)
.venvGoogle Gemini API - Audio analysis and music detection
.venvShazam API - Music identification
.venv (shazamio)Setup:
Step 1: Install Dependencies
Run the installation script on first use:
bash ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/install_dependencies.sh
This creates a Python virtual environment at ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/ and installs the required packages (ffmpeg-python, openai-whisper, google-genai, shazamio).
Step 2: Configure API Keys
Set up Gemini API key for audio analysis:
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/setup_api_keys.py gemini YOUR_API_KEY
Get your Gemini API key from: https://aistudio.google.com/app/apikey
Optional: If you want to use RapidAPI's Shazam endpoint instead of the public one:
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/setup_api_keys.py shazam YOUR_RAPIDAPI_KEY
Note: Shazam/music identification works without an API key using shazamio's public endpoint.
IMPORTANT: Check API Key Setup First
Before analyzing videos with audio, verify that the Gemini API key is configured:
/Users/emdash/Dev/claude-code-plugins/emdashcodes/.video-toolkit-config.jsongemini.apiKey, run the setup first (see Prerequisites above)When analyzing a video file, follow this comprehensive workflow:
1. Frame Extraction
Extract visual frames using either interval mode or scene detection mode:
Interval Mode (time-based):
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/extract_frames.py <video_path> <interval_seconds> <output_dir>
interval_seconds: Time between frames (e.g., 2 for every 2 seconds)output_dir: Temporary directory for framesScene Detection Mode (change-based):
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/extract_frames.py <video_path> --scene-detect <output_dir> [threshold]
threshold: Sensitivity (0.0-1.0, optional, default: 0.02)Choosing the Right Mode:
| Video Type | Recommended Mode | Threshold/Interval | Recommended Value | |------------|------------------|-------------------|-------------------| | Screen recordings | Interval | 2-3 seconds | Use 2-3s | | Presentations/Slides | Scene detection | 0.05 - 0.10 | Use 0.05 | | Movies/Hard cuts | Scene detection | 0.20 - 0.30 | Use 0.20 | | Surveillance/Static | Interval | 5-10 seconds | Use 5s | | Interviews/Dialogue | Interval | 3-5 seconds | Use 3s | | Action videos | Interval | 1-2 seconds | Use 1s |
Agent Decision Making:
IMPORTANT: Evaluate and Re-run if Needed
Scene Detection Limitation:
Scene detection only captures frames when visual content changes significantly. If the video has long static periods (e.g., same screen for 30+ seconds), scene detection will miss that content entirely. For videos with static content, USE INTERVAL MODE instead with --mode interval --interval 2 or --interval 3.
2. Audio Extraction
Extract audio track using scripts/extract_audio.py:
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/extract_audio.py <video_path> <output_wav_path>
3. Audio Analysis (Sequential Workflow)
The audio analysis follows a sequential workflow to maximize accuracy and efficiency:
3a. Speech Transcription (Whisper - Local)
Transcribe speech using Whisper (runs locally, no API key required):
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/transcribe_audio.py <wav_path> [model_name]
model_name: Whisper model (default: base)
3b. Audio Understanding (Gemini Audio API)
Analyze audio comprehensively using Gemini Audio API:
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/analyze_audio_gemini.py <wav_path> [output_markdown]
[output_markdown]gemini_audio.json in same directoryhas_music flag and music_segments array for downstream processing3c. Music Identification (Shazam - Conditional)
If Gemini detects music, identify songs using Shazam:
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/identify_music.py <wav_path> <gemini_json> [output_markdown]
<gemini_json>: Path to the gemini_audio.json file created by analyze_audio_gemini.pyhas_music flag in JSON)Sequential Workflow Summary:
1. Whisper (local) → Speech transcription
2. Gemini Audio (API) → Detect music + timestamps, analyze non-speech audio
3. FFmpeg → Extract music segments (if music detected)
4. Shazam (API) → Identify songs (if music detected)
This approach ensures:
Gemini vs Shazam - Complementary Capabilities:
Gemini Audio API: AI-powered audio understanding
Shazam: Audio fingerprint matching
Both work together: Gemini describes what the music sounds like, Shazam identifies which specific track it is.
3.5. Context Clarification with Intelligent Question Generation
CRITICAL STEP: After gathering initial data (frames, audio, music identification), ask the user context questions to understand the video's perspective and avoid misidentification.
Why this step is critical: Visual evidence alone can be ambiguous. For example, seeing streaming platform with a BRB screen could mean either:
Only the user can clarify their relationship to the content.
Step 1: Complete Initial Data Gathering
Before asking any questions, run the full analysis pipeline through Step 3c:
Step 2: Build Contextual Evidence
Review what was gathered:
Step 3: Ask Context Question (Q1) with Evidence
Present Q1 with context from your gathered evidence to help the user answer:
# Build context string from evidence
# Adapt summary based on what was found in the video
# Example 1: Stream Viewer
evidence_summary = """I've analyzed the video and found:
- Visual: Streaming platform interface with BRB screen, person visible, workspace shown
- Audio: Music playing ("Song Title" by Artist Name), minimal speech
- Duration: 1:57
"""
# Example 2: Tutorial/How-to
evidence_summary = """I've analyzed the video and found:
- Visual: Person's hands visible working with materials, step-by-step demonstrations
- Audio: Instructional narration explaining process, background music
- Duration: 8:34
"""
# Example 3: Screen Recording (Software Demo)
evidence_summary = """I've analyzed the video and found:
- Visual: VS Code editor with code visible, cursor movements, terminal commands
- Audio: Voiceover explaining code changes, keyboard typing sounds
- Duration: 5:12
"""
# Example 4: Screen Recording (Gameplay)
evidence_summary = """I've analyzed the video and found:
- Visual: Game interface with HUD elements, character movement, gameplay
- Audio: Game sounds, background music, occasional commentary
- Duration: 12:45
"""
# Example 5: Personal/Casual
evidence_summary = """I've analyzed the video and found:
- Visual: Person on camera in home setting, handheld/phone camera movement
- Audio: Person speaking directly to camera, ambient room sounds
- Duration: 2:18
"""
# Example 6: Personal/Casual (Phone Clip)
evidence_summary = """I've analyzed the video and found:
- Visual: Quick clip of outdoor scene, vertical/portrait orientation, casual framing
- Audio: Ambient sounds, brief speech, wind noise
- Duration: 0:43
"""
# Example 7: Event/Performance
evidence_summary = """I've analyzed the video and found:
- Visual: Stage with performers, audience visible, concert venue setting
- Audio: Live music performance, crowd noise, applause
- Duration: 3:56
"""
# Example 8: Presentation/Slides
evidence_summary = """I've analyzed the video and found:
- Visual: Slide deck with bullet points and diagrams, presenter occasionally visible
- Audio: Presenter speaking, slide transition sounds
- Duration: 18:24
"""
# Example 9: Casual Documentation (Workspace)
evidence_summary = """I've analyzed the video and found:
- Visual: Desk setup with monitors and equipment, informal camera angles
- Audio: Background music, ambient office sounds, no narration
- Duration: 1:34
"""
# Example 10: Behind-the-Scenes
evidence_summary = """I've analyzed the video and found:
- Visual: Production equipment, camera setup, people working on set
- Audio: Technical discussions, equipment sounds, music playing
- Duration: 4:27
"""
# Template for implementation:
# Select the most relevant description based on gathered evidence
def build_evidence_summary(frames_analysis, audio_analysis, music_data, duration):
"""
Generate evidence summary based on analyzed content
Returns contextual summary that helps user identify video type
"""
visual_desc = describe_visual_content(frames_analysis)
audio_desc = describe_audio_content(audio_analysis, music_data)
return f"""I've analyzed the video and found:
- Visual: {visual_desc}
- Audio: {audio_desc}
- Duration: {format_duration(duration)}
"""
# Present Q1 with evidence context
AskUserQuestion(
questions=[
{
"question": f"{evidence_summary}\n\nWhat type of video is this?",
"header": "Video Type",
"multiSelect": false,
"options": [
{
"label": "Tutorial/How-to",
"description": "Teaching or demonstrating something step-by-step"
},
{
"label": "Personal",
"description": "Personal video, daily life content, a moment or activity, clip from phone"
},
{
"label": "Screen recording",
"description": "Recording your own screen (software demo, debugging video, gameplay, etc.)"
},
{
"label": "Watching something",
"description": "Recording yourself viewing someone else's content"
},
{
"label": "Event/Performance",
"description": "Concert, presentation, ceremony, or live event"
},
{
"label": "Other/Not sure",
"description": "Type a custom description of what this video shows"
}
]
}
]
)
Key Patterns in Evidence Summaries:
Be specific about what's visible: Not just "screen content" but "VS Code editor with code" or "Streaming platform interface with BRB screen"
Mention identifying audio: "Music playing (song identified)" vs. "instructional narration" vs. "live performance audio"
Note video characteristics: Portrait orientation, handheld movement, production quality
Keep it concise: 2-3 bullet points max, user can read it quickly
Help user self-identify: Evidence should make the correct option obvious
Step 4: Generate Smart Follow-up (Q2) Based on Evidence + Q1 Answer
CRITICAL: Generate Q2 dynamically based on:
Skip Q2 entirely if:
Generate custom Q2 if needed:
If Q1 = "Watching something" AND you identified what they were watching:
# Example: You saw streaming platform + identified music
AskUserQuestion(
questions=[
{
"question": "I can see streaming platform with a BRB screen and identified 'Song Title' by Artist Name playing in the background. Is this what was happening?",
"header": "Confirm Context",
"multiSelect": false,
"options": [
{
"label": "Yes, that's correct",
"description": "Watching a stream with music playing in my space"
},
{
"label": "Partially correct",
"description": "Some of that is right, but let me clarify"
},
{
"label": "No, different context",
"description": "That's not quite what was happening"
}
]
}
]
)
If Q1 = "Watching something" AND you're uncertain what:
AskUserQuestion(
questions=[
{
"question": "What were you watching?",
"header": "Content Source",
"multiSelect": true, # Allow multiple selections
"options": [
{
"label": "Live stream",
"description": "Streaming platforms, Twitch, YouTube live, etc."
},
{
"label": "Video content",
"description": "YouTube video, movie, TV show, etc."
},
{
"label": "Music playing",
"description": "Music was playing in your space"
},
{
"label": "Other media",
"description": "Game, presentation, or other content"
}
]
}
]
)
If Q1 = "Personal" AND video shows workspace/music:
# You already know music was playing and workspace is visible
# Generate confirmation question instead of generic tags
AskUserQuestion(
questions=[
{
"question": "I can see your workspace and 'Song Title' playing. What tags describe this video?",
"header": "Context Tags",
"multiSelect": true,
"options": [
{
"label": "Music playing",
"description": "Background music in your space"
},
{
"label": "Workspace/setup",
"description": "Showing desk, equipment, or environment"
},
{
"label": "Activity in progress",
"description": "Doing something while recording"
},
{
"label": "Artistic/experimental",
"description": "Creative or artistic intent"
}
]
}
]
)
If Q1 = "Screen recording":
# Only ask if genuinely uncertain from frames
# You likely already saw what's on screen
AskUserQuestion(
questions=[
{
"question": "I can see {specific_app_or_content}. What were you recording?",
"header": "Screen Content",
"multiSelect": false,
"options": [
{
"label": "Software/application",
"description": "Demonstrating a program or workflow"
},
{
"label": "Gameplay",
"description": "Playing a game"
},
{
"label": "Design/creative work",
"description": "Working in design tools, editors, etc."
},
{
"label": "General computer use",
"description": "Browsing, chatting, or mixed activities"
}
]
}
]
)
If Q1 = "Tutorial/How-to" or "Event/Performance": → Skip Q2 - these are self-explanatory, proceed to analysis
If Q1 = "Other/Not sure" (custom text provided): → Use the custom text to inform analysis, skip Q2 unless truly necessary
After Q1 (and Q2 if asked), store structured metadata:
video_metadata = {
"video_type": user_q1_answer, # "Watching something"
"perspective": derived_perspective, # "First-Person Viewer"
"content_tags": user_q2_answers, # ["live stream", "music playing"]
"evidence": {
"visual": visual_summary, # From frames
"audio": audio_summary, # From Gemini
"music": music_identification, # From Shazam
"duration": video_duration
}
}
Use this metadata throughout remaining analysis:
Scenario: User shares video showing streaming platform with BRB screen, person visible, music playing
Data Gathered (Steps 1-3d):
Q1 with Context:
I've analyzed the video and found:
- Visual: Streaming platform interface with BRB screen, person visible, workspace shown
- Audio: Music playing ("Song Title" by Artist Name), minimal speech
- Duration: 1:57
What type of video is this?
→ User selects: "Watching something"
Smart Q2 Generation:
# You already know:
# - Streaming platform with BRB = watching a stream
# - Music identified = "Song Title" playing in their space
# - Person visible = user on camera
# Instead of generic "What were you watching?", generate specific confirmation:
"I can see streaming platform with a BRB screen and identified 'Song Title'
<!-- Content truncated for initial SEO render. Open the source file tab for the full file. -->
Generate or edit images via Gemini 3 Pro Image (Nano Banana Pro).
Batch-generate images via OpenAI Images API. Random prompt sampler + `index.html` gallery.
Generate spectrograms and feature-panel visualizations from audio with the songsee CLI.
Extract frames or short clips from videos using ffmpeg.
Search GIF providers with CLI/TUI, download results, and extract stills/sheets.
Category:media-generate