download-docs

Download documentation files from various sources using manifest configurations.

Quick Start

Download from a specific manifest:

.claude/skills/download-docs/scripts/download.sh {manifest-name}

Download from all manifests:

.claude/skills/download-docs/scripts/download.sh

Output location: output/{manifest-name}/

Manifest Format

Manifests are JSON files in scripts/manifests/. Use $schema for IDE validation. Two source types are supported:

GitHub Source (for large doc sets)

Clones repository and extracts markdown files:

{
  "$schema": "./manifest.schema.json",
  "_source": {
    "type": "github",
    "repo": "owner/repo-name",
    "branch": "main",
    "path": "docs",
    "extensions": [".md", ".mdx"],
    "exclude": ["**/internal/**"],
    "noDefaultExcludes": false
  }
}

| Field | Required | Description | |-------|----------|-------------| | repo | Yes | GitHub repository (owner/name) | | branch | No | Branch to clone (default: "main") | | path | No | Directory path within repo (default: ".") | | extensions | No | File extensions to include (default: [".md", ".mdx"]) | | exclude | No | Additional glob patterns to exclude | | noDefaultExcludes | No | Set true to disable default excludes |

Default exclude patterns:

• CHANGELOG*, CONTRIBUTING*, LICENSE*, SECURITY*, CODE_OF_CONDUCT*
• .github/**, /test/, /tests/, /tests/
• /spec/, /fixtures/, /examples/, /node_modules/

URL Source (for individual files)

Downloads specific files from URLs:

{
  "$schema": "./manifest.schema.json",
  "_source": { "type": "url" },
  "files": {
    "category-name": {
      "doc-name": "https://example.com/path/to/doc.md"
    }
  }
}

URL Source with HTML Conversion

Downloads HTML and converts to markdown (requires pandoc):

{
  "$schema": "./manifest.schema.json",
  "_source": {
    "type": "url",
    "convert": "html"
  },
  "files": {
    "category-name": {
      "doc-name": "https://example.com/docs/page.html"
    }
  }
}

Note: HTML conversion only works for static HTML. SPA sites (React, Angular, etc.) will produce empty output.

Disabling a Manifest

Add _disabled to skip processing:

{
  "$schema": "./manifest.schema.json",
  "_source": {
    "type": "url",
    "_disabled": true,
    "_reason": "SPA site - requires Playwright handler"
  },
  "files": { ... }
}

Requirements

• jq - JSON parsing
• git - GitHub source cloning
• curl - URL downloads
• pandoc - HTML conversion (optional)

Adding New Manifests

Create a JSON file in scripts/manifests/ following the format above. The manifest filename (without .json) becomes the output directory name.