context-analysis

Analyze plain text documents to understand their semantic structure and token distribution. This skill helps visualize how tokens are distributed across different semantic components of a document.

Workflow

When analyzing a document, follow these steps:

1. Parse the Text

Convert the input text file to JSON format:

./scripts/parse.sh input.txt > parsed.json

2. Count Tokens

Add token counts using tiktoken (GPT-4o encoding):

./scripts/count-tokens.sh parsed.json > counted.json

3. Segment Large Parts

For parts with more than 500 tokens, identify semantic breakpoints and split them.

You (Claude) should do this directly:

• Read the text of each large part
• Identify natural semantic boundaries (topic changes, section breaks, logical divisions)
• Split into coherent chunks
• Update the JSON with new parts (use IDs like part-1.1, part-1.2, etc.)

After segmenting, recount tokens:

./scripts/count-tokens.sh segmented.json > recounted.json

4. Identify Components

Analyze the document and identify the distinct semantic components it contains.

You (Claude) should do this directly:

• Read all parts
• Identify categories/themes (e.g., "introduction", "methodology", "examples", "conclusion")
• Assign each part to a component
• Add to JSON: components array and component field on each part
• Calculate component_tokens totals

5. Assign Colors

Run the colorization script to assign consistent colors:

./scripts/colorise.sh componentised.json > colored.json

6. Generate Visualizations

./scripts/waffle-chart.sh colored.json > waffle.html
./scripts/bar-chart.sh colored.json > bar-chart.html
./scripts/text-view.sh colored.json > text-view.html

Quick Commands

For a complete analysis pipeline:

# Parse and count
./scripts/parse.sh input.txt > /tmp/1-parsed.json
./scripts/count-tokens.sh /tmp/1-parsed.json > /tmp/2-counted.json

# You segment and componentise the JSON directly, then:
./scripts/colorise.sh /tmp/3-componentised.json > /tmp/4-colored.json
./scripts/waffle-chart.sh /tmp/4-colored.json > waffle.html

For multiple files:

./scripts/group.sh input_folder/ output_dir/

Data Format

The JSON format used throughout:

{
  "source": "filename.txt",
  "parts": [
    {
      "id": "part-1",
      "text": "The actual text content...",
      "token_count": 150,
      "component": "introduction"
    }
  ],
  "components": ["introduction", "methodology", "results"],
  "component_tokens": {
    "introduction": 500,
    "methodology": 1200,
    "results": 800
  },
  "total_tokens": 2500
}

Segmentation Guidelines

When segmenting large parts (>500 tokens):

1. Look for natural breakpoints:

   • Markdown headings (#, ##, etc.)
   • Topic transitions
   • Logical section boundaries
   • List groupings

2. Create semantically coherent chunks:

   • Each segment should cover one topic/concept
   • Aim for 100-500 tokens per segment
   • Preserve context within segments

3. Update IDs hierarchically:

   • part-1 splits into part-1.1, part-1.2, etc.

Component Identification Guidelines

When identifying components:

1. Read all parts to understand the document structure

2. Identify 3-10 distinct semantic categories

3. Use descriptive, lowercase names with underscores

4. Common patterns:

   • Documents: introduction, methodology, results, conclusion
   • Technical: configuration, implementation, examples, api_reference
   • Instructional: overview, prerequisites, steps, troubleshooting

5. Assign each part to exactly one component

6. Use other for parts that don't fit elsewhere

Available Colors

Components are assigned these colors:

• blue - Primary content, introductions
• emerald - Workflows, processes, methodology
• purple - Style, personality, guidelines
• orange - Context, examples, highlights
• indigo - Code, technical content
• slate - Environment, configuration
• gray - Tools, utilities, other

References

See the references/ folder for detailed documentation on each step.