doing-a-simple-two-stage-fanout

What this skill does

# Two-Stage Fan-Out Analysis

Divide a corpus across Worker subagents, review with Critic subagents, synthesize with a Summarizer. Every stage writes to files; every subagent gets its own task.

## Overview

```
Corpus → [Workers] → [Critics] → Summarizer → Report
```

**Workers** each analyze a slice of the corpus. **Critics** each review all Worker reports for a subset of segments, checking for gaps and inconsistencies. A single **Summarizer** reads all Critic reports and produces the final output.

## Step 0: Gather Inputs

If the user's intent is not already clear, ask two questions using AskUserQuestion:

**Question 1: What to analyze.** Ask what corpus to analyze and what the analysis goal is. Skip if obvious from context.

**Question 2: Effort level.** Present these options in this order (do not reorder to put recommended first):

| Level | SEGMENTS_PER | REVIEWS_PER | When to use |
|-------|-------------|-------------|-------------|
| Some effort | 3 | 2 | Default for most analyses |
| A lot of effort | 3 | 3 | When thoroughness matters more than speed |
| Herculean effort | 2 | 3 | When you cannot afford to miss anything |

Recommend one if you have enough context, by appending "(Recommended)" to that option's label. But keep the options in the order shown above regardless.

**Definitions:**
- `SEGMENTS_PER` — how many corpus segments each Worker processes
- `REVIEWS_PER` — how many independent Critic reviews each segment receives

## Step 1: Compute the Layout

You need to determine how many segments, workers, and critics the analysis requires. This depends on corpus size and agent context capacity.

### Estimating Corpus Size

If you have file paths, estimate tokens:
- **Prose**: 1 token per 4 characters
- **Source code**: 1 token per 3 characters
- **By word count**: 1 word is roughly 1.33 tokens

Use the Bash tool to count characters: `wc -c file1 file2 ...` or `find /path -type f -exec cat {} + | wc -c`.

For more precise estimates, run the [compute_layout.py](./compute_layout.py) script bundled with this skill:

```bash
python3 /path/to/compute_layout.py --corpus-chars 800000 --segments-per 3 --reviews-per 2
python3 /path/to/compute_layout.py --corpus-files file1.txt file2.txt --segments-per 3 --reviews-per 2
python3 /path/to/compute_layout.py --corpus-tokens 200000 --segments-per 3 --reviews-per 2 --json
```

### Computing Manually

If you cannot run the script, compute by hand. **Use the Bash tool with `python3 -c "..."` for all arithmetic** — do not compute in your head.

**Agent capacity:**
```
AGENT_CONTEXT  = 200,000 tokens
RESERVED       = 35%  (for prompt, reasoning, output)
AVAILABLE      = AGENT_CONTEXT * 0.65 = 130,000 tokens
SEGMENT_BUDGET = AVAILABLE / SEGMENTS_PER
```

**Segment count:**
```
OVERLAP  = 10% of SEGMENT_BUDGET
STRIDE   = SEGMENT_BUDGET - OVERLAP
SEGMENT_COUNT = ceil((CORPUS_TOKENS - SEGMENT_BUDGET) / STRIDE) + 1
```

If `CORPUS_TOKENS <= SEGMENT_BUDGET`, then `SEGMENT_COUNT = 1` (no fan-out needed).

**Agent counts:**
```
WORKER_COUNT = ceil(SEGMENT_COUNT / SEGMENTS_PER)
TOTAL_CRITIC_ASSIGNMENTS = SEGMENT_COUNT * REVIEWS_PER
CRITIC_COUNT = ceil(TOTAL_CRITIC_ASSIGNMENTS / SEGMENTS_PER)
```

### What These Numbers Mean

- Each **Worker** reads `SEGMENTS_PER` consecutive segments of raw corpus and writes an analysis report.
- Each **Critic** reads all Worker reports that cover a subset of segments and writes a review.
- Each segment gets reviewed by `REVIEWS_PER` different Critics (redundancy for thoroughness).

### Assigning Critics to Segments

The critic count tells you how many critics to create, but you also need to decide which segments each critic reviews. Use round-robin assignment to distribute `REVIEWS_PER` critic passes evenly across segments:

```
For each segment S (1 to SEGMENT_COUNT):
    Assign REVIEWS_PER different critics to review S
    Rotate through critics: critic index = (S * review_pass + offset) % CRITIC_COUNT
```

In practice, use `python3 -c "..."` to generate the assignment table. Example for 6 segments, 4 critics, REVIEWS_PER=2:

```
C01 reviews: S01, S03, S05
C02 reviews: S02, S04, S06
C03 reviews: S01, S04, S06
C04 reviews: S02, S03, S05
```

Each segment appears in exactly 2 critics' lists. Each critic reads the Worker reports that cover its assigned segments. Include this assignment table in the orchestration plan so the mapping is explicit and verifiable.

## Step 2: Set Up the Temp Directory

If the user specified a working directory, use it. Otherwise, create one:

```bash
WORK_DIR=$(mktemp -d -t fanout-XXXXXX)
mkdir -p "$WORK_DIR/segments" "$WORK_DIR/workers" "$WORK_DIR/critics"
```

All paths in prompts and file references are **absolute paths**. Subagents cannot resolve relative paths reliably.

## Step 3: Enter Plan Mode and Write the Orchestration Plan

Enter plan mode. Write a plan document that includes:

1. **Layout summary**: corpus size, segment count, worker count, critic count, effort level
2. **Fan-out diagram**: a Mermaid diagram showing the pipeline (see [diagram-templates.md](./diagram-templates.md) for syntax). For large layouts (>10 workers), collapse worker ranges (e.g., `W01-W10`) into summary nodes. If the user requests Graphviz instead, use the DOT template from the same file.
3. **Worker assignment table**: which segments each Worker handles (e.g., `W01: S01-S03`)
4. **Critic assignment table**: which segments each Critic reviews, generated using the round-robin method from Step 1. Verify that each segment appears exactly `REVIEWS_PER` times across all critics.
5. **Stage descriptions**: for each stage (Workers, Critics, Summarizer), describe what agents will do, their input/output paths, and which agents run in parallel
6. **File layout**: show the directory tree that will be produced

Do not include time estimates in the plan. Agent execution time is unpredictable and estimates are misleading.

Exit plan mode. Do not proceed until the user approves the plan.

### Diagram Guidelines

Worker nodes should show their segment assignments: `W01<br/>S01-S03`. Critic nodes show their review scope. Cap visible nodes at ~15; collapse ranges for larger layouts. See [diagram-templates.md](./diagram-templates.md) for full Mermaid and Graphviz templates with styling.

## Step 4: Create All Tasks

Before launching any subagents, create ALL tasks upfront using TaskCreate:

- One task per Worker (`W01`, `W02`, ...)
- One task per Critic (`C01`, `C02`, ...)
- One task for the Summarizer

Then set up dependencies with TaskUpdate `addBlockedBy`:
- Each Critic task is blocked by the Worker tasks whose segments it reviews
- The Summarizer task is blocked by all Critic tasks

This creates the full dependency graph before any work starts.

## Step 5: Launch Workers

Mark Worker tasks as `in_progress`, then launch all Workers in parallel (one Task tool call per worker, all in the same message).

### Worker Prompt Template

Each Worker gets a prompt structured like this:

```
You are {WORKER_NAME}, a corpus analysis worker.

## Your Assignment
Analyze segments {FIRST_SEG} through {LAST_SEG} of the corpus.

## Input
Read these files:
- {ABSOLUTE_PATH_TO_SEGMENT_FILE_1}
- {ABSOLUTE_PATH_TO_SEGMENT_FILE_2}
- ...

## Analysis Goal
{WHAT_THE_USER_WANTS_ANALYZED}

## Output Format
Write your report to: {ABSOLUTE_PATH_TO_WORK_DIR}/workers/{WORKER_NAME}.md

Structure your report as:

### Summary
2-3 sentence overview of findings for your segments.

### Detailed Findings
For each significant finding:
- **Finding**: one-line description
- **Location**: file/section where found
- **Evidence**: relevant quote or reference
- **Significance**: why this matters

### Segment Coverage
List each segment you analyzed and confirm you read it completely.
If any segment was too large to process fully, state which parts you skipped.
```

Adapt the analysis goal and output format to match what the user asked for. The template above is a starting point — be specific about what constitutes a "findin