seo-research

What this skill does

# DataForSEO SEO Research

Answer SEO questions by routing them to the right DataForSEO API v3 endpoint(s), fetching the **full raw response** to a file, and analysing that file. DataForSEO is a paid API — every call costs money (reported in the response), so pick the right endpoint and confirm scope before running broad/expensive queries.

The core principle: **fetch raw, save to `/tmp`, then analyse the saved file.** Never try to slim or pre-extract the API response on the way in — the responses are rich and the schema changes as DataForSEO adds fields, so anything you drop at fetch time is data you can't get back. With a large context window there's no need to extract; read the whole raw file and pull what you need from it during analysis.

## Prerequisites

Environment variables (set in `~/.bashrc`):
- `DATAFORSEO_LOGIN` — DataForSEO account login (email)
- `DATAFORSEO_PASSWORD` — **API password** from https://app.dataforseo.com/api-access (generated by DataForSEO, *not* the account password)

If either is missing, tell the user exactly which one and how to set it (`export DATAFORSEO_LOGIN=...` / `export DATAFORSEO_PASSWORD=...` in `~/.bashrc`, then restart the shell). Do not proceed without both. To test without spending money, add `--sandbox` to the script — it returns dummy data with the same structure, free of charge.

## The model (read `references/api-mechanics.md` for detail)

- **Uniform API:** every endpoint uses HTTP Basic auth, a POST body that is a **JSON array of task objects**, and a response envelope `tasks[] → result[] → items[]`. One script (`dfs.mjs`) calls any of them.
- **Prefer LIVE endpoints** (`.../live`) for interactive research — instant, no polling. Use task-based (`task_post`/`task_get`) only for cheap, high-volume batches.
- **Success = top-level `status_code` 20000.** The script enforces this and surfaces errors with hints.
- **Cost is in every response** — report it to the user after each call.
- Most endpoints need a **location and language** (`location_name`/`location_code` + `language_name`/`language_code`).

## Workflow

### 1. Locate the script
```
Glob: **/seo-research/scripts/dfs.mjs   → store as SCRIPT_PATH
```
The install path varies across systems, so resolve it with Glob rather than hardcoding.

### 2. Plan the fetches
Map the user's question to one or more endpoints using the routing table below. A single question often needs several endpoints (e.g. "research these competitors" → `ranked_keywords` + `competitors_domain` + `domain_rank_overview`). For each one, read its reference file to get the exact parameters before building the payload.

Resolve location/language: if the user named a country/market, use it; otherwise ask (via `AskUserQuestion`) or default to the project's primary market and say so.

Present the plan as a short checklist so the user can see what will run and what it will roughly cost:
```
Planned fetches:
1. ranked_keywords      — keywords nike.com ranks for (US/en, limit 500)
2. competitors_domain   — nike.com organic competitors (US/en, limit 50)
3. domain_rank_overview — nike.com traffic/positions scorecard (US/en)
```
For broad or potentially expensive queries (large `limit`/`depth`, many tasks, SERP `live/advanced`), confirm before running.

### 3. Fetch — raw to `/tmp`

**Build each payload** as a JSON array of task objects per the reference. For anything beyond a trivial payload, write it to a temp file to avoid shell-quoting issues.

**One endpoint → run it directly** with `Bash`:
```bash
node "$SCRIPT_PATH" \
  --endpoint "/v3/dataforseo_labs/google/ranked_keywords/live" \
  --payload-file /tmp/dfs-payload-ranked.json \
  --output /tmp/dfs-raw-ranked-$(date +%s).json
```
The script saves the **full raw response** to `--output`, prints a compact summary (status, cost, per-task `result_count`/`items_count`) to stdout, and the cost to stderr. Add `--sandbox` to test free, `--method GET` for GET utility endpoints (locations/languages, `task_get`).

**Two or more endpoints → fetch them in parallel via subagents.** Don't run them one after another — DataForSEO calls have real latency, so dispatch all of them in the **same turn** as parallel `Task` (general-purpose) subagents and let the responses come back asynchronously. Each subagent does the heavy plumbing (reads its reference, builds the payload, runs the script, handles errors) in its own context, so the main context stays clean. Give each subagent this brief:

```
You are fetching one DataForSEO endpoint. Do exactly this, nothing more:
- Read the reference file: <path to references/*.md for this endpoint>
- Endpoint: <path>
- Goal: <what to fetch, e.g. "keywords nike.com ranks for">
- Constraints: location=<...>, language=<...>, limit=<...>, plus any filters
- Build the JSON-array payload, write it to /tmp/dfs-payload-<slug>.json
- Run: node "<SCRIPT_PATH>" --endpoint "<path>" --payload-file <payload> --output /tmp/dfs-raw-<slug>-<unique>.json
- Return ONLY: the endpoint, the output file path, status_code, cost, and items_count.
- Do NOT read, summarise, or paste the response body. Do NOT use --raw. The raw file on disk is the deliverable.
```

Collect the returned file paths and costs. If a subagent reports a non-20000 status, surface its error and stop — don't retry blindly.

### 4. Analyse the raw files
Read the saved raw file(s) directly and answer the user's question from them — every field DataForSEO returned is there, nothing was dropped on the way in. The data lives under `tasks[] → result[] → items[]`; each reference describes what those items contain.

For large files, `jq` is a convenient way to slice out the rows you want to present (it's a tool for *reading* the saved raw, not for pre-extraction):
```bash
jq -r '.tasks[0].result[0].items[]
        | [.keyword_data.keyword,
           .ranked_serp_element.serp_item.rank_absolute,
           .keyword_data.keyword_info.search_volume] | @tsv' /tmp/dfs-raw-ranked-*.json | head -50
```
If `jq` is unavailable, use a short `node -e` snippet or `Read` the file. Present results as a compact table relevant to the question, and report the total cost across all fetches.

---

## Endpoint routing table

Pick by what the user wants to know. Paths are appended to `https://api.dataforseo.com`. Read the linked reference to build the payload.

### Keyword research → `references/labs-keyword-research.md` (+ `keywords-data.md`)

| The user wants… | Endpoint |
|---|---|
| Keyword ideas from seed terms (broad topical set) | `/v3/dataforseo_labs/google/keyword_ideas/live` |
| Long-tail keywords containing a phrase | `/v3/dataforseo_labs/google/keyword_suggestions/live` |
| "Searches related to" / topic clusters | `/v3/dataforseo_labs/google/related_keywords/live` |
| Full metrics for a known keyword list | `/v3/dataforseo_labs/google/keyword_overview/live` |
| Keyword difficulty for many keywords | `/v3/dataforseo_labs/google/bulk_keyword_difficulty/live` |
| Search intent classification | `/v3/dataforseo_labs/google/search_intent/live` |
| **Official Google Ads** volume/CPC/competition | `/v3/keywords_data/google_ads/search_volume/live` |
| Volume/CPC trend over time | `/v3/dataforseo_labs/google/historical_keyword_data/live` |
| Relative interest trend (Google Trends) | `/v3/keywords_data/google_trends/explore/live` |
| PPC traffic/cost forecast at a bid | `/v3/keywords_data/google_ads/ad_traffic_by_keywords/live` |

For search volume: DataForSEO Labs (`keyword_overview`) gives richer, more precise metrics; Google Ads `search_volume` gives the official rounded Ads numbers. Choose based on what the user needs.

### Competitor & domain analysis → `references/labs-domain-competitor-research.md`

| The user wants… | Endpoint |
|---|---|
| Which keywords a domain/page ranks for (+ positions) | `/v3/dataforseo_labs/google/ranked_keywords/live` |
| A domain's SEO scorecard (traffic, positions) | `/v3/dataforseo_labs/google/domain_rank_overview/live` |
| Find a domain's organic competitors |