macro-intelligence

What this skill does

# Macro Intelligence Skill v1.0 — Agent Instructions

## Purpose
Unified macro intelligence feed. Reads news from 7 sources (NewsNow, Polymarket, Telegram, 6551.io OpenNews, Finnhub, FRED, Fear & Greed Index), classifies macro events, scores sentiment, generates AI insights, and exposes clean signals via HTTP API. **No trading logic** — downstream skills consume signals.

## Architecture

```
  NewsNow (HTTP, 120s) ──────┐
  Polymarket (HTTP, 120s) ────┤
  Finnhub (HTTP, 180s) ───────┤──→ process_signal() ──→ UnifiedSignal ──→ API :3252
  6551.io OpenNews (WebSocket)─┤    │ noise filter       │ classify       │ sentiment
  Telegram (Telethon WS) ─────┘    │ dedup              │ reputation     │ AI insight
                                    │                    │ token extract  │ store
  FRED (HTTP, 3600s) ──────────→ context data ──→ /api/fred + significant change → process_signal()
  Fear & Greed (HTTP, 300s) ───→ context data ──→ /api/fng
  Price Tickers (HTTP, 60s) ───→ context data ──→ /api/prices (SPY, GLD, SLV, BTC, ETH)
```

## Startup Protocol

1. `python3 macro_news.py` — starts all collectors + HTTP server on `:3252`
2. `python3 macro_news.py setup` — interactive mode to list Telegram groups/channels

### Requirements
- Python 3.9+
- `pip install telethon` (optional — runs without it)
- `pip install websockets` (optional — needed for 6551.io OpenNews WebSocket)
- Env: `ANTHROPIC_API_KEY` for LLM classification + AI insights (optional)
- Env: `TG_API_ID`, `TG_API_HASH` (or set in config.py)
- Env: `OPENNEWS_TOKEN` for 6551.io (free — get token at https://6551.io/mcp)
- Env: `FINNHUB_API_KEY` for Finnhub market news (free — register at https://finnhub.io)
- Env: `FRED_API_KEY` for FRED macro indicators (free — register at https://fred.stlouisfed.org/docs/api/api_key.html)

All new sources are **disabled by default** if their API key env var is empty — graceful degradation.

## Files

| File | Purpose |
|------|---------|
| `config.py` | All tunable parameters — sources, filters, keywords, playbook, sentiment lexicon |
| `macro_news.py` | Main runtime — collectors, pipeline, classifier, API server, dashboard |
| `dashboard.html` | Dark-theme monitoring UI with price tickers, FNG gauge, FRED indicators, signal feed |
| `skill.md` | This file — agent instructions |
| `state/state.json` | Persisted state (signals, dedup hashes, reputation, finnhub_last_id) |

## Configuration

Edit `config.py` to:
- Add Telegram groups/channels in `GROUPS` / `CHANNELS` dicts
- Set Telethon credentials (`TELETHON_API_ID`, `TELETHON_API_HASH`)
- Adjust noise filter thresholds
- Add/modify `MACRO_KEYWORDS` regex patterns for new event types
- Update `MACRO_PLAYBOOK` with direction/magnitude/affects for new events
- Tune sentiment lexicon (`POSITIVE_WORDS`, `NEGATIVE_WORDS`)
- Change `DASHBOARD_PORT` (default: 3252)
- Configure new sources: `OPENNEWS_*`, `FINNHUB_*`, `FRED_*`, `PRICE_TICKER_POLL_SEC`

### New Source Config Summary

| Source | Env Var | Default Poll | Enable Flag | Config Prefix |
|--------|---------|-------------|-------------|---------------|
| 6551.io OpenNews | `OPENNEWS_TOKEN` | WebSocket (realtime) / 120s REST fallback | `OPENNEWS_ENABLED` | `OPENNEWS_*` |
| Finnhub | `FINNHUB_API_KEY` | 180s | `FINNHUB_ENABLED` | `FINNHUB_*` |
| FRED | `FRED_API_KEY` | 3600s | `FRED_ENABLED` | `FRED_*` |
| Price Tickers | `FINNHUB_API_KEY` + CoinGecko (free) | 60s | Always on if Finnhub key present | `PRICE_TICKER_POLL_SEC` |

## Signal Schema

Every signal from all sources follows this schema:

```python
{
    "ts": int,                # Unix timestamp
    "ts_human": str,          # "04-02 14:30:05"
    "source_type": str,       # "newsnow" | "polymarket" | "telegram" | "opennews" | "finnhub" | "fred"
    "source_name": str,       # "wallstreetcn" | "Reuters" | "CNBC" | "fred" | etc.
    "event_type": str,        # "fed_cut_expected" | "whale_buy" | etc.
    "direction": str,         # "bullish" | "bearish" | "neutral"
    "magnitude": float,       # 0.0–1.0
    "urgency": float,         # 0.0–1.0
    "affects": list,          # ["rwa", "perps", "spot_long", "meme"]
    "tokens": list,           # ["ONDO", "PAXG"] extracted tickers
    "sentiment": float,       # -1.0 to +1.0
    "text": str,              # First 400 chars of headline/message
    "insight": str,           # AI-generated 2-3 sentence analysis (requires ANTHROPIC_API_KEY)
    "sender": str,            # Username or source name
    "sender_rep": float,      # Sender reputation at signal time
    "classify_method": str,   # "keyword" | "llm_confirm" | "llm_discover" | "polymarket"
    "group_category": str,    # "macro" | "whale" | "http_news" | "opennews" | "macro_data" | etc.
}
```

## Data Sources Detail

### 6551.io OpenNews (WebSocket + REST fallback)
- Aggregates 84+ sources (Bloomberg, Reuters, FT, CoinDesk, The Block)
- AI scores each article 0-100 with long/short/neutral signal
- WebSocket: subscribes to `news.update` + `news.ai_update`, filters by score >= `OPENNEWS_MIN_SCORE` (40)
- REST fallback: polls `GET /open/free_hot?category=news` every 120s when WS disconnects
- Reconnects with exponential backoff (5s, 10s, 30s, 60s)
- Dedicated thread (`_start_opennews_thread()`) — same pattern as Telethon

### Finnhub Market News (REST)
- Covers general market news + crypto news
- Uses `minId` parameter for incremental fetching (no duplicate articles)
- `_finnhub_last_id` persisted in state.json across restarts
- Categories configurable via `FINNHUB_CATEGORIES` (default: `["general", "crypto"]`)
- Also provides stock/ETF quotes for the price ticker bar (SPY, GLD, SLV)

### FRED Macro Indicators (REST)
- Hard macro data: Fed Funds Rate, CPI, GDP, Unemployment, 10Y-2Y Spread, 10Y Yield
- Does NOT go through `process_signal()` normally — stored as context data like Fear & Greed
- **Significant change detection**: when an indicator moves beyond its threshold, emits a signal via `process_signal()` (e.g., Fed Funds changes >= 10 bps, CPI changes >= 0.3%)
- Thresholds defined in `_FRED_CHANGE_THRESHOLDS`
- Served via `/api/fred` endpoint and displayed in dashboard sidebar

### Price Tickers
- SPY, GLD, SLV: Finnhub `/quote` endpoint (requires `FINNHUB_API_KEY`)
- BTC, ETH: CoinGecko free API (no key needed)
- Refreshes every 60s, displayed in dashboard ticker bar
- Served via `/api/prices` endpoint

### AI Insights (LLM Enrichment)
- When `ANTHROPIC_API_KEY` is set and `LLM_INSIGHT_ENABLED = True`
- Calls Haiku for every classified signal (event_type != "unclassified")
- Generates 2-3 sentence analysis: key takeaway + specific asset impact
- Stored in signal's `insight` field, displayed in dashboard card body
- Config: `LLM_INSIGHT_ENABLED`, `LLM_INSIGHT_TIMEOUT_SEC`, `LLM_INSIGHT_MAX_TOKENS`

## Classification Pipeline (3 Layers)

1. **Layer 1: Keyword regex** — 24+ event types with bilingual patterns (EN/CN). Free, instant.
2. **Layer 2: LLM confirm** — Headlines in ambiguous confidence band (0.55–0.80) go to Haiku for confirmation.
3. **Layer 3: LLM discover** — Relevant messages that missed keywords get LLM classification.

Pre-screen: Only messages containing `LLM_PRESCREEN_KEYWORDS` are sent to LLM (saves cost).

## Event Types

| Category | Event Types |
|----------|-------------|
| Fed/Rates | `fed_cut_expected`, `fed_cut_surprise`, `fed_hold_hawkish`, `fed_hike`, `fed_dovish` |
| CPI | `cpi_hot`, `cpi_cool` |
| Gold | `gold_breakout`, `gold_selloff` |
| Geopolitical | `geopolitical_escalation`, `geopolitical_deesc` |
| Trade/Tariff | `tariff_escalation`, `tariff_relief` |
| RWA | `rwa_catalyst`, `sec_rwa_positive`, `sec_rwa_negative` |
| Whale | `whale_buy`, `whale_sell` |
| Liquidation | `liquidation_cascade` |
| Employment/GDP | `nfp_strong`, `nfp_weak`, `gdp_strong`, `gdp_weak` |

## Public API (port 3252)

| Endpoint | Params | Returns |
|----------|--------|---------|
| `GET /api/state` | — | Full dashboard state (signals, sentiment, polymarket, FNG, FRED, prices) |
| `GET /api/sig