skill-system-insight

What this skill does

# Skill System Insight

Deprecated compatibility surface. `skill-system-soul` is now the canonical owner of observe/evolve behavior, but this skill remains available so existing `insight` routes, docs, and tests do not break during the merge window.

Unified **OBSERVE → EVOLVE** engine for user adaptation.

This skill turns sessions into structured behavioral insights ("facets"), uses those facets to maintain a dual-matrix soul state, and periodically evolves soul profiles and workflow recipes — all versioned and rollbackable.

## Two Phases, One Skill

```
Phase 1 — OBSERVE
  Extract facets → Update matrix buffers → (optional) Trigger synthesis

Phase 2 — EVOLVE
  Read accumulated data → Plan evidence-backed changes → Apply with versioning → Snapshot for audit/rollback
```

Phase 1 produces raw signal. Phase 2 consumes it to make durable changes.

The evolve lifecycle can run through an arena comparison loop:

`session -> facets -> evolve proposals -> arena A/B -> merge suggestion`

The arena stage only emits recommendations. It never auto-switches profiles and never auto-merges changes.

## Architecture: Hybrid 3-Layer System

```
Layer 1: Base Profile (balanced.md)
  - Static skeleton: section format + safety/quality defaults.

Layer 2: Dual Matrix (soul-state)
  - Personality Matrix (slow/stable): openness, directness, autonomy, rigor, warmth
  - Emotion Matrix (faster baseline): patience, enthusiasm, caution, empathy

Layer 3: Synthesized Profile (user.md)
  - Periodically regenerated from Layer 1 + Layer 2 + accumulated facets.
  - Every regeneration creates a versioned snapshot.
```

Layer 2 is the ground truth. Layer 3 is a readable, versioned projection.

## Data Model

- Facets (per-session extraction): `schema/facet.yaml`
- Soul state (dual matrix + counters/buffers): `schema/soul-state.yaml`
- Evolution plan (proposed changes): `schema/evolution-plan.yaml`
- Evolution snapshot (versioned artifact): `schema/evolution-snapshot.yaml`

Storage uses the Postgres `agent_memories` table and typed sub-tables:

- Facets: `memory_type='episodic'`, `category='insight-facet'` → typed table: `insight_facets`
- Soul state: `memory_type='semantic'`, `category='soul-state'` → typed table: `soul_states`
- Evolution snapshots: `memory_type='episodic'`, `category='evolution-snapshot'` → typed table: `evolution_snapshots`

Typed functions (dual-write to both typed table and agent_memories):
- `insert_insight_facet(user, session_id, yaml, ...)` → stores facet
- `upsert_soul_state(user, yaml, personality, emotion, ...)` → stores/updates soul state
- `get_soul_state(user)` → reads from `soul_states`
- `get_recent_facets(user, limit)` → reads from `insight_facets`
- `insert_evolution_snapshot(user, version_tag, target, ...)` → stores versioned snapshot
- `get_evolution_history(user, limit)` → reads from `evolution_snapshots`

## Phase 1: OBSERVE Pipeline

```
Trigger (manual or suggested) → Extract Facet → Update Matrix → (optional) Synthesize Profile
```

References:
- Facet extraction prompt: `prompts/facet-extraction.md`
- Soul synthesis prompt: `prompts/soul-synthesis.md`
- Extraction procedure: `scripts/extract-facets.md`
- Matrix update algorithm: `scripts/update-matrix.md`
- Profile regeneration procedure: `scripts/synthesize-profile.md`

### How To Trigger (OBSERVE)

This is a manual workflow.

- User can ask explicitly: "insight", "extract facets", "learn my preferences", "update my profile".
- Router suggestion pattern (lightweight, non-pushy):
  - "Want me to run an insight pass to learn from this session? (stores a facet + may update your matrix)"

When the user asks (or agrees), run:

1) `scripts/extract-facets.md`
2) `scripts/update-matrix.md`
3) If the synthesis trigger fires: `scripts/synthesize-profile.md`

## Phase 2: EVOLVE Pipeline

```
Trigger (manual or scheduled) → Read facets + matrix → Plan evolution → Safety check → Apply → Snapshot
```

References:
- Evolution planning prompt: `prompts/evolution-planning.md`
- Recipe evolution prompt: `prompts/recipe-evolution.md`
- Clinical review prompt: `prompts/clinical-organism-review.md`
- Soul evolution procedure: `scripts/evolve-soul.md`
- Recipe evolution procedure: `scripts/evolve-recipes.md`
- Version listing: `scripts/list-versions.md`
- Rollback procedure: `scripts/rollback.md`

### How To Trigger (EVOLVE)

- User can ask: "evolve my profile", "evolve recipes", "list versions", "rollback to v2".
- Automatic trigger when synthesis trigger fires during OBSERVE phase.

### Version Tags

Format: `v{N}_{target}_{timestamp}`
Example: `v3_soul_20260211`

Algorithm: `N = 1 + max(existing N)` for the same user+target. Start at N=1 if none exist.

## Clinical Organism Review (Compatibility Lens)

Insight now includes a compatibility procedure `clinical-organism-review` for BH/SK body-map triage.

The procedure emits review rows with required fields:

- `body_part`, `technical_surface`, `tissue_class`
- `maturity` (`mature | prototype | embryonic | waste | retired`)
- `current_pain`, `evidence`
- `disposition` (`keep | grow | merge | retire | archive | excrete | bundle`)
- `owner`, `close_gate`

Hard constraint: **diagnosis is not progress**. Any pain finding without containment + repair + verification must remain `close_gate=pain_mapped`.

Use: `prompts/clinical-organism-review.md`

### Writes

- Evolution snapshots → Postgres (`evolution_snapshots` + `agent_memories`)
- Soul profiles (filesystem) → `../skill-system-soul/profiles/<user>.md`
- Workflow recipes (filesystem) → `../skill-system-workflow/recipes/*.yaml`

## Configuration

All runtime settings (rate limits, thresholds, step sizes, approval gates) are in `config/insight.yaml`.

**Config is the single source of truth.** Values below are documentation defaults — if they differ from config/, config/ wins.

See: `../../config/insight.yaml`

## Constraints (Non-Negotiable)

### Transparency

Always explain what was learned and why.

- Facets must contain evidence strings tied to concrete moments.
- Matrix updates must add short context lines explaining each applied adjustment.
- Every evolution change must include evidence strings from facets or state context.

### Rate Limiting (defaults — see config/insight.yaml)

- OBSERVE: Max `observe.max_facets_per_day` facets per user per rolling 24 hours.
  - If over limit: summarize what you would have captured, ask the user to pick 1 session to record.
- EVOLVE: Max `evolve.max_passes_per_day` evolution pass per day per user (across all targets).
  - Use the most recent `evolution-snapshot` date as the limiter.

### Confidence Threshold (defaults — see config/insight.yaml)

Do not change matrix values on a single observation.

- Threshold: `observe.confidence_threshold`+ similar observations in the same direction.
- Personality step size: +/- `observe.personality_step` per qualifying adjustment.
- Emotion baseline step size: +/- `observe.emotion_step` per qualifying adjustment.

Accumulation tracked in buffers for testability and explainability.

### Safety (Evolution)

- Cannot remove core safety constraints from soul profiles.
- Evolved profiles must retain: honest uncertainty, no hallucinated authority, refusal of harmful/illegal instructions.
- Major changes (any dimension change > `evolve.approval_drift_threshold`) require explicit user approval before applying.

## Where Layer 1 and Layer 3 Live

- Base profile (Layer 1): `../skill-system-soul/profiles/balanced.md`
- Synthesized user profile (Layer 3): `../skill-system-soul/profiles/<user>.md`

The synthesis step preserves the 6-section format:
1. Identity
2. Decision Heuristics
3. Communication Style
4. Quality Bar
5. Tool Preferences
6. Anti-Patterns

## Storage Pattern (agent_memories)

Example SQL templates:

```sql
-- Store a facet
SELECT store_memory(
  'episodic', 'insight-facet',
  ARRAY['session:ses_xxx', 'user:arthu'],
  'Session Facet: <brief_summary>',
  '<full facet YAML as text>',
  '{"session_type": "...", "outcome": "..."}',