agent-artifex:assess

What this skill does

# agent-artifex:assess — AI Services Design & Testing Gap Assessment

## When to Use

Diagnostic skill that identifies design quality issues and testing gaps in an existing project. Produces a prioritized gap analysis with specific recommendations covering both how well the AI integration is designed and how well it is tested. For learning the concepts first, use `agent-artifex:foundations` or `agent-artifex:learn`. For applying design principles, use `agent-artifex:design`. For writing tests or implementing design fixes after assessment, use `agent-artifex:implement`.

## Shared References

Read these when you need precise definitions during assessment:

| Reference | When to read |
|---|---|
| `agent-artifex/references/framework.md` | Causal chain diagnostic flow, testing pyramid layers, two-tier model, impact/effort table |
| `agent-artifex/references/metrics.md` | Formulas for any metric (SR/AE/AS, Faithfulness/Completeness, CRR/WCR/DASR) |
| `agent-artifex/references/rubric.md` | Six-component rubric, labeling rules, structural detection markers |
| `agent-artifex/references/evidence.md` | Key numbers to cite when justifying recommendations |

---

## Assessment Process

Three passes: inventory, gap analysis, and prioritized recommendations.

---

## Pass 1: Inventory

### Questions to ask

1. **What are you building?**
   - MCP server (how many tools?)
   - Agent/chatbot that uses MCP servers
   - Both (server + client)
   - Something else that uses LLM tool calling

2. **What tests exist today?**
   - Unit tests for server logic?
   - Integration tests against the MCP protocol?
   - Structural checks on tool descriptions?
   - Tests that involve an LLM?
   - End-to-end tests with seeded data and ground truth?
   - Multi-turn conversation tests?
   - Recorded replay infrastructure?

3. **What problems are you experiencing?**
   - FM picks the wrong tool
   - FM uses wrong arguments
   - Server returns errors the FM can't recover from
   - Final answers are inaccurate or hallucinated
   - Quality degrades in long conversations
   - Tests are flaky or expensive
   - "We don't know what we don't know"

4. **Have you followed design guidelines for your tool descriptions, error messages, and schemas?**
   - Are tool descriptions substantive (4+ sentences with usage guidelines and limitations)?
   - Do parameters have `.describe()` annotations beyond property names?
   - Do error messages suggest recovery actions?
   - Are output schemas declared?
   - Is the system prompt sized appropriately relative to tool count?

### What to look for in the codebase

If you have access to the codebase, also check:

- **Tool descriptions:** Read the tool registration code or `tools/list` output. Note description length (< 3 sentences = likely smells), whether parameters have descriptions, whether tools cross-reference each other, whether limitations are concrete or vague.
- **Output schemas:** Do tools declare `outputSchema`? Without it, clients and LLMs can't validate returned data.
- **Error handling:** Do error responses contain stack traces (`/Error\s+at\s/`)? Are error messages > 20 characters? Do they suggest recovery actions?
- **Test files:** Look for test directories and frameworks. Note what's being asserted — exact values? Schema shapes? LLM output? Are tests isolated (clean state per run)?
- **CI configuration:** What runs on every commit vs. on-demand? Are LLM-calling tests in CI (anti-pattern)?
- **System prompt size:** How large is the system prompt relative to tool definition footprint? With many tools, definitions alone can consume 50K+ tokens.
- **Tool count and discovery:** How many tools are registered? Is dynamic discovery used, or are all tools loaded at once?
- **Response format consistency:** Do tools use consistent field names and structures across responses? Are the same concepts named differently in different tools?
- **Result verbosity:** Are tool results sized for context budget, or do they return full data dumps that will fill the context window in multi-turn conversations?

---

## Pass 2: Gap Analysis

### Design Quality Assessment

Assess the quality of the AI integration design across seven areas. For each, assign a status:

| Status | Meaning |
|---|---|
| **Not designed** | No intentional design effort in this area |
| **Partially designed** | Some attention paid but significant gaps remain |
| **Well-designed** | Follows established design principles; low risk of FM-facing issues |

#### Tool Description Design

| Signal | Indicates |
|---|---|
| Descriptions < 4 sentences | Missing substantive content — likely no Usage Guidelines or Limitations |
| No cross-references between similar tools | Missing inter-tool disambiguation |
| Limitations are vague ("may not work in all cases") | Worse than no limitations — degrades SR by up to 10pp |

#### Parameter & Schema Design

| Signal | Indicates |
|---|---|
| No `.describe()` annotations beyond property name | Opaque Parameters (84.3% prevalence) |
| No output schemas declared | MCP spec violation; clients can't validate results |
| Tools with > 20 parameters | Out of model distribution — expect degradation |

#### Error Message Design

| Signal | Indicates |
|---|---|
| Stack traces in error output | Internal details leak to FM — anti-pattern |
| Error messages < 20 characters | Too terse for FM to act on |
| No recovery action in errors | FM can't determine what to try next |

#### System Prompt Design

| Signal | Indicates |
|---|---|
| Domain knowledge duplicated in prompt and tool descriptions | Capability overlap — FM may answer from prompt instead of calling tool |
| System prompt > 2000 tokens with > 10 tools | Context pressure risk — tool definitions may already consume 50K+ tokens |

#### Multi-Turn Conversation Design

| Signal | Indicates |
|---|---|
| No testing at conversation depths > 5 turns | Context pressure effects unknown |
| Tool results include full data dumps | Verbose results accelerate context pressure |
| Entity references use display names, not IDs | Coreference resolution will fail |

#### Tool Set Architecture

| Signal | Indicates |
|---|---|
| > 20 tools without dynamic discovery | Context overload — 85% token reduction possible with dynamic loading |
| Overlapping tools don't reference each other | FM will confuse similar tools |
| Multi-intent tools with mode parameters | Should be separate tools |

#### Response Format Design

| Signal | Indicates |
|---|---|
| Different field names for same concept across tools | Inconsistent schemas degrade FM parsing |
| No pagination for large result sets | Context will fill up in multi-turn conversations |
| Results contain narrative instead of discrete facts | Can't run claim decomposition for verification |

### Testing Gap Analysis

Map current testing against the five areas. For each, assign a status:

| Status | Meaning |
|---|---|
| **Not tested** | No tests exist for this area |
| **Partially tested** | Some tests exist but significant gaps remain |
| **Adequately tested** | Core scenarios covered; diminishing returns from more investment |
| **Over-invested** | More testing effort than impact justifies (e.g., LLM tests in CI) |

#### Tool Description Quality (Discovery)

| Signal | Indicates |
|---|---|
| Descriptions < 3 sentences | Rubric score < 3 on Length; likely missing Usage Guidelines, Limitations, Parameters |
| No parameter `.describe()` annotations | Opaque Parameters smell (84.3% prevalence). FM defaults to overly broad argument values. |
| Tools with overlapping names don't reference each other | Missing inter-tool disambiguation. FM will confuse similar tools. |
| No structural checks in CI | Description changes ship without quality verification. |
| Limitations are vague or self-referential | Worse than no Limitations — empirically degrades SR by up to 10pp. |
| No FM-scored rubric evaluation | Semantic quality is unmeasured; structural checks are necessary but insufficient. |

#### Ag