momentic-result-classification

What this skill does

# Momentic result classification (MCP)

Momentic is an end-to-end testing framework where each test is composed of browser interaction steps. Each step combines Momentic-specific behavior (AI checks, natural-language locators, ai actions, etc.) with Playwright capabilities wrapped in our YAML step schema. When these tests are run, they produce results data that can be used to analyze the outcome of the test. The results data contains metadata about the run as well as any assets generated by the run (e.g. screenshots, logs, network requests, video recordings, etc.). Your job is to use these test results to classify failures that occurred in Momentic test runs.

## Instructions

1. Given a failing test run, identify the earliest point where the current run entered a bad state. Do not stop at the final failing assertion or missing locator target.
2. Explain the root cause at action/state level: what step tried to do, what specific element or state it relied on, what actually happened, and what evidence proves it.
3. Bucket the failure into one of the below categories, explaining the reasoning for choosing the specific category.

## Helpful MCP tools

`momentic_get_run` — Returns some metadata about the run and a summary of the full run results. Use the metadata to help you parse through the run results (e.g. which attempt to look at, which step failed, etc.). If the current run details were already supplied in the initial context, do not call this again for that same run unless you explicitly need a different attempt.

`momentic_list_runs` — Recent runs for a test so you can compare the result of past runs over time. **Always pass `gitBranchName` when it exists on the run in question** so that it's more likely you're looking at the same version of the test. Pass `recovered=true` when you want to inspect recovered runs.

`momentic_get_step_result` — Returns the result of a specific step, with other information such as full step trace and before/after screenshots. Use `parentStepIdChain` for steps nested inside other steps. Only request `includeTrace=true` when you need it, because it can be very large.

`momentic_get_test_steps_for_run` — Returns the simplified test steps recorded on a run (`stepsSnapshot`, `beforeStepsSnapshot`, `afterStepsSnapshot`). You can use this to understand the intent of the test if you need more information than what you can glean from the test name and description.

## Investigation workflow

Start with the current run before relying on history.

1. Call `momentic_get_run` and identify the failing attempt, section (`beforeSteps`, main steps, or `afterSteps`), failing step, and any `parentStepIdChain`.
2. Pull the failing step result with screenshots and trace. If the step is nested, also pull the nearest parent container or module result.
3. Decide whether the failing step's before-screenshot is the correct baseline for that action. If it is already wrong, walk backward through the current run until you find the step/container that produced that bad state.
4. For repeated modules or repeated workflows, compare invocations inside the same current run before comparing older runs. The later failure is often caused by an earlier invocation that succeeded, recovered, or left an invalid postcondition.
5. Treat successful containers with failed or recovered child steps as partial failures until you inspect the container's final after-screenshot and URL.
6. Use past runs only for specific comparison questions once the current-run behavior is understood.

Before classifying, be able to answer:

- What is the test's intended behavior?
- What is the earliest divergent step/container?
- What did that step intend to do?
- Which element/state did it actually interact with or observe?
- What changed in the screenshot, URL, DOM, trace, or recovery log after the step?
- Why is the later failure a consequence of that earlier divergence?

Avoid vague root causes such as "setup was unreliable" or "the page was in the wrong state." Name the broken postcondition directly: for example, "the row-level plus button was clicked, but the app stayed on the parent page instead of opening the child-page editor; the following global `Add to` assertion passed against unrelated page text, so the untargeted type step never entered the child title."

## Evidence standards

- Screenshots are the default truth source for page state. Use trace fields and DOM/HTML to explain why the screenshot changed or did not change.
- Verify every causal claim. Do not say an overlay, side peek, modal, or menu was present unless the relevant before/after screenshot, URL, or DOM proves it.
- Separate "the target is missing now" from "the browser is in the state where that target should exist." A missing target is often a symptom of an earlier failed action.
- For click/type/action steps, record the intended action, actual interacted element when available, before/after URL, and whether the expected UI state appeared.
- For assertions, check whether the assertion is scoped enough to prove the intended state. A broad page-content assertion can pass for unrelated text.
- For recovery, inspect both the failed child step and the recovered container final state. Recovery can pass a retried assertion while leaving state that later steps did not expect.

## Background

### Test run result structure

When momentic tests are run via the CLI, the results are stored in a "run group". The data for this run group is stored in a single directory within the momentic project. By default, the directory is called `test-results`, but can be changed in momentic project settings or on a single run of a run group. The run group results folder has the following structure:

```
test-results/
├── metadata.json         data about the run group, including git metadata and timing info.
└── runs/                 On zip for each test run in the run group.
    ├── <runId_1>.zip         a zipped run directory containing data about this specific test run.  Follows the structure described below.
    └── <runId_2>.zip
```

When unzipped, run directories have the following structure:

```
<runId>/
├── metadata.json           run-level metadata.
└── attempts/<n>/           one folder per attempt (1-based n).
    ├── metadata.json       attempt outcome and step results.
    ├── console.json        optional browser console output.
    └── assets/
        ├── <snapshotId>.jpeg     before/after screenshot for each step (see attempt metadata.json for snapshot ID).
        ├── <snapshotId>.html     before/after DOM snapshot for each step (see attempt metadata.json for snapshot ID).
        ├── har-pages.log         HAR pages (ndjson).
        ├── har-entries.log       HAR network entries (ndjson).
        ├── resource-usage.ndjson CPU/memory samples taken during the attempt.
        ├── <videoName>           video recording (when video recording is enabled).
        └── browser-crash.zip     browser crash dump (only present on crash).
```

When getting run results via the momentic MCP, tools such as `momentic_get_run` will return links to the MCP working directory (default `.momentic-mcp`). This directory will contain unzipped run result folders, following the structure above, named `run-result-<runId>`.

### Element locators

Certain step types that interact with elements have a "target" property, or **locator**, that specifies which element the step should interact with.

#### Locator caches

Locators identify elements by sending the page state html/xml to an llm as well as a screenshot. The llm identifies which element on the page the user is referring to. Momentic will attempt to "cache" the answer from the llm so that future runs don't require AI calls. On future runs, the page state is checked against the cached element to determine whether the element is still usable, or the page has changed enough such that another AI call is required.

A locator cache can bust for a variety of reasons:

- the element description