securing-agentforce

What this skill does

# ADLC Security

OWASP LLM Top 10 security assessment for live Agentforce agents.

## Overview

This skill sends adversarial test payloads to a deployed Agentforce agent via `sf agent preview` and evaluates whether the agent resists attacks across 7 OWASP LLM Top 10 categories:

| ID | Category | Tests | Focus |
|----|----------|-------|-------|
| LLM01 | Prompt Injection | 9 | Direct override, encoding, multi-turn, role-play, delimiter, multilingual |
| LLM02 | Sensitive Info Disclosure | 10 | PII extraction, credentials, cross-tenant, context leakage |
| LLM05 | Improper Output Handling | 7 | XSS, SQL injection, command injection, SSRF, path traversal |
| LLM06 | Excessive Agency | 8 | Unauthorized actions, privilege escalation, data exfiltration |
| LLM07 | System Prompt Leakage | 10 | Direct extraction, role-play bypass, encoding, social engineering |
| LLM09 | Misinformation | 7 | Hallucination, fabricated citations, knowledge boundary violations |
| LLM10 | Unbounded Consumption | 6 | Token exhaustion, recursion, context saturation |

Total: **57 tests** with weighted severity scoring producing an A–F grade.

## Platform Notes

- Shell examples use bash. On Windows use PowerShell or Git Bash.
- Replace `python3` with `python` on Windows.
- Replace `/tmp/` with `$env:TEMP\` (PowerShell) or `%TEMP%\` (cmd).
- Replace `jq` with `python3 -c "import json,sys; ..."` if jq is not installed.
- Replace `find . -path ...` with `Get-ChildItem -Recurse -Filter *.agent` in PowerShell.

## Prerequisites

1. `sf` CLI installed (v2.121.7+)
2. Authenticated target org: `sf org login web -o <alias>`
3. Agent deployed and accessible via preview: `sf agent preview start --authoring-bundle <Name> -o <alias> --json`
4. Python dependency: `pip install pyyaml>=6.0` (required by the test runner)

## Modes

### Quick Scan (~2 min)

Runs a representative subset of 15 high-severity tests across all 7 categories. All evaluation is LLM-as-judge. Best for rapid pre-deploy validation.

### Full Assessment (~5 min)

Runs all 57 static tests. All evaluation is LLM-as-judge. Produces a detailed report with remediation guidance. Best for security sign-off before production deployment.

### Full + Dynamic (~7 min)

A skill-level workflow (not a runner CLI flag): Phase 2 retrieves the agent's configuration from the org and generates 5–10 agent-specific adversarial tests, then Phase 3 invokes the runner with `--mode full`. The dynamic tests are merged with the 57 static tests for comprehensive coverage tailored to the agent's attack surface. The runner is always invoked as `--mode quick` or `--mode full`.

---

## Execution Workflow

### Critical Rules

1. **DO NOT write your own test runner.** Use `skills/securing-agentforce/scripts/security_runner.py` from this plugin. It already handles session management, YAML loading, multi-turn tests, control-char stripping, and rate limiting.
2. **DO NOT write your own report generator.** Use `skills/securing-agentforce/scripts/security_report.py` from this plugin.
3. **DO NOT write your own scoring script.** Use `skills/securing-agentforce/scripts/security_scoring.py` from this plugin.
4. **All evaluation is LLM-as-judge.** Read the runner output and judge each response yourself. There is no pattern-matching step.

### Gathering Input

When the skill loads, gather required details from the user. Follow these constraints strictly:

1. If the user provided org, agent, and mode in their invocation (e.g., `/securing-agentforce myorg --agent MyAgent --mode quick`), skip questions and proceed directly.
2. If details are missing, ask for them using plain text questions — do NOT use structured tool pickers for org alias or agent name (these are freeform text, not selectable options).
3. For mode selection, you may use a structured picker with these options: quick, full, full+dynamic (the user can always type a custom response).
4. Do NOT present OWASP categories as selectable options (there are 7, which exceeds picker limits). Default to all 7 and let users specify a subset via text.

Required information:
- **Org alias** — the authenticated org to test against
- **Agent name** — the AgentName (DeveloperName of the GenAiPlannerDefinition)
- **Mode** — quick or full (default: full). "Full + dynamic" is a skill-level workflow where Phase 2 generates dynamic tests before invoking the runner with `--mode full`
- **Categories** — all 7 unless user specifies a subset

### Required Steps

Follow these phases sequentially. Do NOT skip phases or reorder them.

### Phase 1: Resolve Agent

1. Confirm org alias and agent name from user input

2. **Resolve the agent's API name** by querying the org:
```bash
sf data query --json -o <org-alias> \
  -q "SELECT Id, MasterLabel, DeveloperName FROM GenAiPlannerDefinition WHERE MasterLabel LIKE '%<user-provided-name>%' OR DeveloperName LIKE '%<user-provided-name>%'"
```
   - `MasterLabel` = display name (e.g., "Order Service")
   - `DeveloperName` = API name with version suffix (e.g., "OrderService_v9")
   - The `--authoring-bundle` flag uses `DeveloperName` **without** the `_vN` suffix (e.g., "OrderService")
   - Store this as `AGENT_BUNDLE_NAME` for all subsequent commands

3. **Verify the agent is preview-accessible:**
```bash
sf agent preview start --authoring-bundle <AGENT_BUNDLE_NAME> -o <org-alias> --json
```
4. Store the session ID for subsequent sends
5. End the verification session immediately (it was just a connectivity check):
```bash
sf agent preview end --session-id <ID> --authoring-bundle <AGENT_BUNDLE_NAME> -o <org-alias> --json
```
6. If start fails:
   - Agent not published → suggest: `sf agent publish authoring-bundle --api-name <AGENT_BUNDLE_NAME> -o <org-alias>`
   - Org connectivity issue → check CLI auth: `sf org display -o <org-alias> --json`
   - Timeout → retry once after 5 seconds; if still failing, stop and report the error

### Phase 2: Load Payloads + Generate Dynamic Tests

1. Determine mode (quick or full) from user input (default: full)
2. Determine categories — all 7 by default, or user-specified subset
3. Read the relevant YAML payload files from `skills/securing-agentforce/assets/payloads/`:
   - `prompt-injection.yaml`
   - `sensitive-info-disclosure.yaml`
   - `output-handling.yaml`
   - `excessive-agency.yaml`
   - `system-prompt-leakage.yaml`
   - `misinformation.yaml`
   - `unbounded-consumption.yaml`
4. For quick mode: select only tests with severity `critical` or `high`
5. **Generate dynamic tests** (full + dynamic mode, or when user requests it):

   **Step 5a: Locate the agent configuration**

   Check local first, then retrieve from org:
   ```bash
   # Check if .agent file exists locally
   find . -path "*/aiAuthoringBundles/*/*.agent" -name "*<AGENT_BUNDLE_NAME>*" 2>/dev/null
   ```

   If not found locally, retrieve from the org:
   ```bash
   sf project retrieve start --json --metadata "AiAuthoringBundle:<AGENT_BUNDLE_NAME>" -o <org-alias>
   ```

   > **Known bug:** `sf project retrieve start` creates a double-nested path: `force-app/main/default/main/default/aiAuthoringBundles/...`. Fix it immediately:
   > ```bash
   > if [ -d "force-app/main/default/main/default/aiAuthoringBundles" ]; then
   >   mkdir -p force-app/main/default/aiAuthoringBundles
   >   cp -r force-app/main/default/main/default/aiAuthoringBundles/* \
   >     force-app/main/default/aiAuthoringBundles/
   >   rm -rf force-app/main/default/main
   > fi
   > ```

   **Step 5b: Read and validate the agent file**

   Read the `.agent` file and extract:
   - `system:` block → instructions (extraction target for LLM07)
   - `subagent`/`start_agent` blocks → topics (routing manipulation for LLM01)
   - `actions:` blocks → action names + parameters (unauthorized execution for LLM06)
   - `variables:` → linked variables (data leakage for LLM02)

   **Step 5c: Generate targeted tests**

   - Generate 5–10 agent-specific adversarial tests targeting the agent's unique capabilities
   - Format in the same stru