arize-compliance-audit

What this skill does

# Arize Compliance Audit Skill

Use this skill when the user wants to **audit their AI agent or LLM application for regulatory compliance**. The skill scans the codebase for compliance gaps, cross-references Arize instrumentation for audit trail coverage, and produces a tailored checklist with optional remediation.

**Triggers:** "audit my app for compliance", "EU AI Act requirements", "NIST AI RMF checklist", "GDPR for AI", "is my AI app compliant", "compliance checklist", "regulatory audit", "ISO 42001", "AI management system", "AIMS certification".

## Disclaimer

**Before doing anything else, present this disclaimer verbatim to the user:**

---

> ⚠️ **Legal disclaimer**
>
> This audit is for **guidance only** and does **not** constitute legal advice or a complete compliance assessment. It identifies common technical patterns and gaps based on publicly available regulatory frameworks, but cannot assess your organisation's specific legal obligations, contractual commitments, data processing agreements, or operational processes.
>
> **Do not rely on this output as a substitute for qualified legal counsel.** Regulatory compliance is a complex, jurisdiction-specific, and fact-dependent determination. Always engage a qualified attorney or compliance specialist for binding assessments.

---

## Core principles

- **Prefer inspection over mutation** — understand the codebase before suggesting changes.
- **Be practical, not legal** — produce developer-actionable items, not legal opinions.
- **Tailor to jurisdiction and use case** — a chatbot has different obligations than a hiring tool. Do not dump the entire regulatory framework.
- **Cross-reference instrumentation** — compliance requires audit trails; check whether Arize tracing captures what regulators expect.
- **Offer remediation, always confirm** — after presenting the checklist, offer to implement specific fixes, but never modify code without explicit user confirmation.
- **Keep output concise and production-focused** — do not generate extra documentation or summary files unless requested.
- **Never embed literal credential values** — always reference environment variables.

## Phase 0: Framework selection and use case

Before scanning code, determine which compliance frameworks apply.

### Step 1 — Framework selection

Use the `AskUserQuestion` tool to ask the user which frameworks apply. **Do not infer or auto-select** — always ask explicitly.

Ask:

```
Which compliance frameworks should this audit cover?
Select all that apply (reply with numbers, e.g. "1, 3"):

1. EU frameworks — EU AI Act, GPAI Code of Practice, GDPR
   (choose if end-users or data subjects are located in the EU)

2. US frameworks — NIST AI RMF, state laws (Colorado AI Act, NYC LL144),
   HIPAA (if processing health data)
   (choose if operating in the United States)

3. ISO 42001 — International AI Management System standard
   (choose if pursuing ISO 42001 certification, operating globally,
   or wanting an internationally recognised baseline)

You can select any combination. If unsure, select all that seem relevant
and we can narrow down during the audit.
```

Based on the selection:
- **1 selected** — EU AI Act, GPAI Code of Practice, GDPR apply. See references/eu-ai-act-gpai.md.
- **2 selected** — NIST AI RMF, Colorado AI Act, NYC LL144, HIPAA may apply. See references/us-ai-compliance.md.
- **3 selected** — ISO 42001 AIMS controls apply. See references/iso-42001.md. Note: ISO 42001 is an organisational management system — the audit will cover technically-auditable controls only; purely organisational clauses (leadership review, internal audits) are flagged separately.
- **Multiple selected** — all selected frameworks apply; the audit covers the union of requirements, with cross-references where frameworks overlap.

### Step 2 — Determine use case category

Use the `AskUserQuestion` tool to ask: **What does your AI application do?**

- **General chatbot / assistant** — Limited risk (EU), general obligations (US)
- **Hiring / HR** — High risk (EU Art. 6, Annex III); Colorado AI Act applies; NYC LL144 applies if NYC
- **Healthcare** — High risk (EU); HIPAA applies if processing PHI
- **Credit / financial** — High risk (EU); Colorado AI Act applies
- **Education** — High risk (EU)
- **Content generation** — Limited risk (EU Art. 50 transparency); general obligations (US)
- **GPAI model provider** — GPAI Code of Practice applies (EU)

### Step 3 — Determine risk tier

Based on the use case and selected frameworks:
- **EU selected**: Classify as Unacceptable / High / Limited / Minimal per references/eu-ai-act-gpai.md
- **US selected**: Classify as High-risk (consequential decisions per Colorado AI Act) or General
- **ISO 42001 selected**: Risk tier is not a formal classification in ISO 42001, but note whether the system is high-stakes (which elevates the priority of impact assessment and bias controls)

### Phase 0 output

Present a brief summary:

```
Frameworks selected: {EU / US / ISO 42001 / combination}
Use case:            {category}
Risk tier:           {EU tier if applicable} / {US tier if applicable}
Applicable:          {list of specific regulations and standards}
ISO 42001 note:      {if selected} Audit covers technically-auditable controls only;
                     organisational clauses will be flagged but not code-audited.
```

Then proceed directly to Phase 1.

## Phase 1: Codebase audit (read-only)

**Do not write any code or create any files during this phase.**

Systematically scan the codebase for evidence of compliance and gaps across seven domains. For each domain, run the listed searches and record findings.

### A. Transparency and disclosure

**What to look for:**
- User-facing strings disclosing AI involvement: search for terms like `AI`, `artificial intelligence`, `automated`, `bot`, `machine learning`, `generated by`, `powered by` in UI templates, API responses, and user-facing code
- Content labelling: markers on AI-generated output (text, images, audio)
- Terms of service, privacy policy references in the codebase

**Signals of concern:** Absence of any AI disclosure in user-facing code, especially if the application generates content or makes recommendations.

### B. Data protection and privacy

**What to look for:**
- PII field names in code: `email`, `phone`, `ssn`, `social_security`, `date_of_birth`, `address`, `name` in prompts, context, or retrieved documents
- PII in trace span attributes: check if `input.value` or `output.value` could contain personal data sent to Arize without redaction
- Consent mechanisms: `consent`, `opt-in`, `opt-out`, `gdpr`, `ccpa` references
- DPIA or privacy assessment references
- Data retention and deletion handlers
- Data subject rights: `right_to_access`, `right_to_erasure`, `data_subject_request`, `data_protection_officer`

### C. Security

**What to look for:**
- Prompt injection defences: input validation, guardrail libraries (`guardrails-ai`, `nemo-guardrails`, `rebuff`, `lakera`), content filtering, system prompt protection
- Data loss prevention: output scanning before returning to users, sensitive data detection
- Tool/function calling controls: permission boundaries, allowlists, sandboxing for tool execution
- Rate limiting and authentication on AI endpoints
- Hardcoded secrets: `api_key`, `secret`, `password`, `token` literals in source files (not env var references)

### D. Testing and evaluation

**What to look for:**
- Bias and fairness testing: references to demographic parity, impact ratios, fairness metrics
- Red teaming or adversarial test suites: prompt injection tests, jailbreak tests
- Evaluation frameworks: Arize evaluators, custom eval scripts, `pytest`-based evals, experiment infrastructure
- A/B testing or model comparison infrastructure

### E. Documentation

**What to look for:**
- Model cards: `MODEL_CARD.md`, `model_card.json`, `model_card.yaml`, or similar
- System architecture documentation
- Change logs or version tracking for