verify-archive

What this skill does

# verify-archive

Detect bit rot, tampering, and transfer errors in media archives through cryptographic checksum verification. Maintain provenance records and fixity information for long-term preservation.

## Purpose

Media archives degrade over time through:
- **Bit rot**: Silent data corruption in storage media
- **Transfer errors**: Corruption during network transfer or backup
- **Tampering**: Unauthorized modifications
- **Media failure**: Gradual deterioration of physical storage

This command generates self-verifying checksum manifests, performs integrity verification, and optionally creates W3C PROV provenance records with PREMIS fixity metadata.

## Parameters

| Parameter | Required | Description |
|-----------|----------|-------------|
| `archive_path` | Yes | Path to media archive directory |
| `--generate` | No | Generate new CHECKSUMS.sha256 manifest |
| `--verify` | No | Verify existing manifest against files |
| `--provenance` | No | Generate PROVENANCE.jsonld with PREMIS fixity |
| `--fix` | No | Regenerate manifest after archive changes |

**Mutually exclusive modes**: Use `--generate` for initial setup, `--verify` for routine checks, `--fix` after making changes.

## CHECKSUMS.sha256 Format

Self-verifying manifest with cryptographic integrity protection:

```
# MANIFEST_HASH: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
# Generated: 2026-02-14T18:45:22.387654321Z
# Verify with: tail -n +4 CHECKSUMS.sha256 | sha256sum
1a2b3c4d...  ./audio/episode-001.opus
5e6f7g8h...  ./audio/episode-002.opus
9i0j1k2l...  ./video/recording-2026-02-14.mp4
3m4n5o6p...  ./playlists/favorites.m3u
7q8r9s0t...  ./README.md
```

**Format specification**:
- **Line 1**: `# MANIFEST_HASH: <sha256>` - SHA-256 hash of manifest content (lines 4+)
- **Line 2**: `# Generated: <timestamp>` - ISO 8601 UTC timestamp with nanosecond precision
- **Line 3**: `# Verify with: tail -n +4 CHECKSUMS.sha256 | sha256sum` - Self-verification command
- **Lines 4+**: `<hash>  <path>` - File checksums in sha256sum format

**Self-verification property**: Modifying any hash entry invalidates the manifest hash. The manifest is tamper-evident.

**Coverage**: ALL files in the archive are checksummed:
- Audio files (opus, mp3, flac, m4a, aac)
- Video files (mp4, mkv, webm, avi)
- Images (jpg, png, webp, svg)
- Text files (md, txt, srt, vtt)
- Playlists (m3u, m3u8, pls)
- Scripts and metadata (json, yaml, sh)

**Exclusions**: Only `CHECKSUMS.sha256` itself is excluded to prevent circular dependency.

## Verification Procedures

### Quick Verification (Manifest Integrity)

Verify manifest has not been tampered with (sub-second):

```bash
cd /path/to/archive

# Extract manifest hash from header
EXPECTED=$(grep '^# MANIFEST_HASH:' CHECKSUMS.sha256 | awk '{print $3}')

# Compute hash of manifest content (lines 4+)
ACTUAL=$(tail -n +4 CHECKSUMS.sha256 | sha256sum | awk '{print $1}')

# Compare
if [ "$EXPECTED" = "$ACTUAL" ]; then
  echo "✓ Manifest integrity verified"
else
  echo "✗ Manifest has been tampered with"
  exit 1
fi
```

**Use case**: Daily automated checks to detect manifest corruption without verifying all files.

### Full Verification (All Files)

Verify all files match their checksums (minutes to hours depending on archive size):

```bash
cd /path/to/archive

# First verify manifest integrity
EXPECTED=$(grep '^# MANIFEST_HASH:' CHECKSUMS.sha256 | awk '{print $3}')
ACTUAL=$(tail -n +4 CHECKSUMS.sha256 | sha256sum | awk '{print $1}')

if [ "$EXPECTED" != "$ACTUAL" ]; then
  echo "✗ Manifest integrity check failed - stopping"
  exit 1
fi

echo "✓ Manifest integrity verified"

# Then verify all files
tail -n +4 CHECKSUMS.sha256 | sha256sum -c
```

**Use case**: Weekly/monthly deep verification to detect bit rot.

### Quiet Mode (Failures Only)

Show only files that failed verification:

```bash
cd /path/to/archive
tail -n +4 CHECKSUMS.sha256 | sha256sum -c --quiet
```

**Exit codes**:
- `0` - All files verified successfully
- `1` - One or more files failed verification

**Use case**: Cron jobs and automated monitoring.

## Timestamp Standard

All timestamps use ISO 8601 UTC with nanosecond precision:

**Format**: `YYYY-MM-DDTHH:MM:SS.NNNNNNNNNZ`

**Example**: `2026-02-14T18:45:22.387654321Z`

**Generation command**:
```bash
date -u +%Y-%m-%dT%H:%M:%S.%NZ
```

**Requirements**:
- Always UTC (trailing `Z`), never local timezone
- Nanosecond precision (9 decimal places)
- ISO 8601 compliant
- Monotonic (later timestamps are lexicographically greater)

## Workflow

### Initial Setup (--generate)

```bash
aiwg verify-archive /path/to/archive --generate
```

**Steps**:
1. Find all files recursively (excluding `CHECKSUMS.sha256`)
2. Compute SHA-256 hash for each file
3. Sort results by path (deterministic order)
4. Generate timestamp in ISO 8601 UTC format
5. Write checksums to temporary file
6. Compute SHA-256 of checksum content
7. Add self-verifying header with manifest hash
8. Write final `CHECKSUMS.sha256`
9. Generate `VERIFY.md` with human-readable instructions

**Bash implementation**:
```bash
ARCHIVE_PATH="/path/to/archive"
TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%S.%NZ)

cd "$ARCHIVE_PATH"

# Generate checksums (exclude manifest itself)
find . -type f ! -name "CHECKSUMS.sha256" -print0 | \
  sort -z | \
  xargs -0 sha256sum > /tmp/checksums.tmp

# Compute manifest hash
MANIFEST_HASH=$(sha256sum /tmp/checksums.tmp | awk '{print $1}')

# Write final manifest with self-verifying header
{
  echo "# MANIFEST_HASH: $MANIFEST_HASH"
  echo "# Generated: $TIMESTAMP"
  echo "# Verify with: tail -n +4 CHECKSUMS.sha256 | sha256sum"
  cat /tmp/checksums.tmp
} > CHECKSUMS.sha256

rm /tmp/checksums.tmp

echo "✓ Generated CHECKSUMS.sha256 with $MANIFEST_HASH"
```

### Routine Verification (--verify)

```bash
aiwg verify-archive /path/to/archive --verify
```

**Steps**:
1. Read `MANIFEST_HASH` from header
2. Compute hash of manifest content (lines 4+)
3. Compare hashes (quick integrity check)
4. If manifest is intact, verify all files against checksums
5. Report any mismatches or missing files

### Regeneration After Changes (--fix)

```bash
aiwg verify-archive /path/to/archive --fix
```

**Use case**: After adding/removing/modifying files in the archive.

**Steps**:
1. Backup existing `CHECKSUMS.sha256` to `CHECKSUMS.sha256.bak`
2. Regenerate manifest (same as `--generate`)
3. Report changes: added files, removed files, modified files
4. Keep backup for comparison

**Example output**:
```
Backed up existing manifest to CHECKSUMS.sha256.bak
Regenerating checksums...
Changes detected:
  Added: ./audio/episode-003.opus
  Modified: ./README.md (hash changed)
  Removed: ./audio/episode-001-draft.opus
✓ Generated new CHECKSUMS.sha256
```

## VERIFY.md Template

Human-readable verification instructions placed in archive root:

```markdown
# Archive Integrity Verification

This archive contains a self-verifying checksum manifest for detecting corruption, tampering, or transfer errors.

## Quick Verification (Manifest Integrity)

Verify the manifest has not been tampered with (sub-second):

\`\`\`bash
cd "$(dirname "$0")"
EXPECTED=$(grep '^# MANIFEST_HASH:' CHECKSUMS.sha256 | awk '{print $3}')
ACTUAL=$(tail -n +4 CHECKSUMS.sha256 | sha256sum | awk '{print $1}')
[ "$EXPECTED" = "$ACTUAL" ] && echo "✓ Manifest integrity verified" || echo "✗ Manifest tampered"
\`\`\`

## Full Verification (All Files)

Verify all files match their checksums:

\`\`\`bash
cd "$(dirname "$0")"
tail -n +4 CHECKSUMS.sha256 | sha256sum -c
\`\`\`

## Archive Information

- **Generated**: {TIMESTAMP}
- **File count**: {FILE_COUNT}
- **Total size**: {TOTAL_SIZE}
- **Manifest hash**: {MANIFEST_HASH}

## Recommended Verification Schedule

- **Daily**: Quick manifest integrity check
- **Weekly**: Full file verification
- **After transfer**: Full verification immediately after copying/downloading
- **Before backup**: Verify source integrity before creating backup

## If Verification Fails

1. **Manifest integrity fai