ssmd-deploy

What this skill does

# ssmd-deploy

Deploy ssmd components from code change through to running in GKE prod.

## Overview

ssmd images are built by GitHub Actions on tag push. Deployments are GitOps via Flux: update image tags in `clusters/gke-prod/` manifests, push, and Flux reconciles.

There are two repos involved:
- **ssmd** (`899bushwick/ssmd/`) — source code, where tags trigger image builds
- **899bushwick** — deployment manifests and DQ/data scripts

## Component Registry

### ssmd repo (Rust + Deno components)

| Component | Tag Format | Image | Manifest Path(s) |
|-----------|-----------|-------|-------------------|
| connector | `v*` (e.g. `v0.9.11`) | `ghcr.io/aaronwald/ssmd-connector` | `clusters/gke-prod/apps/ssmd/generated/connector-{crypto,kraken-futures,polymarket}.yaml` |
| archiver | `v*` (same tag as connector) | `ghcr.io/aaronwald/ssmd-archiver` | `clusters/gke-prod/apps/ssmd/generated/archiver-{kalshi-crypto,kraken-futures,polymarket}.yaml` |
| parquet-gen | `parquet-gen-v*` | `ghcr.io/aaronwald/ssmd-parquet-gen` | `clusters/gke-prod/apps/ssmd/archiver/cronjob-parquet-gen.yaml` |
| operator | `operator-v*` | `ghcr.io/aaronwald/ssmd-operator` | `clusters/gke-prod/apps/ssmd/operator/deployment.yaml` |
| data-ts | `data-ts-v*` | `ghcr.io/aaronwald/ssmd-data-ts` | `clusters/gke-prod/apps/ssmd/data-ts/deployment.yaml` |
| cli-ts | `cli-ts-v*` | `ghcr.io/aaronwald/ssmd-cli-ts` | (used in CronJobs: secmaster-crypto, kraken-sync, polymarket-sync, fees-daily, health-daily) |
| worker | `worker-v*` | `ghcr.io/aaronwald/ssmd-worker` | `clusters/gke-prod/apps/ssmd/worker/deployment.yaml` |
| signal-runner | `signal-runner-v*` | `ghcr.io/aaronwald/ssmd-signal-runner` | `clusters/gke-prod/apps/ssmd/signals/` |

### 899bushwick repo (Python/DQ components)

| Component | Tag Format | Image | Manifest Path |
|-----------|-----------|-------|---------------|
| dq | `dq-v*` | `ghcr.io/aaronwald/ssmd-dq` | `clusters/gke-prod/apps/ssmd/cronjobs/dq-daily.yaml` |

## Deployment Procedure

### Step 1: Determine what to deploy

Identify which components changed. Common groupings:
- **Connector schema change** (e.g. new parquet column): connector + archiver (`v*`), parquet-gen (`parquet-gen-v*`)
- **Connector-only change**: connector + archiver (`v*`)
- **DQ logic change**: dq (`dq-v*`)
- **parquet-gen only**: parquet-gen (`parquet-gen-v*`)
- **Operator CRD change**: operator (`operator-v*`), also copy CRDs to `clusters/gke-prod/apps/ssmd/operator/crds.yaml`

### Step 2: Determine next version numbers

Check the latest tags:
```bash
# For ssmd repo components
git -C <ssmd-path> tag --sort=-creatordate | grep "^v[0-9]" | head -3      # connector/archiver
git -C <ssmd-path> tag --sort=-creatordate | grep "^parquet-gen-v" | head -3
git -C <ssmd-path> tag --sort=-creatordate | grep "^operator-v" | head -3
git -C <ssmd-path> tag --sort=-creatordate | grep "^data-ts-v" | head -3

# For 899bushwick components
git -C <899bushwick-path> tag --sort=-creatordate | grep "^dq-v" | head -3
```

Increment the patch version (e.g. `v0.9.10` -> `v0.9.11`).

### Step 3: Commit and push source changes

```bash
# ssmd repo
git -C <ssmd-path> add <changed-files>
git -C <ssmd-path> commit -m "<type>: <description>"
git -C <ssmd-path> push origin main

# 899bushwick repo (for DQ/data changes)
git -C <899bushwick-path> add <changed-files>
git -C <899bushwick-path> commit -m "<type>: <description>"
git -C <899bushwick-path> push origin main
```

### Step 4: Tag and push to trigger builds

```bash
# ssmd repo tags
git -C <ssmd-path> tag <tag>
git -C <ssmd-path> push origin <tag>

# 899bushwick tags
git -C <899bushwick-path> tag <tag>
git -C <899bushwick-path> push origin <tag>
```

Multiple tags can be pushed at once: `git push origin v0.9.11 parquet-gen-v0.7.0`

### Step 5: Wait for GitHub Actions builds

```bash
gh run list --repo aaronwald/ssmd --limit 5 --json name,status,conclusion,headBranch
gh run list --repo aaronwald/899bushwick --limit 3 --json name,status,conclusion,headBranch
```

Poll until all relevant builds show `"conclusion":"success"`. Rust builds take 3-5 minutes; Python/Deno builds take 1-2 minutes.

### Step 6: Update deployment manifests

Update image tags in the manifest files listed in the Component Registry table above.

The `image:` field format is: `ghcr.io/aaronwald/<image-name>:<version-without-prefix>`
- Tag `v0.9.11` -> image tag `0.9.11`
- Tag `parquet-gen-v0.7.0` -> image tag `0.7.0`
- Tag `dq-v0.8.0` -> image tag `0.8.0`

**Important**: Connector and archiver share the same version number (same `v*` tag triggers both builds).

### Step 7: Commit and push manifests

```bash
git -C <899bushwick-path> add <manifest-files> ssmd  # include ssmd submodule if updated
git -C <899bushwick-path> commit -m "deploy: <component> <version>[, <component> <version>...]"
git -C <899bushwick-path> push origin main
```

### Step 8: Verify Flux reconciliation

Flux auto-reconciles on git push. To force or check:
```bash
flux reconcile kustomization ssmd --with-source
kubectl get pods -n ssmd -w  # watch for rolling restarts
```

## Notes

- Connector and archiver always deploy together (same `v*` tag). Even if only one changed, both get rebuilt.
- The ssmd submodule pointer in 899bushwick should be updated when ssmd code changes are deployed.
- CronJobs (parquet-gen, DQ, secmaster, etc.) pick up new images on next scheduled run, not immediately.
- For Deployments (connector, archiver, operator, data-ts, worker), Flux triggers a rolling restart on image change.
- **Never** `kubectl apply` directly — all changes go through git + Flux.

## Examples

### Deploy a connector + parquet-gen change
```
Components: connector v0.9.11, parquet-gen v0.7.0
Tags: v0.9.11, parquet-gen-v0.7.0 (in ssmd repo)
Manifests: 3 connector yamls, 3 archiver yamls, 1 parquet-gen cronjob
```

### Deploy a DQ-only change
```
Components: dq v0.8.0
Tags: dq-v0.8.0 (in 899bushwick repo)
Manifests: 1 dq-daily cronjob
```

### Deploy operator CRD update
```
Components: operator v0.5.6
Tags: operator-v0.5.6 (in ssmd repo)
Extra step: copy CRDs from ssmd-operators to clusters/gke-prod/apps/ssmd/operator/crds.yaml
Manifests: 1 operator deployment + 1 crds.yaml
```