podium-webhook-reliability

What this skill does

# Podium Webhook Reliability

## Overview

Receive Podium webhooks in production without forged events, double-charged AI side-effects, lost notifications, or out-of-order conversation events. This is not an introductory webhook walkthrough — it is the receiver code your integration runs when Podium retries a 5xx response six times over 24 hours, when a leaked secret lets an attacker POST forged events, when a batch delivery arrives with `conversation.deleted` ahead of `conversation.created`, and when on-call needs to drain and replay 800 failed events without re-firing the ones that already succeeded.

The six production failures this skill prevents:

1. **Missing signature verification** — a webhook endpoint that accepts any POST will accept forged events. An attacker who learns the URL can create phantom contacts, fire phantom review requests, or impersonate a real customer in a webchat. HMAC-SHA256 over the raw request body is non-optional and must run before any handler logic.
2. **Replay attacks against a stateless handler** — a valid signed event POSTed twice (or 1000 times) re-runs every side effect each time. Signature validity alone is not enough — the receiver must reject events whose timestamp falls outside a 5-minute window AND whose nonce has already been seen.
3. **Duplicate event processing from Podium retries** — Podium retries webhook delivery on 5xx for up to 24 hours. Without an idempotency cache, every retry re-runs the handler (writes the contact again, fires the review request again, double-charges an AI call). `SET NX EX 86400` on the event_id is the cheapest fix that exists.
4. **Lost events without a dead-letter queue** — if a handler raises and Podium retries six times and gives up, the event is gone. On-call has nothing to replay. Every handler exception must persist the raw signed payload to a DLQ before the response returns 5xx, so the event is recoverable independent of Podium's retry clock.
5. **Batch event reordering** — Podium can deliver multiple events in one POST and ordering across deliveries is not guaranteed. A naive handler processes `conversation.deleted` before `conversation.created` and the system observes a delete on a contact that does not exist. Within a batch, sort by `occurred_at` before dispatch; across batches, gate causally-dependent handlers on the precondition existing.
6. **Timing-attack vulnerability on signature compare** — `received_sig == computed_sig` with `==` short-circuits on the first byte mismatch. An attacker measures response latency to recover the signature byte-by-byte over a few thousand probes. Always use `hmac.compare_digest`, which is constant-time over the longer of the two inputs.

## Prerequisites

- Python 3.10+ with `fastapi`, `uvicorn`, `httpx`, and `redis` (in-memory fallback for dev is provided)
- Podium account with an OAuth app authorized for webhook delivery: Settings → Developer → Apps → Webhooks
- The webhook signing secret from the app's Webhooks tab (saved to a secret store — never committed)
- A receiver URL reachable from Podium (publicly resolvable HTTPS endpoint with valid cert)
- Redis 6+ for production dedup + DLQ; an in-memory dict + SQLite file fallback exists for dev
- A `podium-auth` instance if your handler needs to call back into the Podium API after processing

## Instructions

Build in this order. Each section neutralizes one production failure mode.

### 1. HMAC-SHA256 signature verification on the raw body (neutralizes forgery)

Verify the signature against the **raw, unparsed** request body. Any framework middleware that JSON-decodes-and-re-encodes before signature check will fail because whitespace and key ordering change. Read the body once, verify, then parse:

```python
import hmac, hashlib
from fastapi import FastAPI, Request, HTTPException, Header

app = FastAPI()
SIGNING_SECRET = os.environ["PODIUM_WEBHOOK_SECRET"].encode("utf-8")

@app.post("/webhooks/podium")
async def receive(request: Request, x_podium_signature: str = Header(None)):
    raw = await request.body()              # bytes — DO NOT decode/re-encode
    if not x_podium_signature:
        raise HTTPException(401, "missing X-Podium-Signature")
    if not verify_signature(raw, x_podium_signature):
        raise HTTPException(401, "signature mismatch")
    # ... continue with replay/dedup/dispatch
```

```python
def verify_signature(body: bytes, header_value: str) -> bool:
    # Podium signature header format: "t=<unix_ts>,v1=<hex_hmac>"
    # Adapt to current spec — verify against the Podium developer docs at integration time.
    parts = dict(p.split("=", 1) for p in header_value.split(",") if "=" in p)
    ts, sig = parts.get("t"), parts.get("v1")
    if not ts or not sig:
        return False
    signed_payload = f"{ts}.".encode("utf-8") + body
    expected = hmac.new(SIGNING_SECRET, signed_payload, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, sig)    # constant-time, byte-by-byte safe
```

The `t=` timestamp is what makes the next mitigation possible. A signature alone with no timestamp is replayable forever.

### 2. Replay-attack window (neutralizes timestamp replay)

Reject any event whose signed timestamp is more than 5 minutes from now (in either direction — clock skew goes both ways). This bounds the replay window an attacker has even if they capture a valid signed event off the wire:

```python
import time

REPLAY_WINDOW_SECONDS = 300        # 5 minutes; tune to your clock-skew tolerance

def within_replay_window(ts_str: str) -> bool:
    try:
        ts = int(ts_str)
    except (TypeError, ValueError):
        return False
    return abs(time.time() - ts) <= REPLAY_WINDOW_SECONDS
```

Wire `within_replay_window(parts["t"])` immediately after signature verification. A failed window check is a 401 — do not return 200, do not enqueue, do not log the body (the attacker is probing).

### 3. Idempotent dedup with `SET NX EX 86400` (neutralizes duplicate processing)

Every Podium webhook carries an `event_id` (or equivalent unique identifier — verify against the current schema). Reject any event whose `event_id` is already in the dedup cache. Use Redis `SET key value NX EX 86400` so the check and the claim are atomic; 86400 seconds matches Podium's 24-hour retry ceiling:

```python
import redis.asyncio as redis

REDIS = redis.from_url(os.environ.get("REDIS_URL", "redis://localhost:6379/0"))

async def claim_event(event_id: str) -> bool:
    # Returns True if this process is the first to see this event_id.
    # Returns False if the event_id is already in the cache (duplicate).
    return await REDIS.set(f"podium:evt:{event_id}", "1", nx=True, ex=86400)
```

In the handler:

```python
event = json.loads(raw)
event_id = event["id"]
if not await claim_event(event_id):
    return {"status": "duplicate", "event_id": event_id}     # 200 — Podium stops retrying
```

Returning 200 on duplicate is correct — Podium has correctly delivered, the receiver has correctly identified it as already processed. The handler is idempotent by construction.

For dev / smoke environments without Redis, fall back to an in-memory `set()` with a periodic eviction loop. Documented in `references/implementation.md`.

### 4. Dead-letter queue before responding 5xx (neutralizes silent event loss)

Wrap every handler invocation in a try/except. On any exception, persist the **raw signed payload plus the timestamp plus the signature** to the DLQ before letting the exception bubble. The DLQ entry is the recovery anchor — `dlq_replay.py` can re-POST it to the handler later:

```python
async def safe_dispatch(event: dict, raw: bytes, sig_header: str):
    try:
        await dispatch(event)
    except Exception as e:
        await dlq_persist({
            "event_id": event.get("id"),
            "event_type": event.get("type"),
            "raw_body": raw.decode("utf-8", errors="replace"),
            "signature_header": sig_header,
            "occurred_at": event.get("occurred_at"),