aws-cost-investigation

What this skill does

# AWS Cost Investigation

Operational skill for diagnosing AWS cost spikes and auditing for ongoing waste. The focus is on **diagnostic flow** (data-first, not guess-first) plus a **concrete trap inventory** with detection CLI for each.

## When to invoke

**Symptoms:**

- "The bill is up $X with no deploys" / "AWS bill spiked last month."
- An anomaly notification from AWS Cost Anomaly Detection.
- Cost Explorer dashboards show large `(no tag)` slices despite tagging policies.
- NAT Gateway charges growing month over month.
- Looking at a Savings Plan / Reserved Instance commitment decision.
- A general account audit ("find the waste").
- Designing a cost-allocation tagging strategy.

## Cross-cutting rules

1. **Data first, guesses never.** When asked to diagnose a spike, the first action is to query Cost Explorer. Do NOT guess "probably S3" or "probably NAT" without numbers. Naming a likely culprit without data is anti-pattern #1.
2. **Compare windows of equal length.** A 7-day spike compares to the prior 7 days, not month-to-date. A monthly spike compares to the same days of the prior month, not the full prior month.
3. **Never quote a specific dollar amount as a pricing fact.** AWS prices change. State relative magnitudes (Gateway endpoints are free; Interface endpoints are cheaper than NAT for high egress) and link to the [AWS pricing page](https://aws.amazon.com/pricing/) for current numbers when a precise answer is needed.
4. **Activation is the silent gate for tag-based analysis.** A tag on a resource is invisible to Cost Explorer / CUR until it's *activated* as a Cost Allocation Tag. See [Cost allocation tagging](#cost-allocation-tagging).
5. **Stop-the-bleeding before re-architecting.** When a runaway cost is identified, the first action is to cap it (set retention, delete idle, add a budget alarm) — not redesign the workload.

## Diagnostic flow for spikes

Run the steps in order. Don't skip ahead.

### Step 1 — Service-level diff

```bash
# Compare two equal-length windows (here: last 7d vs prior 7d)
END=$(date -u +%Y-%m-%d)
START=$(date -u -d '-7 days' +%Y-%m-%d)
PRIOR_END=$START
PRIOR_START=$(date -u -d '-14 days' +%Y-%m-%d)

for window in "$PRIOR_START $PRIOR_END" "$START $END"; do
  read s e <<<"$window"
  aws ce get-cost-and-usage \
    --time-period Start=$s,End=$e \
    --granularity DAILY \
    --metrics UnblendedCost \
    --group-by Type=DIMENSION,Key=SERVICE \
    --query 'ResultsByTime[].Groups[]' \
    --output json > /tmp/ce-$s.json
done
# Diff per-service totals between the two files to find the top growers.
```

Identify the top-growing service. Do NOT proceed to step 2 without this.

### Step 2 — Drill into the top service by usage type

```bash
aws ce get-cost-and-usage \
  --time-period Start=$START,End=$END \
  --granularity DAILY \
  --metrics UnblendedCost \
  --filter '{"Dimensions":{"Key":"SERVICE","Values":["Amazon Elastic Compute Cloud - Compute"]}}' \
  --group-by Type=DIMENSION,Key=USAGE_TYPE
```

Usage-type taxonomy worth recognizing:

| Usage type prefix | What it is |
|---|---|
| `BoxUsage:<instance>` | EC2 instance hours by instance type |
| `EBS:VolumeUsage*` / `EBS:SnapshotUsage` | Block storage / snapshots |
| `DataTransfer-Out-Bytes` | Egress to internet from this region |
| `DataTransfer-Regional-Bytes` | Cross-AZ within region |
| `NatGateway-Bytes` / `NatGateway-Hours` | NAT processing / hourly |
| `LoadBalancerUsage` / `LCUUsage` | ELB hourly / load balancer capacity units |
| `Requests-Tier1` / `Requests-Tier2` (S3) | S3 PUT/COPY vs GET/SELECT |
| `TimedStorage-*` (S3) | Storage by class |
| `CW:Requests` / `CW:GMD-Metrics` | CloudWatch API + custom metric storage |
| `<service>:Lambda-GB-Second` | Lambda compute-time |

### Step 3 — Resource-level attribution (opt-in)

`aws ce get-cost-and-usage-with-resources` returns per-resource costs but requires:

- Opt-in via Cost Explorer Settings page (one-time).
- Daily granularity for everything except EC2 (which supports hourly).
- 14-day lookback window only.

```bash
aws ce get-cost-and-usage-with-resources \
  --time-period Start=$START,End=$END \
  --granularity DAILY \
  --metrics UnblendedCost \
  --filter '{"Dimensions":{"Key":"SERVICE","Values":["Amazon Elastic Compute Cloud - Compute"]}}' \
  --group-by Type=DIMENSION,Key=RESOURCE_ID
```

For costs older than 14 days, the **CUR** is the right tool — see [CUR + Athena](#cost--usage-report-cur).

### Step 4 — Tag axis (if cost-allocation tags are activated)

```bash
aws ce get-cost-and-usage \
  --time-period Start=$START,End=$END \
  --granularity DAILY \
  --metrics UnblendedCost \
  --group-by Type=TAG,Key=team Type=DIMENSION,Key=SERVICE
```

If the response shows large `(no tag)` slices, the tag isn't activated — see [Cost allocation tagging](#cost-allocation-tagging).

### Step 5 — Sanity-check against AWS's anomaly detector

```bash
aws ce get-anomalies \
  --date-interval StartDate=$PRIOR_START,EndDate=$END
```

AWS Cost Anomaly Detection runs its own ML and may have already flagged the cause with `RootCauses` populated. Free service; underused.

## The cost-trap inventory

These are recurring sources of silent cost growth. Audit each in any account you don't know well.

### CloudWatch log groups with no retention

Lambda, ECS, CodeBuild, API Gateway all auto-create log groups with retention `null` = never expire.

```bash
aws logs describe-log-groups \
  --query 'logGroups[?retentionInDays==`null`].[logGroupName,storedBytes]' \
  --output table

# Fix: set retention org-wide (example: 30 days)
aws logs describe-log-groups --query 'logGroups[].logGroupName' --output text \
  | xargs -n1 -I{} aws logs put-retention-policy --log-group-name {} --retention-in-days 30
```

Going forward, Terraform pattern:

```hcl
resource "aws_cloudwatch_log_group" "lambda" {
  name              = "/aws/lambda/${aws_lambda_function.x.function_name}"
  retention_in_days = 30
  # Create this resource BEFORE the Lambda — otherwise Lambda creates the group with null retention.
}
```

### Unattached EBS volumes

```bash
aws ec2 describe-volumes \
  --filters Name=status,Values=available \
  --query 'Volumes[].[VolumeId,Size,VolumeType,CreateTime]' \
  --output table
```

Volumes in `available` state bill at full storage cost. Common after EC2 termination with "delete on termination" not set. Snapshot then delete, or just delete if not needed.

### Old EBS snapshots (no lifecycle)

```bash
aws ec2 describe-snapshots --owner-ids self \
  --query 'Snapshots[?StartTime<=`2024-01-01`].[SnapshotId,StartTime,VolumeSize,Description]' \
  --output table
```

Snapshots accumulate. Use **Data Lifecycle Manager (DLM)** to set automated retention going forward; clean up old ones one-time.

### Unattached Elastic IPs

```bash
aws ec2 describe-addresses \
  --query 'Addresses[?AssociationId==`null`].[PublicIp,AllocationId,Domain]' \
  --output table
```

EIPs not attached to a running instance bill hourly. After AWS's 2024 pricing change, attached EIPs also bill — but unattached is always wasted.

### Idle Load Balancers

```bash
# ALBs and NLBs with low request count over the past 7 days
aws elbv2 describe-load-balancers \
  --query 'LoadBalancers[].[LoadBalancerArn,LoadBalancerName,Type]' \
  --output text \
  | while read arn name type; do
    requests=$(aws cloudwatch get-metric-statistics \
      --namespace AWS/ApplicationELB \
      --metric-name RequestCount \
      --dimensions Name=LoadBalancer,Value=$(echo $arn | cut -d/ -f2-) \
      --start-time $(date -u -d '-7 days' +%Y-%m-%dT%H:%M:%SZ) \
      --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
      --period 604800 --statistics Sum \
      --query 'Datapoints[0].Sum' --output text)
    echo "$name $type $requests"
  done
```

LBs with near-zero traffic are usually leftover from migrated/deleted services.

### NAT Gateway data egress

The single biggest silent killer for workloads in private subnets. Diagnostic:

```bash
# How many bytes did NAT GW process this month?