implementing-secrets-management-with-vault

What this skill does

# Implementing Secrets Management with Vault

## When to Use

- When applications store database passwords, API keys, or certificates in environment variables or config files
- When migrating from static long-lived credentials to dynamic short-lived secrets
- When Kubernetes workloads need secure access to database credentials or cloud provider APIs
- When compliance requirements mandate centralized credential management with audit logging
- When CI/CD pipelines contain hardcoded secrets that represent supply chain risk

**Do not use** for AWS-only environments where AWS Secrets Manager suffices without multi-cloud requirements, for application-level encryption logic (though Vault Transit can help), or for identity federation (see managing-cloud-identity-with-okta).

## Prerequisites

- HashiCorp Vault server deployed in HA mode (Consul or Raft storage backend)
- TLS certificates for Vault listener endpoints
- Vault Enterprise license for namespaces, Sentinel policies, and replication (optional)
- Kubernetes cluster with Vault Agent Injector or CSI provider for workload integration

## Workflow

### Step 1: Deploy Vault in High Availability Mode

Deploy Vault using Integrated Storage (Raft) for HA without external dependencies. Configure TLS, audit logging, and auto-unseal using a cloud KMS.

```hcl
# vault-config.hcl
storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-node-1"

  retry_join {
    leader_api_addr = "https://vault-node-2.internal:8200"
  }
  retry_join {
    leader_api_addr = "https://vault-node-3.internal:8200"
  }
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_cert_file = "/opt/vault/tls/vault.crt"
  tls_key_file  = "/opt/vault/tls/vault.key"
}

seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "alias/vault-unseal-key"
}

api_addr      = "https://vault-node-1.internal:8200"
cluster_addr  = "https://vault-node-1.internal:8201"

telemetry {
  prometheus_retention_time = "30s"
  disable_hostname         = true
}
```

```bash
# Initialize Vault
vault operator init -key-shares=5 -key-threshold=3

# Enable audit logging
vault audit enable file file_path=/var/log/vault/audit.log

# Enable syslog audit for SIEM integration
vault audit enable syslog tag="vault" facility="AUTH"
```

### Step 2: Configure Authentication Methods

Enable authentication backends for human operators, applications, and CI/CD pipelines. Use AppRole for machine authentication and OIDC for human access.

```bash
# Enable OIDC auth for human users via Okta
vault auth enable oidc
vault write auth/oidc/config \
  oidc_discovery_url="https://company.okta.com/oauth2/default" \
  oidc_client_id="vault-client-id" \
  oidc_client_secret="vault-client-secret" \
  default_role="default"

# Enable AppRole for application authentication
vault auth enable approle
vault write auth/approle/role/web-app \
  secret_id_ttl=10m \
  token_num_uses=10 \
  token_ttl=20m \
  token_max_ttl=30m \
  secret_id_num_uses=1 \
  token_policies="web-app-policy"

# Enable Kubernetes auth for pod-based access
vault auth enable kubernetes
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc:443" \
  token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
```

### Step 3: Enable Dynamic Secret Engines

Configure database secret engines to generate short-lived credentials on demand. Each credential set has a TTL and is automatically revoked when it expires.

```bash
# Enable database secrets engine for PostgreSQL
vault secrets enable database
vault write database/config/production-db \
  plugin_name=postgresql-database-plugin \
  allowed_roles="readonly,readwrite" \
  connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/production?sslmode=require" \
  username="vault_admin" \
  password="initial-password"

# Rotate the root credentials so Vault manages them exclusively
vault write -force database/rotate-root/production-db

# Create a readonly role with 1-hour TTL
vault write database/roles/readonly \
  db_name=production-db \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  revocation_statements="REVOKE ALL ON ALL TABLES IN SCHEMA public FROM \"{{name}}\"; DROP ROLE IF EXISTS \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Enable AWS secrets engine for dynamic IAM credentials
vault secrets enable aws
vault write aws/config/root \
  access_key=AKIAEXAMPLE \
  secret_key=secretkey \
  region=us-east-1

vault write aws/roles/deploy-role \
  credential_type=iam_user \
  [email protected] \
  default_sts_ttl=3600
```

### Step 4: Integrate with Kubernetes Workloads

Use the Vault Agent Injector or CSI Provider to deliver secrets to pods without application code changes. Secrets are rendered as files in a shared volume.

```yaml
# Kubernetes deployment with Vault Agent Injector annotations
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "web-app"
        vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/readonly"
        vault.hashicorp.com/agent-inject-template-db-creds: |
          {{- with secret "database/creds/readonly" -}}
          export DB_USERNAME="{{ .Data.username }}"
          export DB_PASSWORD="{{ .Data.password }}"
          {{- end }}
    spec:
      serviceAccountName: web-app
      containers:
        - name: web-app
          image: company/web-app:v2.1
          command: ["/bin/sh", "-c", "source /vault/secrets/db-creds && ./start.sh"]
```

### Step 5: Implement Transit Encryption and PKI

Use the Transit secrets engine for application-level encryption without managing keys in application code. Deploy the PKI engine for automatic TLS certificate management.

```bash
# Enable Transit engine for encryption as a service
vault secrets enable transit
vault write -f transit/keys/payment-data type=aes256-gcm96

# Encrypt sensitive data
vault write transit/encrypt/payment-data \
  plaintext=$(echo "card-number-4111-1111-1111-1111" | base64)

# Enable PKI for internal certificate management
vault secrets enable pki
vault secrets tune -max-lease-ttl=87600h pki

# Generate root CA
vault write pki/root/generate/internal \
  common_name="Internal Root CA" \
  ttl=87600h

# Configure intermediate CA for issuing certificates
vault secrets enable -path=pki_int pki
vault write pki_int/intermediate/generate/internal \
  common_name="Internal Intermediate CA" \
  ttl=43800h

# Create a role for issuing certificates
vault write pki_int/roles/internal-services \
  allowed_domains="internal.company.com" \
  allow_subdomains=true \
  max_ttl=720h
```

### Step 6: Establish Policies and Audit Trail

Define fine-grained ACL policies following least privilege. Enable comprehensive audit logging for all secret access and administrative operations.

```hcl
# web-app-policy.hcl
path "database/creds/readonly" {
  capabilities = ["read"]
}

path "transit/encrypt/payment-data" {
  capabilities = ["update"]
}

path "transit/decrypt/payment-data" {
  capabilities = ["update"]
}

path "secret/data/web-app/*" {
  capabilities = ["read", "list"]
}

# Deny access to admin paths
path "sys/*" {
  capabilities = ["deny"]
}
```

```bash
# Apply the policy
vault policy write web-app-policy web-app-policy.hcl

# Verify audit log captures all operations
vault audit list -detailed
```

## Key Concepts

| Term | Definition |
|------|------------|
| Dynamic Secrets | Credentials generated on-demand with automatic expiration and revocation, eliminating long-lived static credentials |
| Secret Engine | Vault component that stores, generates, or encrypts data; includes KV, database, AWS, PKI, and Transit engines |
| Auto-Unseal | Cloud KMS-based m