devops-pipeline

What this skill does

# DevOps Pipeline

Implement comprehensive DevOps quality gates adapted to project type, with a **shift-left philosophy**: run as many checks as possible locally via pre-commit so developers get fast feedback and CI is a safety net rather than the primary gate.

**Core principle**: If a check can run locally in under ~60 seconds, it belongs in pre-commit. GitHub Actions should handle things that can't run locally: matrix version testing, secrets-based security scans, deployment, and reporting.

To stay within the agent's context budget, this SKILL keeps templates short and links to `references/*.md` for language-specific configs, workflow templates, and the CLI E2E script.

## Repo Sync Before Edits (mandatory)
Before creating/updating/deleting files in an existing repository, sync the current branch with remote:

```bash
branch="$(git rev-parse --abbrev-ref HEAD)"
git fetch origin
git pull --rebase origin "$branch"
```

If the working tree is not clean, stash first, sync, then restore:

```bash
git stash push -u -m "pre-sync"
branch="$(git rev-parse --abbrev-ref HEAD)"
git fetch origin && git pull --rebase origin "$branch"
git stash pop
```

If `origin` is missing, pull is unavailable, or rebase/stash conflicts occur, stop and ask the user before continuing.

## Workflow

### 1. Analyze Project

Detect project characteristics:

```bash
# Check for package files and configs
ls -la package.json pyproject.toml Cargo.toml go.mod pom.xml build.gradle *.csproj 2>/dev/null
ls -la .eslintrc* .prettierrc* tsconfig.json mypy.ini setup.cfg ruff.toml 2>/dev/null
ls -la .pre-commit-config.yaml .github/workflows/*.yml 2>/dev/null
```

Identify:
- **Languages**: JS/TS, Python, Go, Rust, Java, C#, etc.
- **Frameworks**: React, Next.js, Django, FastAPI, etc.
- **Build system**: npm, yarn, pnpm, pip, poetry, cargo, go, maven, gradle
- **Existing tooling**: Linters, formatters, type checkers already configured
- **Is this a CLI tool?** — if yes, enumerate all commands/subcommands (check README, `--help`, `click`/`argparse`/`cobra` source) to build an E2E test suite

### 2. Configure Pre-commit Hooks (maximize local coverage)

Install pre-commit framework:

```bash
pip install pre-commit  # or brew install pre-commit
```

Create `.pre-commit-config.yaml` based on detected stack. See [references/precommit-configs.md](references/precommit-configs.md) for language-specific configurations.

**What to put in pre-commit (run on every commit):**
- Format checks (Prettier, Black/Ruff, gofmt, rustfmt)
- Lint (ESLint, Ruff, golangci-lint, Clippy)
- Type checks (tsc, mypy)
- Security scans that work offline (Bandit, cargo-audit, gosec, `detect-secrets`)
- Unit tests (fast, <10s) — always on `commit` stage
- Build/compile verification (catches import errors, compile failures early)

**What to put in pre-commit on `push` stage (run on git push):**
- Full test suite (unit + integration)
- **End-to-end tests for every CLI command** (see below)
- Coverage checks
- Slower linters (full golangci-lint ruleset)

**What stays in GitHub Actions only:**
- Matrix version testing (multiple Node/Python/Go versions)
- Secrets-based scans (Snyk, SAST tools needing tokens)
- Deployment / release workflows
- Flaky or environment-sensitive tests that need a clean VM

#### CLI End-to-End Testing

If the project is a CLI tool, create `scripts/e2e_test.sh` that exercises every command/subcommand to verify the CLI works end-to-end (not just compiles). Wire it into pre-commit on the `push` stage.

See [references/cli-e2e.md](references/cli-e2e.md) for command discovery patterns, the script template, and the pre-commit hook snippet.

Install hooks:

```bash
pre-commit install
pre-commit install --hook-type pre-push  # also install push-stage hooks
pre-commit run --all-files  # Test on existing code
```

### 3. Create GitHub Actions Workflows (lean CI)

Create `.github/workflows/ci.yml` — but keep it lean since pre-commit already catches most issues. See [references/github-actions.md](references/github-actions.md) for workflow templates.

GitHub Actions responsibilities (things pre-commit can't do):
- Matrix testing across language versions (important for libraries)
- Upload coverage reports (Codecov, etc.)
- Deployment on merge to main
- PR status comments/badges
- Secrets-dependent scans

Since pre-commit already runs lint, format, type-check, unit tests, and E2E tests — the CI workflow can be simpler: install deps → run pre-commit → run tests with coverage upload → build artifact.

```yaml
# Minimal CI when pre-commit covers everything locally:
- name: Run pre-commit
  run: pre-commit run --all-files

- name: Run tests with coverage
  run: <test-command> --cov --cov-report=xml

- name: Upload coverage
  uses: codecov/codecov-action@v4
```

### 4. Verify Pipeline

```bash
# Test all pre-commit hooks (commit stage)
pre-commit run --all-files

# Test push-stage hooks (includes E2E)
pre-commit run --all-files --hook-stage push

# Verify the CLI E2E script directly
bash scripts/e2e_test.sh
```

If all local checks pass, GitHub Actions becomes a thin verification layer, not the primary quality gate.

## Tool Selection by Language

| Language | Formatter | Linter | Type Check | Security | Tests |
|----------|-----------|--------|------------|----------|-------|
| JS/TS | Prettier | ESLint | tsc | npm audit | Jest/Vitest |
| Python | Ruff/Black | Ruff | mypy | Bandit + detect-secrets | pytest |
| Go | gofmt | golangci-lint | built-in | gosec | go test |
| Rust | rustfmt | Clippy | built-in | cargo-audit | cargo test |
| Java | google-java-format | Checkstyle | - | SpotBugs | mvn test |

## What Runs Where

| Check | Pre-commit (commit) | Pre-commit (push) | GitHub Actions |
|-------|---------------------|-------------------|----------------|
| Formatting | ✓ | — | — |
| Linting | ✓ | — | — |
| Type checking | ✓ | — | — |
| Security scan (offline) | ✓ | — | — |
| Unit tests (fast) | ✓ | — | — |
| Full test suite | — | ✓ | ✓ (coverage upload) |
| CLI E2E tests | — | ✓ | — |
| Multi-version matrix | — | — | ✓ |
| Deploy | — | — | ✓ |

## Expected Output

After running the skill, the repository contains:

1. **`.pre-commit-config.yaml`** — hooks for formatting, linting, type-checking, and unit tests on `commit` stage; full test suite and E2E tests on `push` stage.
2. **`.github/workflows/ci.yml`** — lean CI that re-runs pre-commit and uploads coverage; no duplicate lint/format steps.
3. **`scripts/e2e_test.sh`** (CLI projects only) — executable script exercising every CLI command/subcommand.

Example `.pre-commit-config.yaml` snippet for a Python project:
```yaml
repos:
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.4.4
    hooks:
      - id: ruff
        stages: [commit]
      - id: ruff-format
        stages: [commit]
  - repo: local
    hooks:
      - id: mypy
        name: mypy type check
        entry: mypy src/
        language: system
        stages: [commit]
      - id: pytest-fast
        name: fast unit tests
        entry: pytest tests/unit -x -q
        language: system
        stages: [commit]
      - id: pytest-full
        name: full test suite
        entry: pytest --cov=src --cov-report=xml
        language: system
        stages: [push]
```

## Acceptance Criteria

A run passes when **all** of the following are true:

- [ ] `.pre-commit-config.yaml` exists at the repo root and lists at least one hook for the detected primary language (formatter, linter, or type checker).
- [ ] All checks runnable locally in under ~60 seconds are configured in pre-commit, not GitHub Actions.
- [ ] At least one `.github/workflows/*.yml` exists and runs only the things pre-commit cannot (matrix builds, secret-scanning, deployment, or release).
- [ ] `pre-commit run --all-files` succeeds (or its failures are surfaced explicitly to the user, not auto-suppressed).
- [ ] For CLI projects, an E2E test step is wired into either pre-commit or CI per the language reference files.
- [ ] No duplication: