buildkite-pipelines

What this skill does

# Buildkite Pipelines

Pipeline YAML is the core of Buildkite CI/CD. This skill covers writing, optimizing, and troubleshooting `.buildkite/pipeline.yml` — step types, caching, parallelism, annotations, retry, dynamic pipelines, matrix builds, plugins, notifications, artifacts, and concurrency.

## Quick Start

Create `.buildkite/pipeline.yml` in the repository root:

```yaml
steps:
  - label: ":hammer: Tests"
    command: "npm test"
    artifact_paths: "coverage/**/*"

  - wait

  - label: ":rocket: Deploy"
    command: "scripts/deploy.sh"
    branches: "main"
```

Set the pipeline's initial command in Buildkite to upload this file:

```yaml
steps:
  - label: ":pipeline: Upload"
    command: buildkite-agent pipeline upload
```

The agent reads `.buildkite/pipeline.yml` and uploads the steps to Buildkite for execution.

Buildkite looks for `.buildkite/pipeline.yml` by default. Override the path with `buildkite-agent pipeline upload path/to/other.yml`.

> For creating pipelines programmatically, see the **buildkite-api** skill.
> For agent and queue setup, see the **buildkite-agent-infrastructure** skill.

## Step Types

| Type | Purpose | Minimal syntax |
|------|---------|---------------|
| **command** | Run a shell command | `- command: "make test"` |
| **wait** | Block until all previous steps pass | `- wait` |
| **block** | Pause for manual approval | `- block: ":shipit: Release"` |
| **trigger** | Start a build on another pipeline | `- trigger: "deploy-pipeline"` |
| **group** | Visually group steps (collapsible) | `- group: "Tests"` with nested `steps:` |
| **input** | Collect user input before continuing | `- input: "Release version"` with `fields:` |

For detailed attributes and advanced examples of each step type, see `references/step-types-reference.md`.

## Caching

Caching dependencies is the single highest-impact optimization. Use the cache plugin with manifest-based invalidation:

```yaml
steps:
  - label: ":nodejs: Test"
    command: "npm ci && npm test"
    plugins:
      - cache#v1.8.1:
          paths:
            - "node_modules/"
          manifest: "package-lock.json"
```

The cache key derives from the manifest file hash. When `package-lock.json` changes, the cache rebuilds.

**Hosted agents** also support a built-in `cache` key (no plugin needed):

```yaml
steps:
  - label: ":nodejs: Test"
    command: "npm ci && npm test"
    cache:
      paths:
        - "node_modules/"
      key: "v1-deps-{{ checksum 'package-lock.json' }}"
```

> Hosted agent setup and instance shapes are covered by the **buildkite-agent-infrastructure** skill.

## Fast-Fail and Non-Blocking Steps

Cancel remaining jobs immediately when any job fails:

```yaml
steps:
  - label: ":rspec: Tests"
    command: "bundle exec rspec"
    cancel_on_build_failing: true
```

Use `soft_fail` for steps that should not block the build (security scans, linting, coverage):

```yaml
steps:
  - label: ":shield: Security Scan"
    command: "scripts/security-scan.sh"
    soft_fail:
      - exit_status: 1
```

A soft-failed step shows as a warning in the UI but does not fail the build. Combine with `continue_on_failure: true` on a wait step to let downstream steps run regardless.

## Parallelism and Dependencies

### Parallel execution

Steps at the same level run in parallel by default. Use `parallelism` to fan out a single step:

```yaml
steps:
  - label: ":rspec: Tests %n"
    command: "bundle exec rspec"
    parallelism: 10
```

This creates 10 parallel jobs. Each receives `BUILDKITE_PARALLEL_JOB` (0-9) and `BUILDKITE_PARALLEL_JOB_COUNT` (10) as environment variables for splitting work.

> For intelligent test splitting based on timing data, see the **buildkite-test-engine** skill.

### Explicit dependencies

Use `depends_on` to express step-level dependencies without `wait`:

```yaml
steps:
  - label: "Build"
    key: "build"
    command: "make build"

  - label: "Unit Tests"
    depends_on: "build"
    command: "make test-unit"

  - label: "Integration Tests"
    depends_on: "build"
    command: "make test-integration"
```

Unit and integration tests run in parallel after build completes — no `wait` step needed.

## Annotations

Surface build results directly on the build page using `buildkite-agent annotate`. Supports Markdown and HTML.

```yaml
steps:
  - label: ":test_tube: Tests"
    command: |
      if ! make test 2>&1 | tee test-output.txt; then
        buildkite-agent annotate --style "error" --context "test-failures" < test-output.txt
        exit 1
      fi
      buildkite-agent annotate "All tests passed :white_check_mark:" --style "success" --context "test-results"
```

| Flag | Default | Description |
|------|---------|-------------|
| `--style` | `default` | Visual style: `default`, `info`, `warning`, `error`, `success` |
| `--context` | random | Unique ID — reusing a context replaces the annotation |
| `--append` | `false` | Append to existing annotation with same context |

Link to uploaded artifacts in annotations:

```yaml
- command: |
    buildkite-agent artifact upload "coverage/*"
    buildkite-agent annotate --style "info" 'Coverage: <a href="artifact://coverage/index.html">view report</a>'
```

## Retry

### Automatic retry

Retry transient failures by exit status:

```yaml
steps:
  - label: ":hammer: Build"
    command: "make build"
    retry:
      automatic:
        - exit_status: -1    # Agent lost
          limit: 2
        - exit_status: 143   # SIGTERM (spot instance termination)
          limit: 2
        - exit_status: 255   # Timeout or SSH failure
          limit: 2
        - exit_status: "*"   # Any non-zero exit
          limit: 1
```

### Manual retry

Control whether manual retries are allowed:

```yaml
retry:
  manual:
    allowed: false
    reason: "Deployment steps cannot be retried"
```

For comprehensive exit code tables and retry strategy recommendations, see `references/retry-and-error-codes.md`.

## Dynamic Pipelines

Generate pipeline steps at runtime based on repository state. Upload generated YAML with `buildkite-agent pipeline upload`:

```yaml
steps:
  - label: ":pipeline: Generate"
    command: |
      .buildkite/generate-pipeline.sh | buildkite-agent pipeline upload
```

### When to use what

Pipelines exist on a spectrum. Pick the simplest option that does the job:

| Situation | Approach |
|-----------|----------|
| Same steps every build, branch-level filtering at most | Static YAML |
| Org-wide enforcement of pipeline structure, admin-controlled (Enterprise plan) | Pipeline templates |
| Reusable, vetted logic (caching, Docker, artifact transfer) shared across many pipelines | Pinned plugin |
| Skip steps when specific files haven't changed | `if_changed` |
| Monorepo with separate pipelines per service | `monorepo-diff` plugin |
| Combine `if` and `if_changed` with OR logic | Dynamic generation |
| Apply consistent retry / timeout / env config across many pipelines | Dynamic (shared config) |
| Calculate test shards, matrix combos, `parallelism × matrix` at runtime | Dynamic (often SDK) |
| Monorepo with transitive dependencies between services | Dynamic (custom dep graph) |
| Recover from infra failures (OOM → bigger agent) | Dynamic (`pre-exit` hook) |
| Steps depend on output from previous steps (multi-stage) | Dynamic, often `--replace` or chained uploads |
| Cleanup / teardown step that must run regardless of earlier failures | Dynamic (`pre-exit` uploads a finalizer) |
| Fallback step only when the primary step fails | Dynamic (`pre-exit` checking exit status) |
| Pipeline YAML has outgrown what the team can maintain | Dynamic (SDK in Python / TS / Go / Ruby) |

### Don't reach for dynamic pipelines for the wrong job

Dynamic generation is the right tool when the *steps themselves* need to change. For passing data between steps, simpler primitives exist:

- **`buildkite-agent meta-data set/get`** — small key-value pairs any later step in the same build can read (a version string, a commit SHA, a feature fla