ai-llm-inference

What this skill does

# LLMOps - Inference & Optimization - Production Skill Hub

**Modern Best Practices (January 2026)**:

- Treat inference as a **systems problem**: SLOs, tail latency, retries, overload, and cache strategy.
- Use **continuous batching / smart scheduling** when serving many concurrent requests (Orca scheduling: https://www.usenix.org/conference/osdi22/presentation/yu).
- Use **KV-cache aware serving** (PagedAttention/vLLM: https://arxiv.org/abs/2309.06180) and **efficient attention kernels** (FlashAttention: https://arxiv.org/abs/2205.14135).
- Use **speculative decoding** when latency is critical and draft-model quality is acceptable (speculative decoding: https://arxiv.org/abs/2302.01318).
- Quantize only with **measured** quality impact and rollback plan (quantization must be validated on your eval set).

This skill provides **production-ready operational patterns** for optimizing LLM inference performance, cost, and reliability. It centralizes **decision rules**, **optimization strategies**, **configuration templates**, and **operational checklists** for inference workloads.

No theory. No narrative. Only what Codex can execute.

---

## When to Use This Skill

Codex should activate this skill whenever the user asks for:

- Optimizing LLM inference latency or throughput
- Choosing quantization strategies (FP8/FP4/INT8/INT4)
- Configuring vLLM, TensorRT-LLM, or DeepSpeed inference
- Scaling LLM inference across GPUs (tensor/pipeline parallelism)
- Building high-throughput LLM APIs
- Improving context window performance (KV cache optimization)
- Using speculative decoding for faster generation
- Reducing cost per token
- Profiling and benchmarking inference workloads
- Planning infrastructure capacity
- CPU/edge deployment patterns
- High availability and resilience patterns

## Scope Boundaries (Use These Skills for Depth)

- **Prompting, tuning, datasets** -> [ai-llm](../ai-llm/SKILL.md)
- **RAG pipeline construction** -> [ai-rag](../ai-rag/SKILL.md)
- **Deployment, APIs, monitoring** -> [ai-mlops](../ai-mlops/SKILL.md)
- **Safety, governance** -> [ai-mlops](../ai-mlops/SKILL.md)
- **Performance monitoring** -> [qa-observability](../qa-observability/SKILL.md)
- **Infrastructure operations** -> [ops-devops-platform](../ops-devops-platform/SKILL.md)

---

## Quick Reference

| Task | Tool/Framework | Command/Pattern | When to Use |
|------|----------------|-----------------|-------------|
| Latency budget | SLO + load model | TTFT/ITL + P95/P99 under load | Any production endpoint |
| Tail-latency control | Scheduling + timeouts | Admission control + queue caps + backpressure | Prevent p99 explosions |
| Throughput | Batching + KV-cache aware serving | Continuous batching + KV paging | High concurrency serving |
| Cost control | Model tiering + caching | Cache (prefix/response) + quotas | Reduce spend and overload risk |
| Long context | Prefill optimization | Chunked prefill + prompt compression | Long inputs and RAG-heavy apps |
| Parallelism | TP/PP/DP | Choose by model size and interconnect | Models that do not fit one device |
| Reliability | Resilience patterns | Timeouts + circuit breakers + idempotency | Avoid cascading failures |

---

## Decision Tree: Inference Optimization Strategy

```text
Need to optimize LLM inference: [Optimization Path]
    │
    ├─ High throughput (>10k tok/s) OR P99 variance > 3x P50?
    │   └─ YES -> Disaggregated inference (prefill/decode separation)
    │            See references/disaggregated-inference.md
    │
    ├─ Primary constraint: Throughput?
    │   ├─ Many concurrent users? -> batching + KV-cache aware serving + admission control
    │   ├─ Chat/agents with KV reuse? -> SGLang (RadixAttention)
    │   └─ Mostly batch/offline? -> batch inference jobs + large batches + spot capacity
    │
    ├─ Primary constraint: Cost?
    │   ├─ Can accept lower quality tier? -> model tiering (small/medium/large router)
    │   └─ Must keep quality? -> caching + prompt/context reduction before quantization
    │
    ├─ Primary constraint: Latency?
    │   ├─ Draft model acceptable? -> speculative decoding
    │   └─ Long context? -> prefill optimizations + FlashAttention-3 + context budgets
    │
    ├─ Large model (>70B)?
    │   ├─ Multiple GPUs? -> Tensor parallelism (NVLink required)
    │   └─ Deep model? -> Pipeline parallelism (minimize bubbles)
    │
    ├─ Hardware selection?
    │   ├─ Memory-bound? -> more HBM, higher bandwidth
    │   ├─ Latency-bound? -> faster clocks + kernel support
    │   └─ Multi-node? -> prioritize interconnect (NVLink/RDMA) and topology
    │
    │   Notes: treat GPU/SKU advice as time-sensitive; verify with vendor docs and your own benchmarks.
    │   See references/gpu-optimization-checklists.md and references/infrastructure-tuning.md
    │
    └─ Edge deployment?
        └─ CPU + quantization -> llama.cpp/GGUF for constrained resources
```

---

## Intake Checklist (REQUIRED)

Before recommending changes, collect (or infer) these inputs:

- Model + variant (size, context length, precision/quantization, tokenizer)
- Traffic shape (prompt/output length distributions, concurrency, QPS, streaming vs non-streaming)
- SLOs and budgets (TTFT/ITL/total latency targets, error budget, cost per request)
- Serving stack (engine/version, batching/scheduling settings, caching, parallelism, autoscaling)
- Hardware and topology (GPU type/count, VRAM, NVLink/RDMA, CPU/RAM, storage, cluster/runtime)
- Constraints (quality floor, safety requirements, rollout/rollback constraints)

## Core Concepts & Practices

### Core Concepts (Vendor-Agnostic)

- **Latency components**: queueing + prefill + decode; optimize the largest contributor first.
- **Tail latency**: p99 is dominated by queuing and long prompts; fix with admission control and context budgets.
- **Retries**: retries can multiply load; bound retries and use hedged requests only with strict budgets.
- **Caching**: prefix caching helps repeated system/tool scaffolds; response caching helps repeated questions (requires invalidation).
- **Security & privacy**: prompts/outputs can contain sensitive data; scrub logs, enforce auth/tenancy, and rate-limit abuse (OWASP LLM Top 10: https://owasp.org/www-project-top-10-for-large-language-model-applications/).

### Implementation Practices (Tooling Examples)

- **Measure under load**: benchmark TTFT/ITL and p95/p99 with realistic concurrency and prompt lengths.
- **Separate environments**: dev/stage/prod model configs; promote only after passing the inference review checklist.
- **Export telemetry**: request-level tokens, TTFT/ITL, queue depth, GPU memory headroom, and error classes (OpenTelemetry GenAI semantic conventions: https://opentelemetry.io/docs/specs/semconv/gen-ai/).

### Do / Avoid

**Do**
- Do enforce `max_input_tokens` and `max_output_tokens` at the API boundary.
- Do cap concurrency and queue depth; return overload errors quickly.
- Do validate quality after any quantization or kernel change.

**Avoid**
- Avoid unbounded retries (amplifies outages).
- Avoid unbounded context windows (OOM + latency spikes).
- Avoid benchmarking on single requests; always test with realistic concurrency.

---

## Accuracy Protocol (REQUIRED)

- Treat performance ratios (for example, "2x faster") as hypotheses unless a source is cited and the workload is comparable.
- Do not recommend hardware/SKU changes without stating assumptions (model size, context length, concurrency, interconnect).
- Prefer a measured baseline + checklist-driven rollout over "best practice" claims.

---

## Resources (Detailed Operational Guides)

For comprehensive guides on specific topics, see:

### Infrastructure & Serving

- [Disaggregated Inference](references/disaggregated-inference.md) - Prefill/decode separation (2025+ standard)
- [Infrastructure Tuning](references/infrastructure-tuning.md) - OS, container, Kubernetes optimization for GPU workloads
- [Serving Architectures](references/serving-architectures.md) - Producti