building-with-llms

What this skill does

# Building with LLMs

Help the user build effective AI applications using practical techniques from 60 product leaders and AI practitioners.

## How to Help

When the user asks for help building with LLMs:

1. **Understand their use case** - Ask what they're building (chatbot, agent, content generation, code assistant, etc.)
2. **Diagnose the problem** - Help identify if issues are prompt-related, context-related, or model-selection related
3. **Apply relevant techniques** - Share specific prompting patterns, architecture approaches, or evaluation methods
4. **Challenge common mistakes** - Push back on over-reliance on vibes, skipping evals, or using the wrong model for the task

## Core Principles

### Prompting

**Few-shot examples beat descriptions**
Sander Schulhoff: "If there's one technique I'd recommend, it's few-shot prompting—giving examples of what you want. Instead of describing your writing style, paste a few previous emails and say 'write like this.'"

**Provide your point of view**
Wes Kao: "Sharing my POV makes output way better. Don't just ask 'What would you say?' Tell it: 'I want to say no, but I'd like to preserve the relationship. Here's what I'd ideally do...'"

**Use decomposition for complex tasks**
Sander Schulhoff: "Ask 'What subproblems need solving first?' Get the list, solve each one, then synthesize. Don't ask the model to solve everything at once."

**Self-criticism improves output**
Sander Schulhoff: "Ask the LLM to check and critique its own response, then improve it. Models can catch their own errors when prompted to look."

**Roles help style, not accuracy**
Sander Schulhoff: "Roles like 'Act as a professor' don't help accuracy tasks. But they're great for controlling tone and style in creative work."

**Put context at the beginning**
Sander Schulhoff: "Place long context at the start of your prompt. It gets cached (cheaper), and the model won't forget its task when processing."

### Architecture

**Context engineering > prompt engineering**
Bret Taylor: "If a model makes a bad decision, it's usually lack of context. Fix it at the root—feed better data via MCP or RAG."

**RAG quality = data prep quality**
Chip Huyen: "The biggest gains come from data preparation, not vector database choice. Rewrite source data into Q&A format. Add annotations for context humans take for granted."

**Layer models for robustness**
Bret Taylor: "Having AI supervise AI is effective. Layer cognitive steps—one model generates, another reviews. This moves you from 90% to 99% accuracy."

**Use specialized models for specialized tasks**
Amjad Masad: "We use Claude Sonnet for coding, other models for critiquing. A 'society of models' with different roles outperforms one general model."

**200ms is the latency threshold**
Ryan J. Salva (GitHub Copilot): "The sweet spot for real-time suggestions is ~200ms. Slower feels like an interruption. Design your architecture around this constraint."

### Evaluation

**Evals are mandatory, not optional**
Kevin Weil (OpenAI): "Writing evals is becoming a core product skill. A 60% reliable model needs different UX than 95% or 99.5%. You can't design without knowing your accuracy."

**Binary scores > Likert scales**
Hamel Husain: "Force Pass/Fail, not 1-5 scores. Scales produce meaningless averages like '3.7'. Binary forces real decisions."

**Start with vibes, evolve to evals**
Howie Liu: "For novel products, start with open-ended vibes testing. Only move to formal evals once use cases converge."

**Validate your LLM judge**
Hamel Husain: "If using LLM-as-judge, you must eval the eval. Measure agreement with human experts. Iterate until it aligns."

### Building & Iteration

**Retry failures—models are stochastic**
Benjamin Mann (Anthropic): "If it fails, try the exact same prompt again. Success rates are much higher on retry than on banging on a broken approach."

**Be ambitious in your asks**
Benjamin Mann: "The difference between effective and ineffective Claude Code users: ambitious requests. Ask for the big change, not incremental tweaks."

**Cross-pollinate between models**
Guillermo Rauch: "When stuck after 100+ iterations, copy the code to a different model (e.g., from v0 to ChatGPT o1). Fresh perspective unblocks you."

**Compounding engineering**
Dan Shipper: "For every unit of work, make the next unit easier. Save prompts that work. Build a library. Your team's AI effectiveness compounds."

### Working with AI Tools

**Learn to read and debug, not memorize syntax**
Amjad Masad: "The ROI on coding doubles every 6 months because AI amplifies it. Focus on reading code and debugging—syntax is handled."

**Use chat mode to understand**
Anton Osika: "Use 'chat mode' to ask the AI to explain its logic. 'Why did you do this? What am I missing?' Treat it as a tutor."

**Vibe coding is a real skill**
Elena Verna: "I put vibe coding on my resume. Build functional prototypes with natural language before handing to engineering."

## Questions to Help Users

- "What are you building and what's the core user problem?"
- "What does the model get wrong most often?"
- "Are you measuring success systematically or going on vibes?"
- "What context does the model have access to?"
- "Have you tried few-shot examples?"
- "What happens when you retry failed prompts?"

## Common Mistakes to Flag

- **Vibes forever** - Eventually you need real evals, not just "it feels good"
- **Prompt-only thinking** - Often the fix is better context, not better prompts
- **One model for everything** - Different models excel at different tasks
- **Giving up after one failure** - Stochastic systems need retries
- **Skipping the human review** - AI output needs human validation, especially early on

## Deep Dive

For all 110 insights from 60 guests, see `references/guest-insights.md`

## Related Skills

- AI Product Strategy
- AI Evals
- Vibe Coding
- Evaluating New Technology