experiment-design
Design hypothesis-driven experiments and A/B tests with proper methodology. Use when asked to design an A/B test, validate a hypothesis, plan an experiment, or set up a test for a product change. Covers hypothesis writing, sample size, and common mistakes.
What this skill does
Design experiments that actually prove something. Most A/B tests fail because they test vague ideas, run too short, or peek at results. A well-designed experiment has a clear hypothesis, adequate power, and a pre-committed analysis plan. ## Hypothesis Template Every experiment starts with a written hypothesis before any work begins: **"If we [make this specific change] for [this audience], then [this metric] will [change in this direction] by [this amount], because [this reason based on evidence]."** Example: > "If we replace the 5-step onboarding wizard with a single guided first-project flow for new signups, then 7-day activation rate will increase from 23% to 35%, because 4/6 interviewed users said they wanted to 'just start using it' not 'set everything up first.'" Every part matters: - **Specific change:** Not "improve onboarding" — the exact change - **Audience:** Who sees this? New users only? Free tier only? - **Metric + direction + amount:** A number you'll measure - **Because:** The evidence-based reason. No evidence = no experiment. ## Experiment Design ### 1. Primary Metric One metric the experiment is designed to move. Not three. One. Additional metrics are guardrails. ### 2. Guardrail Metrics Metrics that must NOT degrade. These prevent "winning" by breaking something else. ### 3. Sample Size Calculate BEFORE running. Use a sample size calculator with: - Baseline conversion rate (current number) - Minimum detectable effect (smallest change worth caring about) - Statistical significance (95% is standard) - Power (80% minimum) If you need 50,000 users and you get 500/week, the experiment will take 100 weeks. Either increase the MDE or don't run the experiment. ### 4. Duration Run for at least one full business cycle (usually 1-2 weeks minimum) to capture day-of-week effects. NEVER run less than 7 days. ### 5. Analysis Plan Write BEFORE launching: what metric, what threshold, what you'll do if it wins/loses/is inconclusive. Pre-commit to avoid post-hoc storytelling. ## Common Mistakes - **Peeking:** Checking results daily and stopping when you see significance. This inflates false positive rates. Set a duration and don't peek until it's done. - **Underpowered tests:** Running a test with too few users to detect a meaningful effect. The result is "inconclusive" which teaches you nothing. - **Testing too many things:** Changing 5 things at once means you don't know what worked. One change per experiment. - **No guardrails:** Increasing signups by removing the password field "works" but breaks everything else. - **Post-hoc storytelling:** The test was inconclusive but "we noticed an interesting trend in the 25-34 age group." That's noise, not signal. ## Guidelines - CRITICAL: Write the hypothesis and analysis plan BEFORE building anything. If you can't state what you're testing and how you'll decide, you're not ready. - NEVER peek at results before the pre-determined end date. Statistical significance at day 3 of a 14-day test is meaningless. - NEVER run an experiment without calculating required sample size first. - ALWAYS include guardrail metrics. Winning the primary metric while tanking retention is not a win. - NEVER test more than one major change per experiment. Isolate variables. - ALWAYS document learnings from inconclusive tests. "We couldn't detect an effect with N=5,000 users" is still useful information — it suggests the effect is small or nonexistent. --- *Built on controlled experimentation methodology (Kohavi, Tang, Xu). Skills from [productskills](https://github.com/assimovt/productskills).*
Related in Design
contribute
IncludedLocal-only OSS contribution command center. Auto-refreshes the user's in-flight PR and issue state on invoke so conversations start with full context — no need to brief Claude on what's in flight. Helps the user find issues to contribute to on GitHub, builds per-repo dossiers of what each upstream expects (CLA, DCO, branch convention, AI policy, draft-first, review bots, issue templates), runs deterministic gates before any external action so AI-assisted contributions don't reach maintainers as slop. State is markdown-only: candidate files at ~/.contribute-system/candidates/, repo dossiers at ~/.contribute-system/research/, append-only event log at ~/.contribute-system/log.jsonl. No database, no cloud calls. Use when the user asks about their PRs / issues / contributions, wants to find new work to take on, claim an issue, build/refresh a repo's dossier, or draft a Design Issue or PR. Trigger with "/contribute", "what's my PR status", "find a contribution", "claim issue X", "draft a Design Issue for Y", "refresh dossier for Z".
architectural-analysis
IncludedUser-triggered deep architectural analysis of a codebase or scoped subtree across eight modes — information architecture, data flow, integration points, UI surfaces, interaction patterns, data model, control flow, and failure modes. This skill should be used when the user asks to "diagram this codebase," "map the architecture," "show the data flow," "give me an ERD," "trace control flow," "find the integration points," "verify the layout pattern," "audit the UX architecture," or any similar request whose primary deliverable is mermaid diagrams plus cited reports under docs/architecture/. Dispatches haiku/sonnet sub-agents in parallel for per-mode exploration, then verifies every citation mechanically before any node lands in a diagram. Not for one-off prose explanations of code (use code-explanation) or for high-level system design from scratch (use system-design).
mcp
IncludedModel Context Protocol (MCP) server development and tool management. Languages: Python, TypeScript. Capabilities: build MCP servers, integrate external APIs, discover/execute MCP tools, manage multi-server configs, design agent-centric tools. Actions: create, build, integrate, discover, execute, configure MCP servers/tools. Keywords: MCP, Model Context Protocol, MCP server, MCP tool, stdio transport, SSE transport, tool discovery, resource provider, prompt template, external API integration, Gemini CLI MCP, Claude MCP, agent tools, tool execution, server config. Use when: building MCP servers, integrating external APIs as MCP tools, discovering available MCP tools, executing MCP capabilities, configuring multi-server setups, designing tools for AI agents.
react-native-skia
IncludedDesign, build, debug, and optimise high-polish animated graphics in React Native or Expo using @shopify/react-native-skia, Reanimated, and Gesture Handler. Use when the user wants canvas-driven UI, shaders, paths, rich text, image filters, sprite fields, Skottie, video frames, snapshots, web CanvasKit setup, or performance tuning for custom motion-heavy elements such as loaders, hero art, cards, charts, progress indicators, particle systems, or gesture-driven surfaces. Also use when the user asks for fluid, glow, glass, blob, parallax, 60fps/120fps, or GPU-friendly animated effects in React Native, even if they do not explicitly say "Skia". Do not use for ordinary form/layout work with standard views.
plaid
IncludedProduct Led AI Development — guides founders from idea to launched product. Six capabilities: Idea (discover a product idea), Validate (pressure-test the idea against fatal flaws, problem reality, competition, and 2-week MVP feasibility), Plan (vision intake + document generation), Design (translate image references into a design.md spec), Launch (go-to-market strategy), and Build (roadmap execution). Use when someone says "PLAID", "plaid idea", "help me find an idea", "product idea", "idea from my business", "idea from my expertise", "plaid validate", "validate my idea", "pressure-test", "is this idea good", "find fatal flaws", "validate the problem", "plan a product", "define my vision", "generate a PRD", "product strategy", "plaid design", "design from image", "translate image to design", "create design.md", "extract design tokens", "plaid launch", "go-to-market", "launch plan", "GTM strategy", "launch playbook", "plaid build", "build the app", "start building", or "execute the roadmap".
nextjs-framer-motion-animations
IncludedAdds production-safe Motion for React or Framer Motion animations to Next.js apps, including reveal, hover and tap micro-interactions, whileInView, stagger, AnimatePresence, layout and layoutId transitions, reorder, scroll-linked UI, and lightweight route-content transitions. Use when the user asks to add, refactor, or debug Motion or Framer Motion in App Router or Pages Router codebases, especially around server/client boundaries, reduced motion, LazyMotion, bundle size, hydration, or route transitions. Avoid for GSAP-style timelines, WebGL or 3D scenes, heavy scroll storytelling, or CSS-only effects unless Motion is explicitly requested.