dataset-curator

What this skill does

# Dataset Curator

## Overview
This skill covers the full lifecycle of dataset creation and curation for machine learning and LLM tasks. It addresses dataset schema design, data collection strategies, quality filtering, deduplication, class imbalance mitigation, stratified train/val/test splits, annotation guideline writing, and dataset card documentation. Good datasets are the foundation of reliable models — this skill helps teams avoid the most common data quality pitfalls that lead to poor generalization, evaluation leakage, and biased models.

## When to Use
- Designing a new dataset schema for a classification, extraction, or generation task
- Cleaning and deduplicating a raw dataset before model training
- Planning annotation guidelines for human labelers or LLM-assisted labeling
- Addressing class imbalance in a training set (oversampling, undersampling, weighting)
- Creating stratified train/val/test splits without leakage between splits
- Writing a dataset card (model card for data) for reproducibility and documentation
- Auditing an existing dataset for quality, coverage, and potential biases
- Combining multiple data sources into a single unified dataset

## When NOT to Use
- Training or fine-tuning a model (use model training skills)
- Running SQL or analytical queries on a production database (use data analysis skills)
- Building real-time data pipelines or streaming ETL (use data engineering skills)
- Designing evaluation suites for deployed LLMs (use eval-designer skill)
- Web scraping or data collection from APIs (use data collection skills)

## Quick Reference
| Task | Approach |
|------|----------|
| Define dataset schema | List fields, types, required vs optional, allowed values, and examples |
| Remove duplicates | Hash-based exact dedup + MinHash/LSH for near-duplicate detection |
| Fix class imbalance | Oversample minority (SMOTE) or undersample majority; adjust loss weights |
| Create train/val/test splits | Stratified split by label; ensure no overlap of entities across splits |
| Document the dataset | Write a dataset card with provenance, schema, statistics, and limitations |
| Validate annotation quality | Compute inter-annotator agreement (Cohen's kappa or Krippendorff's alpha) |
| Handle missing values | Decide per-field: impute, drop row, or add "unknown" category |
| Detect label noise | Use confident learning (cleanlab) or cross-validation outlier detection |

## Instructions

1. **Define the task and schema** — Before collecting any data, write the schema: every field name, data type, allowed values, and whether it is required. For classification datasets, enumerate all valid labels and their definitions. Ambiguous schemas cause inconsistent annotations and training failures.

2. **Establish collection strategy** — Determine the data source: human-annotated, LLM-generated, web-scraped, synthetic, or a combination. Document collection date, source URLs, licenses, and any sampling decisions. Ensure the collection covers the full input distribution the model will encounter in production.

3. **Write annotation guidelines** — Create a guideline document for labelers that defines every label, provides positive and negative examples for each, and includes decision rules for edge cases. Pilot the guidelines with 2–3 annotators on a sample of 50 items and iterate before full annotation begins.

4. **Run quality filtering** — Remove items that are too short, too long, contain encoding errors, are in the wrong language, or fail domain-specific quality checks. Log how many items were removed at each filter step and why. Preserve a raw snapshot before filtering.

5. **Deduplicate the dataset** — Apply exact deduplication first (hash the text or key fields). Then apply near-duplicate detection using MinHash + LSH (e.g., `datasketch` library) or sentence embedding cosine similarity. Aim to remove items with >80% overlap. Keep the highest-quality copy when deduplicating.

6. **Assess and address class imbalance** — Compute class distribution. If any class has less than 5% of the majority class count, consider: (a) collecting more data for minority classes, (b) oversampling with augmentation, (c) applying class weights in the loss function, or (d) using stratified sampling. Document the chosen approach and its rationale.

7. **Create stratified splits** — Split data into train/val/test sets preserving class distribution in each split. Use an 80/10/10 or 70/15/15 ratio as a starting point. For datasets with identifiable entities (users, documents, companies), ensure the same entity never appears in multiple splits to prevent leakage. Test set must remain untouched until final evaluation.

8. **Measure annotation quality** — For human-labeled data, compute inter-annotator agreement on a sample (at least 10% of the dataset labeled by 2+ annotators). Cohen's kappa > 0.7 is acceptable; < 0.6 indicates guideline issues. Resolve disagreements through adjudication, not random selection.

9. **Audit for biases and coverage gaps** — Analyze the dataset across demographic attributes, time periods, domains, and edge cases. Identify over-represented and under-represented slices. Document known limitations in the dataset card. If possible, compare distribution to real production data.

10. **Write the dataset card** — Document: dataset name, version, task, schema, collection methodology, source licenses, size (rows and tokens), class distribution, split sizes, known limitations and biases, and citation. Use the Hugging Face dataset card format for compatibility with the ecosystem.

## Examples

### Example 1: Text Classification Dataset Design for Customer Intent Detection
**Input:** A support team wants to train a model to classify customer emails into 8 intent categories: billing, refund, technical-issue, account-access, product-question, cancellation, complaint, and other.

**Output:**

**Schema definition:**
```json
{
  "id": "string (UUID)",
  "text": "string (customer email body, min 10 chars, max 2000 chars)",
  "label": "enum: [billing, refund, technical-issue, account-access, product-question, cancellation, complaint, other]",
  "confidence": "float 0.0–1.0 (annotator confidence)",
  "annotator_id": "string",
  "annotation_date": "ISO 8601 date",
  "source": "enum: [zendesk, email-archive, synthetic]"
}
```

**Collection plan:**
- Export 5,000 historical support tickets from Zendesk (past 12 months)
- Strip PII: replace names with `[NAME]`, emails with `[EMAIL]`, account numbers with `[ACCT]`
- Target distribution: at least 300 examples per class (aim for balanced; use augmentation for rare classes)

**Annotation guidelines excerpt:**
```
Label: "refund"
Definition: Customer is requesting money back for a purchase.
Positive examples:
  - "I'd like a refund for my order from last week"
  - "Please return the charge to my credit card"
Negative examples (do NOT label as refund):
  - "I want to exchange my item" → label as product-question
  - "Why was I charged twice?" → label as billing
Edge case: "I want to cancel and get my money back" → label as cancellation (intent is to cancel)
```

**Deduplication approach:**
```python
from datasketch import MinHash, MinHashLSH

def get_minhash(text, num_perm=128):
    m = MinHash(num_perm=num_perm)
    for word in text.lower().split():
        m.update(word.encode('utf8'))
    return m

lsh = MinHashLSH(threshold=0.8, num_perm=128)
# Add items and query for near-duplicates; keep only one per cluster
```

**Split strategy:**
```
Total: 5,200 items (after dedup and filtering)
Train: 4,160 (80%) — stratified by label
Val:   520  (10%) — stratified by label
Test:  520  (10%) — stratified by label; set aside immediately, not used for development
```

### Example 2: Image Dataset Curation Plan for Defect Detection
**Input:** A manufacturing team wants to train a binary classifier to detect product defects from camera images. They have 10,000 raw images, but only ~400 are defective.

**Output:**

**Schem