gallery-dl

What this skill does

# gallery-dl

## Overview

gallery-dl is a command-line tool for downloading image galleries and collections from over 100 websites — including Instagram, Twitter/X, Reddit, Pixiv, DeviantArt, Tumblr, Flickr, ArtStation, Imgur, and dozens of image boards. It handles authentication, pagination, metadata extraction, and flexible file naming. Think of it as yt-dlp but for images and galleries.

## Instructions

### Step 1: Installation

```bash
# pip (recommended)
pip install gallery-dl

# With optional dependencies for broader format support
pip install gallery-dl[yt-dlp]    # adds video download support via yt-dlp

# Standalone binary (Linux)
curl -L https://github.com/mikf/gallery-dl/releases/latest/download/gallery-dl.bin -o /usr/local/bin/gallery-dl
chmod +x /usr/local/bin/gallery-dl

# macOS
brew install gallery-dl

# Update
pip install -U gallery-dl

# Verify
gallery-dl --version
```

### Step 2: Basic Downloads

```bash
# Download from a gallery/post URL
gallery-dl "https://www.instagram.com/p/POST_ID/"
gallery-dl "https://twitter.com/user/status/123456789"
gallery-dl "https://www.reddit.com/r/EarthPorn/top/?t=month"
gallery-dl "https://www.pixiv.net/en/artworks/12345678"
gallery-dl "https://imgur.com/a/ALBUM_ID"

# Download entire user profile
gallery-dl "https://www.instagram.com/username/"
gallery-dl "https://twitter.com/username/media"
gallery-dl "https://www.deviantart.com/username"
gallery-dl "https://www.pixiv.net/en/users/USER_ID"

# Download with custom output directory
gallery-dl -d ./downloads "URL"

# Download with verbose output
gallery-dl -v "URL"
```

### Step 3: Output Templates and Organization

gallery-dl uses a powerful template system for organizing downloaded files into directories.

```bash
# Custom filename template
gallery-dl -o "filename={category}_{id}_{title}.{extension}" "URL"

# Organize by site and user
gallery-dl -o "directory=['{category}', '{subcategory}']" "URL"

# Custom directory per site
gallery-dl -o "base-directory=./media" -o "directory=['{category}', '{user}', '{date:%Y-%m}']" "URL"
```

Default directory structure: `./gallery-dl/{category}/{subcategory}/`

### Step 4: Authentication

Many sites require login for full access (age-restricted content, private profiles, higher rate limits).

```bash
# Username + password
gallery-dl -u "username" -p "password" "URL"

# Cookie-based authentication (recommended for most sites)
# Export cookies from your browser using a browser extension (e.g., "Get cookies.txt")
gallery-dl --cookies cookies.txt "URL"

# Browser cookie extraction (auto-extracts from installed browser)
gallery-dl --cookies-from-browser firefox "URL"
gallery-dl --cookies-from-browser chrome "URL"

# OAuth (for Tumblr, DeviantArt, Reddit, etc.)
gallery-dl oauth:reddit          # opens browser for OAuth flow
gallery-dl oauth:deviantart
gallery-dl oauth:tumblr
```

### Step 5: Filtering and Selection

```bash
# Download only images (skip videos)
gallery-dl --filter "extension in ('jpg', 'jpeg', 'png', 'gif', 'webp')" "URL"

# Download only videos
gallery-dl --filter "extension in ('mp4', 'webm', 'mov')" "URL"

# Minimum image dimensions
gallery-dl --filter "width >= 1920 and height >= 1080" "URL"

# Date range
gallery-dl --filter "date >= datetime(2024, 1, 1)" "URL"

# Limit number of downloads
gallery-dl --range "1-50" "URL"       # first 50 items
gallery-dl --range "10-20" "URL"      # items 10 through 20

# Skip files that already exist
gallery-dl --no-skip "URL"    # download even if exists (default: skip)
```

### Step 6: Configuration File

gallery-dl supports a JSON configuration file at `~/.config/gallery-dl/config.json` (or `~/.gallery-dl.conf`).

```json
{
  "extractor": {
    "base-directory": "./gallery-dl/",
    "archive": "~/.gallery-dl/archive.sqlite3",

    "instagram": {
      "cookies-from-browser": "firefox",
      "directory": ["instagram", "{user}"],
      "filename": "{date:%Y%m%d}_{shortcode}_{num:>02}.{extension}",
      "posts": {
        "include": ["image", "video", "sidecar"]
      }
    },

    "twitter": {
      "cookies-from-browser": "chrome",
      "directory": ["twitter", "{user[name]}"],
      "filename": "{date:%Y%m%d}_{tweet_id}_{num:>02}.{extension}",
      "retweets": false,
      "text-tweets": false
    },

    "reddit": {
      "directory": ["reddit", "{subreddit}", "{id}"],
      "comments": 0,
      "morecomments": false
    },

    "pixiv": {
      "directory": ["pixiv", "{user[name]}"],
      "filename": "{id}_p{num:>02}.{extension}",
      "ugoira": true
    }
  },

  "downloader": {
    "rate": "2M",
    "retries": 3,
    "timeout": 30.0,
    "part": true,
    "part-directory": "/tmp/.gallery-dl/"
  },

  "output": {
    "log": {
      "level": "info",
      "format": "{message}"
    }
  },

  "postprocessor": [
    {
      "name": "metadata",
      "mode": "json"
    }
  ]
}
```

### Step 7: Archive and Deduplication

The archive feature tracks downloaded files in a SQLite database, preventing re-downloads on subsequent runs.

```bash
# Use archive file (skip already downloaded)
gallery-dl --download-archive archive.sqlite3 "URL"

# Clear archive for specific URL
gallery-dl --clear-archive archive.sqlite3 "URL"
```

In config:
```json
{
  "extractor": {
    "archive": "~/.gallery-dl/archive.sqlite3"
  }
}
```

### Step 8: Post-Processing

gallery-dl can run post-processors on downloaded files — metadata export, conversion, classification.

```json
{
  "postprocessor": [
    {
      "name": "metadata",
      "mode": "json",
      "directory": "metadata/"
    },
    {
      "name": "exec",
      "command": ["convert", "{_path}", "-resize", "1920x1080>", "{_path}"]
    },
    {
      "name": "zip",
      "compression": "store",
      "extension": "cbz"
    }
  ]
}
```

### Step 9: Supported Sites (Partial List)

Major platforms supported by gallery-dl:

- **Social media:** Instagram, Twitter/X, Reddit, Tumblr, TikTok
- **Art platforms:** Pixiv, DeviantArt, ArtStation, Newgrounds
- **Image hosting:** Imgur, Flickr, SmugMug, Google Photos
- **Image boards:** Danbooru, Gelbooru, Safebooru, E621, Konachan
- **Manga/comics:** MangaDex, Webtoon, Tapas, Dynasty Scans
- **Stock/photography:** Unsplash, 500px
- **Other:** Patreon, Kemono, Cohost, Mastodon, Bluesky, Pillowfort

Full list: `gallery-dl --list-extractors`

### Step 10: Batch Downloads

```bash
# Download from URL list file
gallery-dl -i urls.txt

# Combine with rate limiting
gallery-dl -i urls.txt --rate-limit 2M --sleep 3-5

# Download with metadata export
gallery-dl -i urls.txt --write-metadata
```

## Examples

### Example 1: Archive an artist's complete portfolio from multiple platforms
**User prompt:** "Download all artwork from this artist — they post on Pixiv, Twitter, and DeviantArt. I want everything organized by platform with metadata saved."

The agent will:
1. Set up a config with per-platform directory organization: `{category}/{user}/{date:%Y-%m}/`
2. Enable archive tracking to avoid duplicates: `--download-archive artist_archive.sqlite3`
3. Run gallery-dl for each platform URL with metadata post-processor enabled.
4. The archive database ensures running the same command later only downloads new posts.
5. Report total files downloaded per platform and total disk usage.

### Example 2: Build an image dataset from Reddit
**User prompt:** "I need to collect high-resolution landscape photos from r/EarthPorn for a machine learning dataset. Only images above 1920x1080, from the last year, save metadata as JSON."

The agent will:
1. Configure gallery-dl with dimension filter: `--filter "width >= 1920 and height >= 1080"`
2. Add date filter for the last year and metadata post-processor.
3. Run: `gallery-dl --filter "width >= 1920 and height >= 1080" --download-archive dataset_archive.sqlite3 -o "postprocessor=[{'name': 'metadata', 'mode': 'json'}]" "https://www.reddit.com/r/EarthPorn/top/?t=year"`
4. The JSON metadata files include title, author, score, dimensions, and original URL — useful f