playwright-web-scraper

What this skill does

# Playwright Web Scraper

Extract structured data from multiple web pages with respectful, ethical crawling practices.

## When to Use This Skill

Use when extracting structured data from websites with "scrape data from", "extract information from pages", "collect data from site", or "crawl multiple pages".

Do NOT use for testing workflows (use `playwright-e2e-testing`), monitoring errors (use `playwright-console-monitor`), or analyzing network (use `playwright-network-analyzer`). Always respect robots.txt and rate limits.

## Quick Start

Scrape product listings from an e-commerce site:

```javascript
// 1. Validate URLs
python scripts/validate_urls.py urls.txt

// 2. Scrape pages with rate limiting
const results = [];
for (const url of urls) {
  await browser_navigate({ url });
  await browser_wait_for({ time: Math.random() * 2 + 1 }); // 1-3s delay

  const data = await browser_evaluate({
    function: `
      Array.from(document.querySelectorAll('.product')).map(el => ({
        title: el.querySelector('.title')?.textContent?.trim(),
        price: el.querySelector('.price')?.textContent?.trim(),
        url: el.querySelector('a')?.getAttribute('href')
      }))
    `
  });

  results.push(...data);
}

// 3. Process results
python scripts/process_results.py scraped.json -o products.csv
```

## Table of Contents

1. Core Workflow
2. Rate Limiting Strategy
3. URL Validation
4. Data Extraction
5. Error Handling
6. Processing Results
7. Supporting Files
8. Expected Outcomes

## Core Workflow

### Step 1: Prepare URL List

Create a text file with URLs to scrape (one per line):

```
https://example.com/products?page=1
https://example.com/products?page=2
https://example.com/products?page=3
```

Validate URLs and check robots.txt compliance:

```bash
python scripts/validate_urls.py urls.txt --user-agent "MyBot/1.0"
```

### Step 2: Initialize Scraping Session

Navigate to the site and take a snapshot to understand structure:

```javascript
await browser_navigate({ url: firstUrl });
await browser_snapshot();
```

Identify CSS selectors for data extraction using the snapshot.

### Step 3: Implement Rate-Limited Crawling

Use random delays between requests (1-3 seconds minimum):

```javascript
const results = [];

for (const url of urlList) {
  // Navigate to page
  await browser_navigate({ url });

  // Wait for content to load
  await browser_wait_for({ text: 'Expected content marker' });

  // Add respectful delay (1-3 seconds)
  const delay = Math.random() * 2 + 1;
  await browser_wait_for({ time: delay });

  // Extract data
  const pageData = await browser_evaluate({
    function: `/* extraction code */`
  });

  results.push(...pageData);

  // Check console for errors/warnings
  const console = await browser_console_messages();
  // Monitor for rate limit warnings
}
```

### Step 4: Extract Structured Data

Use `browser_evaluate` to extract data with JavaScript:

```javascript
const data = await browser_evaluate({
  function: `
    try {
      return Array.from(document.querySelectorAll('.item')).map(el => ({
        title: el.querySelector('.title')?.textContent?.trim(),
        price: el.querySelector('.price')?.textContent?.trim(),
        rating: el.querySelector('.rating')?.textContent?.trim(),
        url: el.querySelector('a')?.getAttribute('href')
      })).filter(item => item.title && item.price); // Filter incomplete records
    } catch (e) {
      console.error('Extraction failed:', e);
      return [];
    }
  `
});
```

See `references/extraction-patterns.md` for comprehensive extraction patterns.

### Step 5: Handle Errors and Rate Limits

Monitor for rate limiting indicators:

```javascript
// Check HTTP responses via browser_network_requests
const requests = await browser_network_requests();
const rateLimited = requests.some(r => r.status === 429 || r.status === 503);

if (rateLimited) {
  // Back off exponentially
  await browser_wait_for({ time: 10 }); // Wait 10 seconds
  // Retry or skip
}

// Check console for blocking messages
const console = await browser_console_messages({ pattern: 'rate limit|blocked|captcha' });
if (console.length > 0) {
  // Handle blocking
}
```

### Step 6: Aggregate and Store Results

Save results to JSON file:

```javascript
// In your scraping script
fs.writeFileSync('scraped.json', JSON.stringify({ results }, null, 2));
```

Process and convert to desired format:

```bash
# View statistics
python scripts/process_results.py scraped.json --stats

# Convert to CSV
python scripts/process_results.py scraped.json -o output.csv

# Convert to Markdown table
python scripts/process_results.py scraped.json -o output.md
```

## Rate Limiting Strategy

### Minimum Delays

Always add delays between requests:

- **Standard sites**: 1-3 seconds (random)
- **High-traffic sites**: 3-5 seconds
- **Small sites**: 5-10 seconds
- **After errors**: Exponential backoff (5s, 10s, 20s, 40s)

### Implementation

```javascript
// Random delay between 1-3 seconds
const randomDelay = () => Math.random() * 2 + 1;
await browser_wait_for({ time: randomDelay() });

// Exponential backoff after rate limit
let backoffSeconds = 5;
for (let retry = 0; retry < 3; retry++) {
  try {
    await browser_navigate({ url });
    break; // Success
  } catch (e) {
    await browser_wait_for({ time: backoffSeconds });
    backoffSeconds *= 2; // Double delay each retry
  }
}
```

### Adaptive Rate Limiting

Adjust delays based on response:

| Response Code | Action |
|---------------|--------|
| 200 OK | Continue with normal delay (1-3s) |
| 429 Too Many Requests | Increase delay to 10s, retry |
| 503 Service Unavailable | Wait 60s, then retry |
| 403 Forbidden | Stop scraping this domain |

See `references/ethical-scraping.md` for detailed rate limiting strategies.

## URL Validation

Use `validate_urls.py` before scraping to ensure compliance:

```bash
# Basic validation
python scripts/validate_urls.py urls.txt

# Check robots.txt with specific user agent
python scripts/validate_urls.py urls.txt --user-agent "MyBot/1.0"

# Strict mode (exit on any invalid/disallowed URL)
python scripts/validate_urls.py urls.txt --strict
```

**Output includes**:

- URL format validation
- Domain grouping
- robots.txt compliance check
- Summary statistics

## Data Extraction

### Basic Pattern

```javascript
// Single page extraction
const data = await browser_evaluate({
  function: `
    Array.from(document.querySelectorAll('.item')).map(el => ({
      field1: el.querySelector('.selector1')?.textContent?.trim(),
      field2: el.querySelector('.selector2')?.getAttribute('href')
    }))
  `
});
```

### Pagination Pattern

```javascript
let hasMore = true;
let page = 1;

while (hasMore) {
  await browser_navigate({ url: `${baseUrl}?page=${page}` });
  await browser_wait_for({ time: randomDelay() });

  const pageData = await browser_evaluate({ function: extractionCode });
  results.push(...pageData);

  // Check for next page
  hasMore = await browser_evaluate({
    function: `document.querySelector('.next:not(.disabled)') !== null`
  });

  page++;
}
```

See `references/extraction-patterns.md` for:

- Advanced selectors
- Data cleaning patterns
- Table extraction
- JSON-LD extraction
- Shadow DOM access

## Error Handling

### Network Errors

```javascript
try {
  await browser_navigate({ url });
} catch (e) {
  console.error(`Failed to load ${url}:`, e);
  failedUrls.push(url);
  continue; // Skip to next URL
}
```

### Content Validation

```javascript
const data = await browser_evaluate({ function: extractionCode });

if (!data || data.length === 0) {
  console.warn(`No data extracted from ${url}`);
  // Log for manual review
}

// Validate data structure
const validData = data.filter(item =>
  item.title && item.price // Ensure required fields exist
);
```

### Monitoring Indicators

Check for blocking/errors:

```javascript
// Monitor console
const console = await browser_console_messages({
  pattern: 'error|rate|limit|captcha',
  onlyErrors: true
});