receipt-scanner-master

What this skill does

# Receipt Scanner Master

Master the receipt scanning system that uses AI-powered OCR to extract structured data from receipt images and store them in the database.

## What This Skill Does

This skill helps you:
1. Parse receipt images (JPG, PNG, WebP, PDF) into structured data
2. Debug OCR accuracy issues and extraction errors
3. Enhance the receipt parsing engine and prompts
4. Test receipt uploads through the web interface
5. Troubleshoot database integration issues
6. Validate extracted data against actual receipts
7. Improve categorization and line item extraction

## System Architecture

### Frontend Components

**Receipt Scanner Component**: `/home/adamsl/planner/office-assistant/js/components/receipt-scanner.js`
- Primary receipt scanning interface at `http://localhost:8080/receipt-scanner.html`
- Drag-and-drop or file upload for receipt images
- Parses receipts and displays line items in a table
- **Each line item has a category-picker dropdown**
- **Items auto-save to database immediately when categorized**
- Only categorized items are saved (uncategorized items ignored)
- No overall receipt-level category picker (removed)

**Upload Component**: `/home/adamsl/planner/office-assistant/js/upload-component.js`
- Alternative upload interface (bank statements)
- Displays recent downloads from the system
- Shows real-time processing feedback via terminal display
- Handles streaming responses from backend (Server-Sent Events)
- Auto-refreshes file list after successful imports

### Backend Components

**Receipt Parser**: `app/services/receipt_parser.py`
- Validates file types and sizes
- Processes and compresses images
- Manages temporary and permanent file storage
- Coordinates with AI engine for extraction

**Receipt Engine**: `app/services/receipt_engine.py`
- Uses Google Gemini AI for OCR and extraction
- Implements strict accuracy validation rules
- Returns structured data via Pydantic models
- **Tries models in order: gemini-2.5-flash (first), 2.0-flash, 2.5-pro, pro-latest**
- Flash model used first to avoid pro quota limits
- Separate quotas for flash vs pro models

**API Endpoints**: `app/api/receipt_endpoints.py`
- `/api/parse-receipt` - Uploads and parses receipt image (returns temp data, doesn't save)
- `/api/receipt-items` - **Auto-saves individual line items when categorized**
- `/api/save-receipt` - Final save for categorized items (batch operation)
- `/api/receipts/{expense_id}` - Retrieves receipt metadata
- `/api/receipts/file/{year}/{month}/{filename}` - Serves stored receipt files

**Data Models**: `app/models/receipt_models.py`
- `ReceiptExtractionResult` - Complete receipt data structure
- `ReceiptItem` - Individual line items with categorization
- `ReceiptTotals` - Subtotal, tax, tip, discount, total
- `ReceiptPartyInfo` - Merchant details
- `ReceiptMeta` - Parsing metadata and model info
- `PaymentMethod` - Enum: CASH, CARD, BANK, OTHER

### Database Integration

**Tables**:
- `expenses` - Main expense entries (amount, date, category, method)
- `receipt_metadata` - Parsing metadata (model, confidence, raw response)

**Storage Structure**:
```
app/data/receipts/
├── YYYY/
│   ├── MM/
│   │   ├── receipt_TIMESTAMP_filename.jpg
│   │   └── receipt_TIMESTAMP_filename.pdf
└── temp/
    └── temp_receipt_TIMESTAMP_filename.jpg
```

## How to Use This Skill

### Step 1: Test Receipt Parsing

Parse a receipt image to extract structured data:

```bash
# Start the API server if not running
python3 api_server.py

# Test with curl (from another terminal)
curl -X POST "http://localhost:8000/api/parse-receipt" \
  -F "file=@/path/to/receipt.jpg"
```

**Expected Response**:
```json
{
  "parsed_data": {
    "transaction_date": "2025-01-15",
    "payment_method": "CARD",
    "party": {
      "merchant_name": "Walmart",
      "merchant_phone": null,
      "merchant_address": "123 Main St",
      "store_location": "Store #1234"
    },
    "items": [
      {
        "description": "MILK WHOLE GAL",
        "quantity": 1.0,
        "unit_price": 4.99,
        "line_total": 4.99
      }
    ],
    "totals": {
      "subtotal": 4.99,
      "tax_amount": 0.35,
      "tip_amount": 0.0,
      "discount_amount": 0.0,
      "total_amount": 5.34
    },
    "meta": {
      "currency": "USD",
      "receipt_number": "12345",
      "model_name": "gemini-2.5-pro"
    }
  },
  "temp_file_name": "temp_receipt_20250115T120000Z_receipt.jpg"
}
```

### Step 2: Debug OCR Accuracy Issues

When OCR produces incorrect amounts or descriptions:

**Common Issues**:
1. **Digit Confusion**: 4↔9, 3↔8, 5↔6, 0↔8, 1↔7
2. **Missing Items**: Items not extracted from receipt
3. **Wrong Totals**: Extracted amounts don't match
4. **Poor Image Quality**: Blurry, dark, or low-resolution images

**Debug Process**:

1. **Check the raw image quality**:
   ```bash
   # View the receipt image
   open /path/to/receipt.jpg
   # or
   xdg-open /path/to/receipt.jpg
   ```
   - Is text clearly readable?
   - Is image properly oriented?
   - Is there sufficient contrast?

2. **Review the Gemini prompt** in `app/services/receipt_engine.py:96-173`:
   - Look for the accuracy rules and verification steps
   - Check if new issue types need specific instructions
   - Verify digit confusion prevention rules are clear

3. **Test with higher quality image**:
   - Increase `RECEIPT_IMAGE_MAX_WIDTH_PX` in settings
   - Increase JPEG quality in `receipt_parser.py:80,83`

4. **Add validation logic**:
   - Check `quantity × unit_price = line_total` for each item
   - Verify `sum(line_totals) ≈ subtotal`
   - Compare `subtotal + tax - discount = total`

5. **Examine the raw AI response**:
   ```python
   # Add debug logging in receipt_engine.py:78
   print(f"Raw Gemini Response: {json_response}")
   ```

### Step 3: Enhance the Receipt Parser

To improve parsing accuracy and features:

**Modify the Gemini Prompt** (`app/services/receipt_engine.py`):

```python
def _get_prompt(self) -> str:
    return """
    You are an expert at extracting structured data from receipt images with EXTREME ACCURACY.

    [Add new instructions here, such as:]

    **NEW RULE**: For grocery store receipts, items often have:
    - Short codes (e.g., "VEG", "DAIRY", "MEAT")
    - Weight-based pricing (price per lb/kg)
    - Multi-buy discounts (e.g., "2 for $5")

    **VALIDATION ENHANCEMENT**: Before returning JSON:
    1. Verify every item's math: quantity × unit_price = line_total
    2. Sum all line_totals and compare to subtotal
    3. Check: subtotal + tax - discount + tip = total_amount
    4. If any validation fails, RE-EXAMINE the receipt more carefully

    ... [rest of prompt]
    """
```

**Improve Image Processing** (`app/services/receipt_parser.py`):

```python
async def _process_image(self, image_data: bytes, mime_type: str):
    if mime_type.startswith("image/"):
        img = Image.open(BytesIO(image_data))

        # Add preprocessing steps:
        # 1. Auto-rotate based on EXIF
        # 2. Increase contrast for faded receipts
        # 3. Sharpen slightly for better OCR
        # 4. Convert to grayscale if color isn't needed
```

**Add Custom Validation** (`app/api/receipt_endpoints.py`):

```python
@router.post("/parse-receipt")
async def parse_receipt_endpoint(file: UploadFile = File(...)):
    parsed_data, temp_file_name = await parser.process_receipt(file)

    # Add validation here:
    validation_errors = validate_receipt_data(parsed_data)
    if validation_errors:
        return JSONResponse(
            status_code=422,
            content={
                "errors": validation_errors,
                "parsed_data": parsed_data,
                "temp_file_name": temp_file_name
            }
        )

    return ParseReceiptResponse(...)
```

### Step 4: Test Through Web Interface

Test the complete workflow including UI:

1. **Start the API server**:
   ```bash
   cd /home/adamsl/planner/nonprofit_finance_db
   python3 api_server.py
   ```

2. **Open the web interface**:
   ```bash
   cd /home/a