detection-sigma

What this skill does

# Sigma Detection Engineering

## Overview

Sigma is to log detection what Snort is to network traffic and YARA is to files - a universal signature format for describing security-relevant log events. This skill helps create, validate, and convert Sigma rules for deployment across multiple SIEM platforms, enabling detection-as-code workflows.

**Core capabilities**:
- Create detection rules using Sigma format
- Convert rules to 25+ SIEM/EDR backends (Splunk, Elastic, QRadar, Sentinel, etc.)
- Validate rule syntax and logic
- Map detections to MITRE ATT&CK framework
- Build threat hunting queries
- Implement compliance-based monitoring

## Quick Start

### Install Dependencies

```bash
pip install pysigma pysigma-backend-splunk pysigma-backend-elasticsearch pyyaml
```

### Create a Basic Sigma Rule

```yaml
title: Suspicious PowerShell Execution
id: 7d6d30b8-5b91-4b90-a71e-4f5a3f5a3c3f
status: experimental
description: Detects suspicious PowerShell execution with encoded commands
references:
    - https://attack.mitre.org/techniques/T1059/001/
author: Your Name
date: YYYY/MM/DD
modified: YYYY/MM/DD
tags:
    - attack.execution
    - attack.t1059.001
logsource:
    category: process_creation
    product: windows
detection:
    selection:
        Image|endswith: '\powershell.exe'
        CommandLine|contains:
            - '-enc'
            - '-EncodedCommand'
            - 'FromBase64String'
    condition: selection
falsepositives:
    - Legitimate administrative scripts
level: medium
```

### Convert Rule to Target SIEM

```bash
# Convert to Splunk
python scripts/sigma_convert.py rule.yml --backend splunk

# Convert to Elasticsearch
python scripts/sigma_convert.py rule.yml --backend elasticsearch

# Convert to Microsoft Sentinel
python scripts/sigma_convert.py rule.yml --backend sentinel
```

## Core Workflows

### Workflow 1: Detection Rule Development

Progress:
[ ] 1. Identify detection requirement from threat intelligence or compliance
[ ] 2. Research log sources and field mappings for target environment
[ ] 3. Create Sigma rule using standard template
[ ] 4. Validate rule syntax: `python scripts/sigma_validate.py rule.yml`
[ ] 5. Test rule against sample logs or historical data
[ ] 6. Convert to target SIEM format
[ ] 7. Deploy and tune based on false positive rate
[ ] 8. Document rule metadata and MITRE ATT&CK mapping

Work through each step systematically. Check off completed items.

### Workflow 2: Threat Hunting Rule Creation

For proactive threat hunting based on TTPs:

1. **Select MITRE ATT&CK Technique**
   - Review threat intelligence for relevant TTPs
   - Identify technique ID (e.g., T1059.001 - PowerShell)
   - See [references/mitre-attack-mapping.md](references/mitre-attack-mapping.md) for common techniques

2. **Identify Log Sources**
   - Determine which logs capture the technique
   - Map log source categories (process_creation, network_connection, file_event)
   - Verify log source availability in your environment

3. **Define Detection Logic**
   - Create selection criteria matching suspicious patterns
   - Add filters to reduce false positives
   - Use field modifiers for robust matching (endswith, contains, re)

4. **Validate and Test**
   - Run validation: `python scripts/sigma_validate.py hunting-rule.yml`
   - Test against known-good and known-bad samples
   - Tune detection logic based on results

5. **Document and Deploy**
   - Add references to threat reports
   - Document false positive scenarios
   - Convert and deploy to production SIEM

### Workflow 3: Bulk Rule Conversion

When migrating between SIEM platforms:

```bash
# Validate all rules first
python scripts/sigma_validate.py --directory rules/ --report validation-report.json

# Convert entire rule set
python scripts/sigma_convert.py --directory rules/ --backend splunk --output converted/

# Generate deployment report
python scripts/sigma_convert.py --directory rules/ --backend splunk --report conversion-report.md
```

Review conversion report for:
- Successfully converted rules
- Rules requiring manual adjustment
- Unsupported field mappings
- Backend-specific limitations

### Workflow 4: Compliance-Based Detection

For implementing compliance monitoring (PCI-DSS, NIST, ISO 27001):

1. **Map Requirements to Detections**
   - Identify compliance control requirements
   - Determine required log monitoring
   - See [references/compliance-mappings.md](references/compliance-mappings.md)

2. **Create Detection Rules**
   - Use compliance rule templates from `assets/compliance-rules/`
   - Tag rules with compliance framework (e.g., tags: [pci-dss.10.2.5])
   - Set appropriate severity levels

3. **Validate Coverage**
   - Run: `python scripts/compliance_coverage.py --framework pci-dss`
   - Review coverage gaps
   - Create additional rules as needed

4. **Generate Compliance Report**
   - Document detection coverage by control
   - Include sample queries and expected alerts
   - Maintain audit trail for compliance evidence

## Rule Structure Reference

### Required Fields

```yaml
title: Human-readable rule name
id: UUID (generate with: python -c "import uuid; print(uuid.uuid4())")
status: stable|test|experimental|deprecated
description: Detailed description of what this detects
author: Your Name
date: YYYY/MM/DD
modified: YYYY/MM/DD
logsource:
    category: process_creation|network_connection|file_event|...
    product: windows|linux|macos|azure|aws|...
detection:
    selection:
        FieldName: value
    condition: selection
level: informational|low|medium|high|critical
```

### Optional Fields

```yaml
references:
    - https://attack.mitre.org/techniques/T1059/
tags:
    - attack.execution
    - attack.t1059.001
falsepositives:
    - Legitimate use cases
fields:
    - CommandLine
    - User
    - ParentImage
```

### Detection Conditions

```yaml
# Simple selection
detection:
    selection:
        Field: value
    condition: selection

# Multiple conditions (AND)
detection:
    selection:
        Field1: value1
        Field2: value2
    condition: selection

# OR conditions
detection:
    selection1:
        Field: value1
    selection2:
        Field: value2
    condition: selection1 or selection2

# NOT conditions
detection:
    selection:
        Field: suspicious_value
    filter:
        Field: legitimate_value
    condition: selection and not filter

# Complex logic
detection:
    selection:
        EventID: 4688
    suspicious_cmd:
        CommandLine|contains:
            - 'powershell'
            - 'cmd.exe'
    filter_legitimate:
        ParentImage|endswith: '\explorer.exe'
    condition: selection and suspicious_cmd and not filter_legitimate
```

### Field Modifiers

Common modifiers for flexible matching:

- `|contains` - Contains substring (case-insensitive)
- `|endswith` - Ends with string
- `|startswith` - Starts with string
- `|re` - Regular expression match
- `|all` - All values must match
- `|base64` - Base64-encoded value matching
- `|base64offset` - Base64 with offset variations

Example:
```yaml
detection:
    selection:
        CommandLine|contains|all:
            - 'powershell'
            - '-enc'
        Image|endswith: '\powershell.exe'
```

## Security Considerations

- **Sensitive Data Handling**: Sigma rules may reference sensitive field names or patterns. Store rules in version control with appropriate access controls. Avoid including actual sensitive data in example values.

- **Access Control**: Detection rules reveal defensive capabilities to adversaries. Implement role-based access for rule repositories. Limit rule modification to authorized detection engineers.

- **Audit Logging**: Log all rule deployments, modifications, and deletions. Track who deployed which rules to which systems. Maintain change history for compliance auditing.

- **Compliance**: Sigma rules support compliance monitoring (PCI-DSS 10.2, NIST SP 800-53 AU family, ISO 27001 A.12.4). Document rule-to-control mappings for audit evidence.

- **Safe Defaul